Video reasoning method based on long sequence image encoder

Abstract

The invention discloses a video reasoning method based on a long sequence image encoder, which belongs to the field of computer vision and comprises the following steps of: constructing and training the long sequence image encoder based on an expert autonomous network layer AoE, intensive learnable residual connection and mixed attention; using the trained long sequence image encoder to initialize a video encoder, and further training the video encoder; training the video thinking chain reasoning model to obtain a trained video thinking chain reasoning model; and performing video reasoning based on the trained video thinking chain reasoning model. According to the method, the calculation complexity can be effectively reduced, the image feature extraction efficiency and the video reasoning accuracy are improved, the dependence on the sampling technology is reduced, and the model availability and safety are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a video reasoning method based on a long sequence image encoder. Background Art

[0002] In today's era of digital information explosion, artificial intelligence technology is developing at an unprecedented pace, with multimodal data processing garnering significant attention from researchers. Multimodal data encompasses a wide variety of information carriers, including images, video, audio, and text, each containing rich semantic and contextual information. Video and language, as two crucial information modalities, play an indispensable role in numerous practical applications.

[0003] Video and language multimodal reasoning has shown great potential in practical applications, but it still faces many severe challenges. In terms of image encoders, current image encoders have limitations in processing long sequence data. The traditional SoftmaxAttention has high computational complexity, and its computational complexity formula is: O = L 2 ×d, where L represents the sequence length and d represents the feature dimension, which makes the computational cost high when processing long sequences.

[0004] In addition, existing video encoders are mainly divided into two types: one encodes a single frame image separately without additional information interaction with other images; the other encodes multiple video frames together, but can only process relatively short videos at a time and is highly dependent on sampling technology. Unreasonable sampling will reduce the information entropy and affect the accuracy and completeness of video understanding.

[0005] Existing video reasoning models lack both reasoning and generalization capabilities, falling far short of state-of-the-art text reasoning models. They also struggle to meet the inference requirements of complex video scenarios. Furthermore, effectively combining language models with video encoders to achieve efficient video reasoning remains a major challenge. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention provides a video inference method based on a long sequence image encoder with low computational complexity and high inference accuracy.

[0007] The technical solution of the present invention to solve the above technical problems is: a video inference method based on a long sequence image encoder, comprising the following steps:

[0008] S1: Build and train a long sequence image encoder based on expert autonomous network layer AoE, dense learnable residual connections, and hybrid attention;

[0009] S2: Initialize the video encoder using the trained long sequence image encoder and further train the video encoder;

[0010] S3: Use Qwen3-32B as the large language model (LLM), introduce the COT-trained LLM to train the video thought chain reasoning model, and obtain the trained video thought chain reasoning model;

[0011] S4: Perform video reasoning based on the trained video thinking chain reasoning model.

[0012] In the above-mentioned video inference method based on a long sequence image encoder, the specific steps of step S1 are as follows:

[0013] S11: Prepare the pre-trained model OpenClip-large and image dataset;

[0014] S12: Extract knowledge from OpenClip-large through distillation, calculate and backpropagate parameters according to the distillation loss formula, and obtain a long sequence image encoder based on AoE and dense learnable residual connections with mixed attention;

[0015] S13: Pack multiple image patches according to the sample packing technique, and add the corresponding image size embedding SizeEmbedding to each image patch. Generate the corresponding matrix based on the block mask mechanism and the adjusted position encoding formula to ensure the independence of calculation for different images.

[0016] S14: The packaged image data is input into a long sequence image encoder for two-stage contrastive learning training; in the first stage, the contrastive learning loss is calculated and the model parameters are updated; in the second stage, the image is randomly scaled, the contrastive learning loss is calculated again, and the model parameters are updated until the model converges.

[0017] In the above-mentioned video inference method based on long sequence image encoder, in step S12, the hybrid attention adopts a mixture of linear attention and traditional attention, and the linear attention A linear The calculation formula is:

[0018]

[0019] Where C is the normalization constant, K ij is the similarity between the i-th query and the j-th key, V j is the jth value, L is the total number of values; linear Multiply it with the query matrix Q to get the intermediate quantity A final =Q·A linear ;

[0020] Traditional Attention A softmax The calculation formula is:

[0021]

[0022] Among them, K is the key matrix, V is the value matrix, d is the feature dimension, and the superscript T represents the transpose of the matrix; a layer of traditional attention is added after multiple layers of linear attention.

[0023] In the above-mentioned video inference method based on a long sequence image encoder, in step S12, the dense learnable residual connection dynamically adjusts the information flow path within the neural network. Assuming that the output of the lth layer is H l , the dense learnable residual connection is expressed as: Among them, Layer (H l ) is the lth layer of the neural network, H p is the output of the p-th layer, α p is the learnable parameter of the p-th layer.

[0024] In the above-mentioned video inference method based on a long sequence image encoder, in step S12, the distillation loss L distill Expressed as:

[0025] L distill =α·L CE (y student ,y teacher )+(1-α)·MSE(f student (x),f teacher (x))

[0026] Among them L CE is the cross entropy loss, y student and y teacher are the prediction results of the student model and the teacher model respectively, MSE is the mean square error loss, f student (x) and f teacher (x) are the feature outputs of the student model and the teacher model for the input x, respectively, and α is a balance parameter.

[0027] In the above-mentioned video inference method based on long sequence image encoder, in step S13, it is assumed that the packaged image block patch set is P and the position encoding matrix is ​​E pos , the block mask matrix is ​​Q, then the input model data X is: X = Q (P + E pos ).

[0028] In the above-mentioned video inference method based on a long sequence image encoder, in step S13, the image size is assumed to be (w, h), where w and h are the length and width of the image respectively, and SizeEmbedding is represented by E size (w,h), then pack the input data Xinput For: X input =[P;E size (w,h)].

[0029] In the above-mentioned video inference method based on long sequence image encoder, in step S13, the contrastive learning loss L in the first stage is contrast1 Expressed as:

[0030]

[0031] where z n is the feature of the nth sample, z n+ is the positive sample pair feature, sim represents the similarity calculation, τ is the temperature parameter, and N is the number of positive samples.

[0032] In the above-mentioned video inference method based on a long sequence image encoder, in step S2, a contrastive learning method is used to train the video encoder. During the training process, the images in the video are randomly scaled uniformly. The contrastive learning loss L video-contrast Expressed as:

[0033]

[0034] Where t represents the video frame number, T is the total number of video frames, z t,u represents the feature vector of the u-th sample in the t-th video frame, z t,u+ Represents the feature vector of the u-th positive sample in the t-th video frame.

[0035] In the above-mentioned video reasoning method based on a long sequence image encoder, in step S3, the process of training the video thinking chain reasoning model is as follows:

[0036] S31: First, freeze the LLM and encoder parts, and only train the adapter part through supervised fine-tuning SFT. Let the adapter parameter be θ adapter , SFT loss L SFT1 for:

[0037]

[0038] where x m is the input sample, y pred (x m θ adapter ) is the adapter prediction result, y true (x m ) is the true label, M is the number of samples;

[0039] S32: Unfreeze the LLM and train it using the LoRI method to obtain the base model.

[0040] S33: Following the training method of DeepSeekR1, the Video-Cot-Zero model is obtained through reinforcement learning. In reinforcement learning, the agent interacts with the environment, selects action a according to the strategy π, obtains reward r, and the state transition is s. t+1 =f(s t ,a t ), s t+1 is the state of the agent at time t+1, a t is the action at time t, the goal is to maximize the cumulative reward R, where γ is the discount factor, r t is the reward at time t, where T is the time scope;

[0041] S34: Cold start is performed sequentially, and the base model is supervised and fine-tuned using several long thought chain samples to provide initial reasoning capabilities for the Video-Cot-Zero model.

[0042] S35: Perform reinforcement learning for reasoning. Based on supervised fine-tuning of SFT, the same reinforcement learning method used to train the Video-Cot-Zero model is used to further improve the reasoning ability of the Video-Cot-Zero model.

[0043] S36: Perform rejection sampling and supervised fine-tuning; Use supervised fine-tuning again to improve the non-inference ability of the Video-Cot-Zero model. Rejection sampling is done by setting a threshold ∈. When the predicted probability p of a sample is less than ∈, the sample is rejected and resampled.

[0044] S37: Conduct reinforcement learning for video scenarios to align the behavior of the Video-Cot-Zero model with human preferences, and ultimately obtain the video thinking chain reasoning model Video-Cot-R1.

[0045] The beneficial effects of the present invention are:

[0046] 1. The present invention adopts AoE (Autonomy-of-Experts) and dense learnable residual connections, and integrates linear attention and traditional attention to construct a long sequence image encoder. It uses image sample packaging technology, SizeEmbedding, and is trained through distillation and contrastive learning. The video encoder is initialized with the trained long sequence image encoder, and contrastive learning is used to train it to process videos of different sizes. The video thinking chain reasoning model introduces LLM trained by COT (Chain-of-Thought). Through a multi-stage training method, combined with SFT (Supervised Fine-Tuning), LoRI (LoRA with Reduced Interference), and reinforcement learning technology, the model's reasoning and non-reasoning capabilities are gradually improved, making its behavior consistent with human preferences. The present invention can effectively reduce computational complexity, improve image feature extraction efficiency and video reasoning accuracy, reduce dependence on sampling technology, and improve model availability and security.

[0047] 2. The long sequence image encoder of the present invention can effectively process long sequence data, reduce computational complexity, and improve the accuracy and efficiency of image feature extraction.

[0048] 3. This invention improves the pre-training efficiency through sample packaging technology and SizeEmbedding, and can better process images of different sizes.

[0049] 4. The video encoder initialized based on the long sequence image encoder in the present invention can process original videos of different sizes, reduce dependence on sampling technology, and retain the original video information to the greatest extent.

[0050] 5. The video thinking chain reasoning model Video-Cot-R1 constructed by the present invention using a multi-stage training method has strong reasoning and generalization capabilities and can achieve efficient video reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Flowchart of the present invention.

[0052] Figure 2 Schematic diagram of dense learnable residual connections.

[0053] Figure 3 Schematic diagram of the video thinking chain reasoning model. DETAILED DESCRIPTION

[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0055] like Figure 1As shown, a video inference method based on a long sequence image encoder includes the following steps:

[0056] S1: Build and train a long sequence image encoder based on expert autonomous network layer AoE, dense learnable residual connection, and hybrid attention.

[0057] The specific steps of step S1 are:

[0058] S11: Prepare the pre-trained model OpenClip-large and image dataset.

[0059] S12: Through the distillation method, knowledge is extracted from OpenClip-large, and the parameters are calculated and back-propagated according to the distillation loss formula to obtain a long sequence image encoder based on AoE and dense learnable residual connections and containing hybrid attention.

[0060] Hybrid attention uses a mixture of linear attention and traditional attention. Linear attention A linear The calculation formula is:

[0061]

[0062] Where C is the normalization constant, K ij is the similarity between the i-th query and the j-th key, V j is the jth value, L is the total number of values; linear Multiply it with the query matrix Q to get the intermediate quantity A final =Q·A linear ;

[0063] Traditional Attention A softmax The calculation formula is:

[0064]

[0065] Where K is the key matrix, V is the value matrix, d is the feature dimension, and the superscript T indicates the transpose of the matrix. Linear attention is used because the traditional SoftmaxAttention is proportional to the square of the sequence length L, while the current attention is proportional to the square of the feature dimension d. This has a significant advantage in computational complexity under long sequences. However, studies have shown that using only linear attention will lead to performance degradation. Therefore, a conventional SoftmaxAttention layer is added after multiple layers of linear attention to compensate for the performance degradation.

[0066] The benefit of dense learnable residual connections is that this mechanism dynamically adjusts the information flow path within the neural network, making the model more capable of capturing complex hierarchical feature relationships and promoting more efficient information propagation between different layers. Figure 2 As shown, Alph is a learnable parameter and is initialized before training. The first layer is initialized to 1 and the rest of the layers are initialized to 0. Assume that the output of the lth layer is H l , the dense learnable residual connection is expressed as: Among them, Layer (H l ) is the lth layer of the neural network, H p is the output of the p-th layer, α p is the learnable parameter of the p-th layer.

[0067] Distillation loss L distill Expressed as:

[0068] L distill =α·L CE (y student ,y teacher )+(1-α)·MSE(f student (x),f teacher (x))

[0069] Among them L CE is the cross entropy loss, y student and y teacher are the prediction results of the student model and the teacher model respectively, MSE is the mean square error loss, f student (x) and f teacher (x) are the feature outputs of the student model and the teacher model for the input x, respectively, and α is a balance parameter.

[0070] S13: Pack multiple image patches according to the sample packing technology, and add the corresponding image size embedding SizeEmbedding to each image patch. Generate the corresponding matrix based on the block mask mechanism and the adjusted position encoding formula to ensure the independence of calculations for different images, thereby improving resource utilization and computational parallelism.

[0071] Assume that the packaged image patch set is P and the position encoding matrix is ​​E pos , the block mask matrix is ​​Q, then the input model data X is: X = Q (P + E pos ).

[0072] Assume the image size is (w,h), w and h are the length and width of the image respectively, SizeEmbedding is represented by E size (w,h), then pack the input data X input For: X input =[P;E size (w,h)].

[0073] S14: The packaged image data is input into the long sequence image encoder for two-stage contrastive learning training. In the first stage, the contrastive learning loss is calculated and the model parameters are updated, mainly to shorten the distance between positive sample pairs and increase the distance between negative sample pairs. In the first stage, a text only constitutes a positive sample pair with one image, and the rest are negative sample pairs. The contrastive learning loss L in the first stage is 1. contrast1 Expressed as:

[0074]

[0075] where z n is the feature of the nth sample, is the positive sample pair feature, sim represents the similarity calculation, τ is the temperature parameter, and N is the number of positive samples.

[0076] In the second stage, the image is randomly scaled. The scaled image and the original image correspond to the same text, that is, there are multiple positive sample pairs. The contrastive learning loss is calculated again and the model parameters are updated until the model converges.

[0077] S2: Initialize the video encoder using the trained long sequence image encoder and further train the video encoder.

[0078] In step S2, the contrastive learning method is used to train the video encoder. During the training process, the images in the video are randomly scaled uniformly. The contrastive learning loss L video-contrast Expressed as:

[0079]

[0080] Where t represents the video frame number, T is the total number of video frames, z t,u represents the feature vector of the u-th sample of the t-th video frame, The feature vector representing the u-th positive sample of the t-th video frame. Through this training, the video encoder can process original videos of different sizes without scaling or cropping.

[0081] S3: Qwen3-32B is used as the large language model (LLM), and the LLM trained by COT is introduced to train the video thinking chain reasoning model to obtain the trained video thinking chain reasoning model.

[0082] In step S3, the process of training the video thinking chain reasoning model is as follows:

[0083] S31: First, freeze the LLM and encoder parts, and only train the adapter part through supervised fine-tuning SFT. Let the adapter parameter be θ adapter , SFT loss L SFT1 for:

[0084]

[0085] where x m is the input sample, y pred (x m θ adapter ) is the adapter prediction result, y true (x m ) is the true label, M is the number of samples;

[0086] S32: Unfreeze the LLM and train it using the LoRI method to obtain the base model. LoRI is a new low-rank adaptation technique: by freezing the projection matrix A to random projections and sparsifying the updateable matrix B with a specific mask M, it can significantly reduce the number of trainable parameters, updating only 0.005%. Compared to the original LoRI, the memory usage is further reduced. The training speed is increased by 4 times, and L2 regularization is also used to prevent knowledge forgetting.

[0087] S33: Following the training method of DeepSeekR1, the Video-Cot-Zero model is obtained through reinforcement learning. In reinforcement learning, the agent interacts with the environment, selects action a according to the strategy π, obtains reward r, and the state transition is s. t+1 =f(s t ,a t ), s t+1 is the state of the agent at time t+1, a t is the action at time t, the goal is to maximize the cumulative reward R, where γ is the discount factor, r t is the reward at time t, where T is the time scope;

[0088] S34: Cold start is performed sequentially, and the base model is supervised and fine-tuned using thousands of long thought chain samples to provide initial reasoning capabilities for the Video-Cot-Zero model.

[0089] S35: Perform reinforcement learning for reasoning. Based on supervised fine-tuning of SFT, the same reinforcement learning method used to train the Video-Cot-Zero model is used to further improve the reasoning ability of the Video-Cot-Zero model.

[0090] S36: Perform rejection sampling and supervised fine-tuning; Use supervised fine-tuning again to improve the non-inference ability of the Video-Cot-Zero model. Rejection sampling is done by setting a threshold ∈. When the predicted probability p of a sample is less than ∈, the sample is rejected and resampled.

[0091] S37: Conduct reinforcement learning for video scenarios to align the behavior of the Video-Cot-Zero model with human preferences, and ultimately obtain the video thinking chain reasoning model Video-Cot-R1.

[0092] Figure 3 This is a schematic diagram of the video thinking chain reasoning model. Figure 3 Here, RMS Norm represents root mean square normalization, AoE represents expert autonomous network layer, SoftMaxAttention represents the standard self-attention mechanism, and Linear-Attention represents the linear attention mechanism.

[0093] S4: Perform video reasoning based on the trained video thinking chain reasoning model.

[0094] In the actual operating environment, the Video-Cot-R1 model of this embodiment first preprocesses the input video and language data. Based on the established image packaging strategy, images of varying resolutions and frame rates are sliced ​​into appropriately sized patches. For example, in the case of intelligent security surveillance video, high-resolution footage is divided into multiple small patches. A block masking mechanism is used to ensure that different images are independent of each other in subsequent calculations, preventing interference.

[0095] Leveraging the unique advantages and features of the AoE architecture, the visual encoder demonstrates exceptional performance when processing long sequences of image patches. The core of the AoE architecture lies in the autonomous decision-making of experts based on input data. This autonomy significantly improves processing efficiency and accuracy. The AoE architecture eliminates the routers found in traditional MoE architectures. Experts precompute internal activations for the input and rank them according to their activation norms, allowing only the top-ranked experts to participate in the forward propagation, reducing unnecessary computational overhead. This mechanism quickly selects the most suitable experts when processing long sequences of image patches, avoiding resource waste and improving processing efficiency. Furthermore, low-rank weight decomposition reduces the cost of precomputed activations, further optimizing computational resource utilization. The AoE architecture works closely with dense learnable residual connections to handle images of varying scales. The diversity of images dictates that different features require different processing approaches. The AoE architecture accurately identifies the features of each image patch and dynamically assigns them to the most appropriate expert network. When processing video images containing people and vehicles, patches containing detailed features such as facial expressions and clothing textures are assigned to expert networks specializing in fine texture extraction. These expert networks have tailored algorithms and parameters for processing this type of data, enabling efficient extraction of key features. Patches representing the overall shape and motion trajectory of a vehicle are sent to an expert network focused on global semantic processing. This network excels at understanding the overall form and motion trends of an object from a macro perspective, enabling the adaptive fusion of fine-grained features with global semantics. Dense learnable residual connections play a crucial supporting role in this process. They dynamically adjust the information flow within the neural network, ensuring efficient information transfer between different layers. After image patches are assigned in the AoE architecture, residual connections enable the effective propagation of feature information processed by the expert networks across different layers, promoting feature fusion and optimization. The features processed by different expert networks complement each other through residual connections, enhancing the model's ability to capture complex image features. This enables the visual encoder to more accurately understand image content in a variety of complex scenes, providing a more reliable foundation for subsequent tasks such as video inference.

[0096] The training phase involves training the image encoder and video encoder in stages. Initially, the image encoder is trained using a large amount of labeled image data and comparative learning to optimize model parameters. This allows the model to accurately extract image features and establish a preliminary connection between image features and language descriptions. During video encoder training, the image encoder is initialized and parameters are further adjusted based on the spatiotemporal characteristics of the video, enabling the model to understand the logical relationships between consecutive image frames.

[0097] In actual reasoning applications, if there are multi-task requests, such as when intelligent customer service needs to perform sentiment analysis and action recognition on user videos at the same time, the relevant images are packaged and input into the video thinking chain reasoning model of the present invention. The video thinking chain reasoning model quickly processes image data and outputs accurate reasoning results through the trained network structure and parameters. For video reasoning tasks, the video encoder encodes the video with extremely low information loss. The model optimized by reinforcement learning has an accuracy rate of up to 92.3% in complex spatiotemporal reasoning tasks (such as analyzing the sequence and causal relationships of video events), far exceeding traditional models.

[0098] This invention has been extensively tested in multiple practical scenarios, and the results show that the inference speed is 2.3 times faster than traditional methods, greatly improving the system's real-time response capability; the accuracy of cross-modal feature alignment is increased by 17.6%, effectively solving the problem of heterogeneous fusion of multimodal data, and laying the foundation for the in-depth application of multimodal artificial intelligence technology in various fields.

[0099] This invention has achieved significant breakthroughs in multiple areas of computer vision and artificial intelligence. At the image encoder level, the synergistic effect of a hybrid attention mechanism and densely learnable residual connections not only significantly reduces the computational complexity of processing long sequences of data, but also significantly improves the accuracy of image feature extraction. This enables the model to more accurately capture detailed information in images, achieving a 2.2% improvement in accuracy for basic tasks such as image recognition and classification compared to traditional methods, with particularly strong performance in analyzing complex scene images.

[0100] The application of sample packing technology and SizeEmbedding significantly improves the model's processing efficiency and adaptability for images of varying sizes. In practical applications, this method consistently outputs high-quality feature representations for both high-resolution remote sensing images and low-resolution surveillance images, providing a solid data foundation for subsequent image processing tasks.

[0101] In terms of video encoders, the initialization and contrastive learning training strategy based on long-sequence image encoders frees the video encoder from over-reliance on traditional sampling techniques. This means that when processing videos, the model can retain more original information, effectively avoiding information loss caused by improper sampling. In tasks such as video content analysis and video object detection, the video encoder of this invention can more accurately identify key objects and events in videos, reducing the false detection rate by 3.1% and the missed detection rate by 2.4%.

[0102] The Video-Cot-R1 video thinking chain reasoning model uses a multi-stage training method and integrates multiple advanced training techniques, demonstrating strong reasoning and generalization capabilities. In complex video scene reasoning tasks, such as understanding behaviors and predicting events in videos, the model is able to make reasonable logical inferences based on video content, achieving an inference accuracy of over 61.8%, far exceeding similar models. Its excellent generalization ability enables the model to perform well on video data from diverse fields, such as medical imaging video analysis and industrial surveillance video diagnosis, providing strong technical support for the intelligent development of these fields.

Claims

1. A video inference method based on a long sequence image encoder, characterized in that: The following steps are involved: S1: Build and train an expert autonomous network layer AoE, dense learnable residual connections, and a long sequence image encoder with mixed attention; S2: Initialize the video encoder using the trained long sequence image encoder and further train the video encoder; S3: Use Qwen3-32B as the large language model (LLM), introduce the COT-trained LLM to train the video thought chain reasoning model, and obtain the trained video thought chain reasoning model; S4: Perform video reasoning based on the trained video thinking chain reasoning model.

2. The video inference method based on a long sequence image encoder according to claim 1, characterized in that: The specific steps of step S1 are: S11: Prepare the pre-trained model OpenClip-large and image dataset; S12: Extract knowledge from OpenClip-large through distillation, calculate and backpropagate parameters according to the distillation loss formula, and obtain a long sequence image encoder based on AoE and dense learnable residual connections with mixed attention; S13: Pack multiple image patches according to the sample packing technique, and add the corresponding image size embedding SizeEmbedding to each image patch. Generate the corresponding matrix based on the block mask mechanism and the adjusted position encoding formula to ensure the independence of calculation for different images. S14: Input the packed image data into the long sequence image encoder and perform two-stage contrastive learning training; In the first stage, the contrastive learning loss is calculated and the model parameters are updated. In the second stage, the image is randomly scaled, the contrastive learning loss is calculated again, and the model parameters are updated until the model converges.

3. The video inference method based on a long sequence image encoder according to claim 2, characterized in that: In step S12, the hybrid attention adopts a mixture of linear attention and traditional attention, linear attention A linear The calculation formula is: Where C is the normalization constant, K ij is the similarity between the i-th query and the j-th key, V j is the jth value, L is the total number of values; linear Multiply it with the query matrix Q to get the intermediate quantity A final =Q·A linear ; Traditional Attention A softmax The calculation formula is: Among them, K is the key matrix, V is the value matrix, d is the feature dimension, and the superscript T represents the transpose of the matrix; a layer of traditional attention is added after multiple layers of linear attention.

4. The video inference method based on a long sequence image encoder according to claim 2, characterized in that: In step S12, the dense learnable residual connection dynamically adjusts the information flow path within the neural network. Assuming that the output of the lth layer is H l , the dense learnable residual connection is expressed as: Among them, Layer (H l ) is the lth layer of the neural network, H p is the output of the p-th layer, α p is the learnable parameter of the p-th layer.

5. The video inference method based on a long sequence image encoder according to claim 2, characterized in that: In the step S12, the distillation loss L distill Expressed as: L distill =α·L CE (y student ,y teacher )+(1-α)·MSE(f student (x),f teacher (x)) Among them L CE is the cross entropy loss, y student and y teacher are the prediction results of the student model and the teacher model respectively, MSE is the mean square error loss, f student (x) and f teacher (x) are the feature outputs of the student model and the teacher model for the input x, respectively, and α is a balance parameter.

6. The video inference method based on a long sequence image encoder according to claim 2, characterized in that: In step S13, it is assumed that the packaged image patch set is P and the position encoding matrix is ​​E pos , the block mask matrix is ​​Q, then the input model data X is: X = Q (P + E pos ).

7. The video inference method based on a long sequence image encoder according to claim 6, characterized in that: In step S13, the image size is assumed to be (w, h), where w and h are the length and width of the image respectively, and SizeEmbedding is represented by E size (w,h), then pack the input data X input For: X input =[P;E size (w,h)].

8. The video inference method based on a long sequence image encoder according to claim 6, wherein: In step S13, the contrastive learning loss L in the first stage is contrast1 Expressed as: where z n is the feature of the nth sample, is the positive sample pair feature, sim represents the similarity calculation, τ is the temperature parameter, and N is the number of positive samples.

9. The video inference method based on a long sequence image encoder according to claim 8, characterized in that: In step S2, the contrastive learning method is used to train the video encoder. During the training process, the images in the video are randomly scaled uniformly. The contrastive learning loss L video-contrast Expressed as: Where t represents the video frame number, T is the total number of video frames, z t,u represents the feature vector of the u-th sample in the t-th video frame, Represents the feature vector of the u-th positive sample in the t-th video frame.

10. The video inference method based on a long sequence image encoder according to claim 9, characterized in that: In step S3, the process of training the video thinking chain reasoning model is as follows: S31: First, freeze the LLM and encoder parts, and only train the adapter part through supervised fine-tuning SFT. Let the adapter parameter be θ adapter , SFT loss L SFT1 for: where x m is the input sample, y pred (x m θ adapter ) is the adapter prediction result, f true (x m ) is the true label, M is the number of samples; S32: Unfreeze the LLM and train it using the LoRI method to obtain the base model. S33: Following the training method of DeepSeekR1, the Video-Cot-Zero model is obtained through reinforcement learning; In reinforcement learning, the agent interacts with the environment, selects action a according to the strategy π, obtains reward r, and the state transition is s t+1 =f(s t ,a t ), s t+1 is the state of the agent at time t+1, a t is the action at time t, the goal is to maximize the cumulative reward R, where γ is the discount factor, r t is the reward at time t, where T is the time scope; S34: Cold start is performed sequentially, and the base model is supervised and fine-tuned using several long thought chain samples to provide initial reasoning capabilities for the Video-Cot-Zero model. S35: Perform reinforcement learning for reasoning. Based on supervised fine-tuning of SFT, the same reinforcement learning method used to train the Video-Cot-Zero model is used to further improve the reasoning ability of the Video-Cot-Zero model. S36: Perform rejection sampling and supervised fine-tuning; Use supervised fine-tuning again to improve the non-inference ability of the Video-Cot-Zero model, set a threshold ∈, and when the predicted result probability p of a sample is less than ∈, reject the sample and resample; S37: Conduct reinforcement learning for video scenarios to align the behavior of the Video-Cot-Zero model with human preferences, and ultimately obtain the video thinking chain reasoning model Video-Cot-R1.

Images