Method and apparatus for processing frame sequence, and electronic device

Abstract

The application provides a frame sequence processing method and device and electronic equipment. The processing method comprises grouping an input frame sequence according to inter-frame differences to obtain a plurality of sub-segments; adjusting preset weights in a visual encoder in a time dimension by using the length of each sub-segment to obtain adjusted coding weights; and coding each sub-segment into a fixed number of feature units by using the adjusted coding weights and through the visual encoder. According to the example embodiments of the application, the visual encoder can adaptively adjust a time perception window according to different segment lengths, and a fixed number of feature units are generated under shared parameters, so that the multi-scale time modeling capability is realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for processing frame sequences, an electronic device, and a non-transitory computer-readable storage medium. Background Technology

[0002] The Video Transformer is a core modeling framework in current video understanding tasks. Its main idea is to extend the Vision Transformer (ViT) from two-dimensional images to three-dimensional spatiotemporal data, enabling unified modeling of video frame sequences. Compared to image tasks, the Video Transformer needs to process both spatial and temporal dynamic information simultaneously, thus facing significant challenges in its structural design.

[0003] Existing methods typically divide videos into spatiotemporal blocks and model cross-frame dependencies using self-attention mechanisms. However, with the introduction of the temporal dimension, the number of feature unit tokens grows linearly or even superlinearly, leading to a significant increase in computational complexity and memory overhead. To alleviate this problem, several improvement strategies have been proposed: one type of method employs spatiotemporal attention decomposition (such as TimeSformer and ViViT), reducing computational complexity by separating spatial and temporal attention; another type of method introduces hierarchical structures (such as MViT and Video Swin Transformer), reducing the number of tokens through layer-by-layer downsampling and enhancing multi-scale modeling capabilities.

[0004] While the methods described above improve efficiency to some extent, they still rely on frame-by-frame or fixed-window video encoding, making it difficult to adaptively adjust computational resource allocation based on dynamic changes in video content. This "uniform modeling" strategy can easily introduce a large amount of redundant computation when processing long or high-frame-rate videos.

[0005] To address the computational burden caused by the large amount of redundant information in video data, token compression technology is widely used in video understanding models. Its core objective is to reduce the number of input tokens while preserving as much key information as possible, thereby reducing computational complexity.

[0006] Existing methods mainly include the following categories: (1) Token pruning: dynamically remove redundant tokens based on importance scores and retain only the parts that contribute to the task; (2) Keyframe selection: select representative frames from the video for encoding to reduce temporal redundancy; (3) Feature pooling / merging: aggregate similar tokens at the feature level to reduce the representation dimension; (4) Pre-encoding compression: crop or downsample video frames or patches before they enter the visual encoder.

[0007] While these methods have achieved some success in reducing computational overhead, they still have significant limitations. On the one hand, most methods perform compression after encoding, failing to reduce the computational cost of front-end visual encoding; on the other hand, some methods (such as keyframe selection) disrupt temporal continuity, thereby affecting the model's ability to model dynamic processes.

[0008] Furthermore, most existing compression methods are based on heuristic rules or fixed strategies, lacking explicit modeling of the dynamic characteristics of video time, making it difficult to achieve an ideal balance between "efficiency" and "information integrity". Summary of the Invention

[0009] This application proposes a frame sequence processing method and apparatus, electronic device, and non-transient computer-readable storage medium to solve the problem that existing video understanding methods cannot simultaneously achieve temporal modeling and computational efficiency.

[0010] According to one aspect of this application, a method for processing frame sequences is proposed, comprising: The input frame sequence is grouped according to the inter-frame differences to obtain multiple sub-segments; The preset weights in the visual encoder are adjusted in the time dimension using the length of each sub-segment to obtain the adjusted encoding weights. Using the adjusted encoding weights, each sub-segment is encoded into a fixed number of feature units by the visual encoder.

[0011] According to some embodiments, before grouping the input frame sequence according to inter-frame differences to obtain multiple sub-segments, the method further includes: The initial weights in the visual encoder are periodically expanded, and the expanded weights are used as the preset weights.

[0012] According to some embodiments, the initial weights in the visual encoder are periodically expanded using the following formula:

[0013] Where r is the expansion ratio, , This indicates the spatial scale of the initial weights. This indicates the spatial scale after the periodic expansion. This represents the initial weights. The weight after period expansion is the preset weight.

[0014] According to some embodiments, the input frame sequence is grouped according to inter-frame differences to obtain multiple sub-segments, including: Determine a reference frame in the frame sequence, wherein the reference frame is the first frame in the current sub-segment; Compare the pixel difference values ​​between the reference frame and subsequent frames of the reference frame; When the pixel difference value is greater than a preset threshold, the frame preceding the frame in which the current pixel difference value is calculated is taken as the last frame of the current sub-segment. Using the frame from which the current pixel difference value is calculated as a reference frame, grouping continues until all sub-segments of the frame sequence are obtained.

[0015] According to some embodiments, the preset weights in the visual encoder are adjusted in the time dimension using the length of each sub-segment to obtain the adjusted encoding weights, including: The time dimension corresponding to the adjusted coding weight is determined by using the length of each sub-segment; The time difference matrix is ​​determined using the time dimension and the initial time dimension corresponding to the preset weight; Calculate the pseudo-inverse matrix corresponding to the time difference matrix; The adjusted encoding weights are obtained using the pseudo-inverse matrix and the preset weights.

[0016] According to some embodiments, the adjusted encoding weights are obtained using the pseudo-inverse matrix and the preset weights, achieved by the following formula:

[0017] in, The weight value corresponding to each spatiotemporal video block in the preset weights, The weight values ​​for each spatiotemporal video block in the adjusted encoding weights. , Let B be the pseudo-inverse matrix, and let B be the time difference matrix.

[0018] According to some embodiments, the number of feature units corresponding to the frame sequence is as follows:

[0019] Where N is the number of feature units corresponding to the frame sequence. and These represent the width and height of each frame in the frame sequence, respectively. K represents the spatial size of each spatiotemporal video block, and K represents the number of sub-segments included in the frame sequence.

[0020] According to some embodiments, before grouping the input frame sequence according to inter-frame differences to obtain multiple sub-segments, the method further includes: The images are copied to obtain a frame sequence comprising multiple images.

[0021] According to some embodiments, the processing method further includes calculating the coding efficiency using the following formula:

[0022] Where TFLOP is a trillion floating-point operations, H and W represent the height and width of the frame sequence, and T is the number of frames included in the frame sequence.

[0023] According to some embodiments, a frame sequence processing apparatus is characterized by comprising: The frame sequence grouping unit is used to group the input frame sequence according to the inter-frame differences to obtain multiple sub-segments; The weight adjustment unit is used to adjust the preset weights in the visual encoder in the time dimension using the length of each sub-segment to obtain the adjusted encoding weights. The feature extraction unit is used to encode each sub-segment into a fixed number of feature units by the visual encoder using the adjusted encoding weights.

[0024] According to one aspect of this application, an electronic device is provided, comprising: a processor; a memory for storing a computer program; wherein when the computer program is executed by the processor, the processor causes the processor to perform the processing method as described in any of the preceding embodiments.

[0025] According to one aspect of this application, a non-transitory computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a processor, cause the processor to perform the processing method as described in any of the preceding embodiments.

[0026] According to an example embodiment of this application, multi-scale temporal modeling capability is achieved by enabling the visual encoder to adaptively adjust the time-aware window according to different segment lengths and to generate a fixed number of feature units under shared parameters. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0028] Figure 1 A flowchart illustrating a method for processing a frame sequence according to an example embodiment of this application is shown.

[0029] Figure 2 A schematic diagram illustrating the processing procedure of a frame sequence according to an example embodiment of this application is shown.

[0030] Figure 3 The diagram illustrates the performance of a frame sequence processing method according to an example embodiment of this application on several mainstream video understanding benchmarks.

[0031] Figure 4 The diagram illustrates a comparison of the efficiency and performance of FlexiVideo and Qwen2.5-VL, a frame sequence processing method according to an example embodiment of this application, as the number of input frames changes.

[0032] Figure 5 A schematic diagram showing the performance and efficiency comparison of different encoding strategies according to example embodiments of this application on MotionBench and MLVU is provided.

[0033] Figure 6 The feature distribution and model output of FlexiVideo (bottom) and Qwen2.5-VL (top) are compared according to an example embodiment of this application.

[0034] Figure 7 A block diagram of a frame sequence processing apparatus according to an example embodiment of this application is shown.

[0035] Figure 8 An electronic device is shown according to an exemplary embodiment of this application. Detailed Implementation

[0036] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0037] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of these specific details, or other methods, components, materials, apparatus, or operations may be employed. In these cases, well-known structures, methods, apparatuses, implementations, materials, or operations will not be shown or described in detail.

[0038] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0039] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0040] This application addresses the common problems in the temporal modeling of existing multimodal video models, such as severe redundant coding, high computational overhead, and visual confusion caused by fixed multi-frame coding, and proposes a frame sequence processing method.

[0041] According to embodiments of this application, firstly, by introducing an adaptive temporal segmentation mechanism, a change-aware analysis is performed on the video frame sequence before visual feature extraction. Continuous frames with subtle changes are aggregated into unified scene segments, while regions with drastic changes are further finely divided. This reconstructs an input structure that better conforms to the inherent temporal dynamics of the video from the source. This approach avoids redundant computation caused by indiscriminately processing all frames, allowing computational resources to be more concentrated on key regions with dense semantic changes, thus improving overall coding efficiency without significantly increasing system complexity.

[0042] Secondly, this scheme utilizes a dynamic spatiotemporal embedding mechanism, enabling the visual encoder to adaptively adjust the temporal perception window based on the duration of different scene segments, achieving multi-granularity temporal modeling capabilities while sharing a parameter space. Compared to multi-frame encoding strategies with fixed time windows, this scheme employs smaller time windows in high-dynamic scenes to preserve fine-grained motion information, and larger time windows in low-dynamic scenes to achieve efficient compression. This establishes a dynamic balance between computational efficiency and representation accuracy, effectively mitigating the visual confusion caused by fixed multi-frame joint encoding during scene transitions or drastic movements. It also improves semantic consistency and feature stability in the temporal dimension, allowing the model to maintain coherent and accurate understanding at event boundaries.

[0043] Furthermore, this application does not require large-scale modifications to the existing backbone network structure. It can significantly enhance the temporal modeling capability through lightweight segmentation and adaptive embedding design, exhibiting good compatibility and scalability. It can also directly adapt to mainstream multimodal large language model frameworks, reducing system transformation costs and shortening the deployment cycle. At the same time, it supports flexible operation under different frame rates, video lengths, and computing power conditions, demonstrating strong engineering application value.

[0044] Furthermore, this application significantly reduces the number of visual feature unit tokens and memory usage while maintaining or improving video understanding accuracy, thereby increasing inference speed and resource utilization efficiency. It is particularly suitable for real-time applications in scenarios with high frame rate videos, long video analysis, and limited computing power. It achieves efficient compression while maintaining fine-grained temporal dynamic capture capabilities, resulting in a better balance between performance and efficiency and enhancing overall robustness and stability.

[0045] The specific embodiments according to this application will now be described in detail with reference to the accompanying drawings.

[0046] Figure 1 A flowchart illustrating a frame sequence processing method according to an example embodiment of this application is shown, such as... Figure 1 The processing method shown includes steps S101, S103, and S105. This embodiment can process both images and video data.

[0047] The following is based on Figure 1 Taking an example, a method for processing a frame sequence according to an example embodiment of this application will be described in detail.

[0048] like Figure 1 As shown, in step S101, the input frame sequence is grouped according to the inter-frame differences to obtain multiple sub-segments.

[0049] When the data to be processed is an image, in order to obtain the frame sequence, according to the embodiments of this application, it is necessary to copy the image before step S101 to obtain a frame sequence including multiple images. When the data to be processed is video data, this step is required, and step S101 can be executed directly.

[0050] Traditional video multi-frame coding schemes typically employ fixed coding strategies, ignoring the inherent variability of video temporal dynamics. Inspired by the efficiency of the human visual system—which exhibits higher temporal sensitivity to dynamic regions and tends to integrate static regions over longer time scales—according to embodiments of this application, in step S101, the input frame sequence is adaptively segmented based on the local motion intensity, utilizing the temporal sensitivity of the "visual system."

[0051] For example, continuous frames with subtle content changes are aggregated into longer scene segments to make full use of temporal redundancy; while dynamic areas with drastic changes are divided into denser scene segments to capture fine-grained motion information.

[0052] In some embodiments, when dividing a frame sequence into multiple sub-segments, each frame in each sub-segment is compared with the first frame to determine whether the difference value exceeds a preset threshold, thereby determining the position of the segment.

[0053] Specifically, step S101 may include the following sub-steps: In sub-step S1011, a reference frame is determined in the frame sequence, and the reference frame is the first frame in the current sub-segment; In sub-step S1013, the pixel difference values ​​between the reference frame and subsequent frames of the reference frame are compared. In sub-step S1015, when the pixel difference value is greater than a preset threshold, the frame preceding the frame in which the current pixel difference value is calculated is taken as the last frame of the current sub-segment. In sub-step S1017, the frame for calculating the current pixel difference value is used as the reference frame, and grouping continues until all sub-segments of the frame sequence are obtained.

[0054] In other embodiments, each frame in each sub-segment can be compared with the previous frame to determine whether the difference value exceeds a preset threshold, thereby determining the position of the segment.

[0055] According to embodiments of this application, a preset index is used to quantify the temporal changes between frames. Given an index containing... Frame sequence and a time-varying threshold From reference frame Initially, the length of the relative static fragment. As shown in formula (1).

[0056] (1) in, This indicates the difference between frames.

[0057] Therefore, the input frame sequence It was divided into a separate scene. Then, frame by frame... As a new reference frame, subsequent frames are further segmented. By iteratively applying this rule starting from the initial reference frame, the original... The frame sequence was reassembled into Scene clips of varying lengths , .

[0058] It should be noted that this application does not limit the specific method of inter-frame difference. In a specific embodiment, the inter-frame difference can be calculated based on the inter-frame pixel-level difference.

[0059] This application achieves a good balance between computational efficiency and representational fidelity by reconstructing frame sequences into a representation that better conforms to the inherent temporal dynamics of video, without introducing almost any additional overhead.

[0060] In step S103, the preset weights in the visual encoder are adjusted in the time dimension using the length of each sub-segment to obtain the adjusted encoding weights.

[0061] Existing video multimodal large language models (such as the Qwen2-VL series) employ 3D spatiotemporal block embedding in the visual encoder, jointly modeling spatiotemporal features through 3D convolution to reduce redundant tokens. According to embodiments of this application, in order to give the model a longer initial temporal receptive field without training from scratch, before step S101 or S103, Figure 1 The method also includes periodically expanding the initial weights of the patch embedding units in the visual encoder, and using the expanded weights as the preset weights.

[0062] In a specific embodiment, given initial visual encoder weights... ,in, The size of the spatial block is used to construct the periodic expansion of the expanded convolutional kernel. .

[0063] For example, the initial weights of the patch embedding can be periodically expanded using formula (2).

[0064] (2) in, r is the expansion ratio. , This indicates the spatial scale of the initial weights. This indicates the spatial scale after the periodic expansion. This represents the initial weights. The weights after period expansion are the preset weights. Indicates the number of input channels (generally RGB, 3-channel input). Indicates the number of channels embedded in the output. Indicates the block size.

[0065] This embodiment uses periodic replication and normalization of the initial weights of the patch embedding to enable the pre-trained image encoder to seamlessly adapt to modeling needs over a long period of time.

[0066] In step S105, each sub-segment is encoded into a fixed number of feature units by the visual encoder using the adjusted encoding weights.

[0067] Because a fixed window size makes dynamic temporal modeling difficult and can easily cause visual confusion when faced with frame sequences with varying degrees of visual change, this application addresses these issues by adaptively adjusting the temporal encoding method based on the generated scene segments of different lengths, while sharing the same patchbedding kernel. This allows different time-aware windows to be compatible, enabling a single 3D convolutional kernel to dynamically adapt to any time length, supporting multi-granularity temporal awareness, and ensuring that each sub-segment can be encoded into a fixed number of feature units.

[0068] According to an embodiment of this application, step S105 further includes the following sub-steps: In sub-step S1051, the time dimension corresponding to the adjusted coding weight is determined using the length of each sub-segment; for example, the length value of the sub-segment is used as the time dimension value corresponding to the adjusted coding weight.

[0069] In sub-step S1053, the time difference matrix is ​​determined using the time dimension and the initial time dimension corresponding to the preset weight; In sub-step S1055, the pseudo-inverse matrix corresponding to the time difference matrix is ​​calculated; In sub-step S1057, the adjusted encoding weights are obtained using the pseudo-inverse matrix and the preset weights.

[0070] In some embodiments, sub-step S1053 obtains the time difference matrix through the linear transformation shown in formula (3).

[0071] (3) in, For step S103, the time dimension corresponding to the adjusted encoding weights is obtained. The time dimension corresponding to each sub-segment is obtained from sub-step S1051. This is the time interpolation matrix.

[0072] In some embodiments, sub-step S1057 can be implemented by formula (5).

[0073] (5) in, The weight value corresponding to each spatiotemporal video block in the preset weights, The weight values ​​for each spatiotemporal video block in the adjusted encoding weights. , Let B be the pseudo-inverse matrix, and let B be the time difference matrix.

[0074] This transformation method is computationally efficient, introduces almost no additional overhead, and can be applied independently across all channels and spatial locations, with any length... Scene clips can all be obtained from shared resources. Dynamically generate the corresponding patch embedding kernel .

[0075] According to an embodiment of this application, the number of feature unit tokens corresponding to the frame sequence is as shown in formula (5).

[0076] (5) Where N is the number of feature units corresponding to the frame sequence. and These represent the width and height of each frame in the frame sequence, respectively. K represents the spatial size of each spatiotemporal video block, and K represents the number of sub-segments included in the frame sequence.

[0077] according to Figure 1 The embodiment shown achieves multi-scale temporal modeling capability by enabling the visual encoder to adaptively adjust the time-aware window according to different segment lengths and generate a fixed number of feature units under shared parameters.

[0078] According to other embodiments, by periodically expanding the initial weights of the patch embedding, the pre-trained model can support modeling over a longer time range without retraining, improving the practicality of the solution. Unlike traditional fixed-window or post-processing compression methods, this application introduces change-aware structured modeling before visual encoding, combining temporal dynamic modeling with token compression. This effectively reduces redundant computation while maintaining temporal continuity and semantic consistency, thereby improving the efficiency and robustness of video understanding.

[0079] The adaptive encoding method proposed in this application can flexibly adjust the temporal awareness granularity without significantly increasing the parameter scale. For example, a larger time window is used to reduce the number of tokens, thereby reducing computational costs and improving inference efficiency; while a smaller time window is used to retain fine-grained temporal details to maintain sensitivity to local motion dynamics. These two approaches work together to achieve a balance between computational efficiency and representation granularity, enabling the model to possess true dynamic video language modeling capabilities. This collaborative mechanism not only enhances the robustness of temporal inputs but also allows the visual encoder to capture more fine-grained visual changes. Simultaneously, it requires no modification to the backbone network, introducing only a very small number of additional trainable parameters. The resulting general encoding framework provides an efficient paradigm for understanding high frame rate, long-duration videos, adaptively capturing multi-layered dynamic semantic information while maintaining representation consistency.

[0080] This application optimizes the video input organization and spatiotemporal coding strategy at the structural level through a change-aware, scene-level temporal modeling mechanism. This not only effectively suppresses visual confusion and improves semantic consistency but also significantly reduces computational overhead and enhances adaptability to complex temporal dynamics. Therefore, it possesses outstanding technological advancement and broad application prospects in the field of video understanding.

[0081] Figure 2 A schematic diagram illustrating a frame sequence processing procedure according to an example embodiment of this application is shown. Figure 2 As shown, for video input, the model first identifies multiple scene segments with relatively smooth visual changes through inter-frame difference analysis. Then, dynamically sized convolutional kernels are used to encode each spatiotemporal block, generating corresponding visual embeddings, thus ensuring that each relatively static segment can be mapped to a fixed number of tokens. For image input, frame repetition is used during training to enhance the learning of static representations. Finally, these visual embeddings are processed by a visual encoder and a visual-language mapping layer to generate visual feature unit tokens for subsequent language modeling.

[0082] To compare the effectiveness of the technical solution proposed in this application, the technical solution proposed in this application (identified by FlexiVideo) is compared with the prior art.

[0083] In this embodiment, FlexiVideo is initialized based on Qwen2.5-VL, where the temporal dimension of the visual encoder is set to 6, and the time variation threshold is... Set to 0.2. In the temporal dimension of multidimensional rotational position embedding, FlexiVideo redesigned the calculation method for temporal position encoding to match different time-aware windows. For example, a larger encoding interval is assigned to scenes with subtle visual changes, while a smaller encoding interval is used for areas of rapid change to more finely depict rapidly changing temporal dynamics.

[0084] During training, the model was fully fine-tuned using a total of 140k samples, with a learning rate set to [value missing]. The global batch size was 64, and the warm-up ratio was 0.03. All experiments were conducted on four servers, each equipped with eight A100-80GB GPUs. It should be noted that this embodiment did not exhaustively tune the hyperparameters; further optimization is expected to bring additional improvements in both accuracy and efficiency.

[0085] In this embodiment, the model training data consists of 35k images and 105k videos. The image data is obtained by random sampling from the PixMo dataset; the video data includes 75k samples from the LLaVA-Video-178K dataset and 30k samples from the ShareGPT4Video dataset.

[0086] First Embodiment First, FlexiVideo was evaluated on a diverse set of video understanding benchmarks, which collectively cover comprehensive understanding, long-duration videos, and various motion-centric application scenarios. Specifically, these include comprehensive benchmarks: Video-MME and MLVU, which cover multiple video types and durations to evaluate the model's general video understanding capabilities; long-video benchmarks: LongVideoBench focuses on evaluating reasoning capabilities for manually annotated long videos, while LVBench focuses on memory retention and delayed understanding capabilities over extremely long time spans; and motion-oriented benchmarks: MotionBench and FavorBench, through structured, action-centric tasks, emphasize the model's ability to understand and reason about fine-grained temporal dynamics and motion relationships.

[0087] Figure 3This diagram illustrates the performance of FlexiVideo, a frame sequence processing method according to an example embodiment of this application, on several mainstream video understanding benchmarks. Experimental results show that FlexiVideo significantly outperforms the compared methods while maintaining high computational efficiency. On MotionBench and FavorBench, FlexiVideo demonstrates a clear advantage in motion understanding, stemming from its adaptive temporal segmentation of frames with subtle visual changes, effectively reducing redundant coding. In Video-MME's "no subtitle" setting, FlexiVideo achieves an accuracy of 62.5%, setting a new current best result.

[0088] On long video understanding benchmarks such as LongVideoBench and LVBench, dynamic spatiotemporal embedding is used to adaptively adjust the time-aware window for scene-level encoding, enabling FlexiVideo to effectively capture complex temporal change patterns. These results fully validate the effectiveness of change-aware temporal dynamic modeling and further highlight the crucial role of this application's technical solution in reducing computational overhead while maintaining fine-grained visual details, thereby improving model efficiency and robustness.

[0089] Second Embodiment Limited by hardware resources, when the total token budget is fixed and the number of input frames increases, it is usually necessary to adaptively reduce the spatial resolution to avoid exceeding the token limit. Therefore, simply using FLOPs (Floating Point Operations) as an efficiency indicator cannot fully reflect the true efficiency of video coding. For this reason, formula (6) is used to characterize the coding efficiency of each technical solution as the number of frames increases.

[0090] (6) TFLOP (Tera Floating Point Operations) is a core metric for measuring the computing power of computing devices (such as GPUs and AI chips); Kpixel Per TFLOPs (Kilo-pixel Per TFLOPs) is the number of kilopixels per second per trillion floating-point operations, and is a core metric for measuring the hardware computing efficiency of image or video processing and computer vision models; H and W represent the height and width of the frame sequence, and T is the number of frames included in the frame sequence.

[0091] The frame sequence processing method FlexiVideo proposed in this application processes video content in a more dynamic way, thereby achieving a significant efficiency improvement. In this embodiment, FlexiVideo was evaluated on Video-MME (w / o sub.) under different input frame number settings. As the number of input frames increases, the similarity between adjacent frames gradually increases, enabling FlexiVideo to jointly encode more frames while maintaining high spatial resolution, and its performance is significantly better than Qwen2.5-VL.

[0092] like Figure 4 As shown, the efficiency metric described in formula (6) is significantly improved at high frame rate settings. This improvement stems from the ability to adaptively capture meaningful temporal changes while effectively suppressing redundant content. By adaptively dividing the video into several scene segments, aggregating static content within segments, and modeling dynamic changes between segments, FlexiVideo maintains higher spatial resolution and achieves efficient encoding while avoiding unnecessary calculations on redundant or static content. Compared to fixed encoding strategies such as Qwen2.5-VL, FlexiVideo demonstrates stronger adaptability in high frame rate scenarios, while also achieving higher efficiency and better video understanding performance.

[0093] Third Embodiment To evaluate the effectiveness of the dynamic modeling mechanism, in this embodiment, the frame sequence processing method FlexiVideo proposed in this application is compared with two comparative models that use the same initialization method.

[0094] like Figure 5 The experimental results shown indicate that the three models share the same language model, while the visual encoders employ multi-frame encoding windows with fixed temporal dimensions of 2 and 6, corresponding to smaller and larger temporal perception ranges, respectively. All experiments were conducted under completely identical training configurations and evaluated on MotionBench (8 fps) and MLVU (576 frames) to ensure fairness in the comparison.

[0095] Larger encoding windows exacerbate visual confusion, leading to performance degradation; while smaller encoding windows, although mitigating visual confusion, significantly increase computational overhead. In contrast, FlexiVideo adaptively adjusts its encoding window based on local temporal changes, effectively mitigating visual confusion while accurately modeling temporal dynamics. Therefore, FlexiVideo not only consistently outperforms traditional multi-frame fixed encoding methods in performance but also surpasses the computational efficiency of Qwen2.5-VL, which uses a 2-frame encoding window, fully demonstrating the practical advantages of the change-aware modeling approach proposed in this application for efficient video understanding.

[0096] Fourth embodiment Natural videos exhibit diverse temporal dynamics, and the deep semantics of events are often embedded in slowly evolving visual features. This semantic consistency, also known as smoothness in the feature space, is crucial for improving the model's video understanding capabilities. To further analyze the semantic consistency features of FlexiVideo, the frame sequence processing method proposed in this application, we compare it with Qwen2.5-VL to examine the continuity of visual semantics encoded in the feature space.

[0097] In this embodiment, a video segment containing two main events is randomly selected. This case consists of two consecutive events: the performance segment of rotating the baton, and the subsequent scoring segment by the judges. Figure 6 As shown, within a single event, both FlexiVideo and Qwen2.5-VL can generate highly consistent feature embeddings, exhibiting smooth feature transitions and stable semantic consistency within the scene.

[0098] However, when events switch, Qwen2.5-VL's fixed multi-frame encoding strategy fails to explicitly model inter-frame changes, performing indiscriminate joint encoding on frames with significant visual differences, thus causing visual confusion and ultimately leading to a hallucinatory model response. In contrast, FlexiVideo explicitly considers inter-frame differences during encoding, reducing the magnitude of visual changes within each time window through dynamic temporal segmentation. Even when events switch, this strategy effectively mitigates visual confusion by emphasizing semantic consistency in the temporal dimension to smooth out large visual differences.

[0099] This dynamic segmentation-based modeling approach ensures more coherent and stable encoding of the entire video sequence in the feature space, especially near event boundaries. Therefore, FlexiVideo exhibits stronger robustness in temporal dynamic modeling, generating more accurate and context-consistent feature representations, thus demonstrating its overall advantage in video understanding tasks.

[0100] The above description primarily focuses on the methodological aspects of the embodiments of this application. Those skilled in the art should readily recognize that, based on the operations or steps described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Those skilled in the art can implement the described functionality in different ways for each specific operation or method, and such implementations should not be considered beyond the scope of this application.

[0101] The apparatus embodiments of this application are described below. For details not described in the apparatus embodiments of this application, please refer to the method embodiments of this application.

[0102] Figure 7 A block diagram of a frame sequence processing apparatus according to an example embodiment of this application is shown, such as... Figure 7 The processing apparatus shown includes a frame sequence grouping unit 701, a weight adjustment unit 703, and a feature extraction unit 705. The frame sequence grouping unit 701 groups the input frame sequence according to inter-frame differences to obtain multiple sub-segments; the weight adjustment unit 703 adjusts the preset weights in the visual encoder in a temporal dimension using the length of each sub-segment to obtain adjusted encoding weights; and the feature extraction unit 705 uses the adjusted encoding weights to encode each sub-segment into a fixed number of feature units through the visual encoder.

[0103] Figure 8 An electronic device according to an exemplary embodiment of this application is shown. Reference is made below. Figure 8 To describe an electronic device 200 according to this embodiment of the present application. Figure 8 The electronic device 200 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0104] like Figure 8 As shown, the electronic device 200 is presented in the form of a general-purpose computing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including storage unit 220 and processing unit 210), a display unit 240, etc.

[0105] The storage unit stores program code that can be executed by the processing unit 210, causing the processing unit 210 to perform the methods described in this specification according to various exemplary embodiments of this application. For example, the processing unit 210 can perform the methods described above.

[0106] Storage unit 220 may include readable media in the form of volatile storage units, such as random access memory (RAM) 2201 and / or cache memory 2202, and may further include read-only memory (ROM) 2203.

[0107] Storage unit 220 may also include a program / utility 2204 having a set (at least one) program module 2205, such program module 2205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0108] Bus 230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0109] Electronic device 200 can also communicate with one or more external devices 300 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 200, and / or with any device that enables electronic device 200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 250. Furthermore, electronic device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 260. Network adapter 260 can communicate with other modules of electronic device 200 via bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0110] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. The technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of this application.

[0111] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0112] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0113] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0114] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the aforementioned functions.

[0115] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0116] According to an embodiment of this application, a computer program is proposed, including a computer program or instructions, which, when executed by a processor, can perform the methods described above.

[0117] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, and on the specific implementation methods and application scope of this application, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for processing frame sequences, characterized in that, include: The input frame sequence is grouped according to the inter-frame differences to obtain multiple sub-segments; The preset weights in the visual encoder are adjusted in the time dimension using the length of each sub-segment to obtain the adjusted encoding weights. Using the adjusted encoding weights, each sub-segment is encoded into a fixed number of feature units by the visual encoder.

2. The processing method according to claim 1, characterized in that, Before grouping the input frame sequence into multiple sub-segments based on inter-frame differences, the process also includes: The initial weights in the visual encoder are periodically expanded, and the expanded weights are used as the preset weights.

3. The processing method according to claim 2, characterized in that, The initial weights in the visual encoder are periodically expanded using the following formula: Where r is the expansion ratio, , This indicates the spatial scale of the initial weights. This indicates the spatial scale after the periodic expansion. This represents the initial weights. The weight after period expansion is the preset weight.

4. The processing method according to claim 3, characterized in that, The input frame sequence is grouped according to inter-frame differences to obtain multiple sub-segments, including: Determine a reference frame in the frame sequence, wherein the reference frame is the first frame in the current sub-segment; Compare the pixel difference values ​​between the reference frame and subsequent frames of the reference frame; When the pixel difference value is greater than a preset threshold, the frame preceding the frame in which the current pixel difference value is calculated is taken as the last frame of the current sub-segment. Using the frame from which the current pixel difference value is calculated as a reference frame, grouping continues until all sub-segments of the frame sequence are obtained.

5. The processing method according to claim 3, characterized in that, The preset weights in the visual encoder are adjusted in the time dimension using the length of each sub-segment to obtain the adjusted encoding weights, including: The time dimension corresponding to the adjusted coding weight is determined by using the length of each sub-segment; The time difference matrix is ​​determined using the time dimension and the initial time dimension corresponding to the preset weight; Calculate the pseudo-inverse matrix corresponding to the time difference matrix; The adjusted encoding weights are obtained using the pseudo-inverse matrix and the preset weights.

6. The processing method according to claim 5, characterized in that, The adjusted encoding weights are obtained using the pseudo-inverse matrix and the preset weights, and are achieved by the following formula: in, The weight value corresponding to each spatiotemporal video block in the preset weights, The weight values ​​for each spatiotemporal video block in the adjusted encoding weights. , Let B be the pseudo-inverse matrix, and let B be the time difference matrix.

7. The processing method according to claim 6, characterized in that, The number of feature units corresponding to the frame sequence is shown in the following formula: Where N is the number of feature units corresponding to the frame sequence. and These represent the width and height of each frame in the frame sequence, respectively. K represents the spatial size of each spatiotemporal video block, and K represents the number of sub-segments included in the frame sequence.

8. The processing method according to claim 7, characterized in that, Before grouping the input frame sequence into multiple sub-segments based on inter-frame differences, the process also includes: The images are copied to obtain a frame sequence comprising multiple images.

9. The processing method according to claim 1, characterized in that, It also includes calculating coding efficiency using the following formula: Where TFLOP is a trillion floating-point operations, H and W represent the height and width of the frame sequence, and T is the number of frames included in the frame sequence.

10. A frame sequence processing apparatus, characterized in that, include: The frame sequence grouping unit is used to group the input frame sequence according to the inter-frame differences to obtain multiple sub-segments; The weight adjustment unit is used to adjust the preset weights in the visual encoder in the time dimension using the length of each sub-segment to obtain the adjusted encoding weights. The feature extraction unit is used to encode each sub-segment into a fixed number of feature units by the visual encoder using the adjusted encoding weights.

11. An electronic device, characterized in that, include: processor; Memory, used to store computer programs; When the computer program is executed by the processor, the processor performs the method as described in any one of claims 1-9.

12. A non-transitory computer-readable storage medium having stored thereon computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-9.

Images