A shallow pruning compression method based on a dynamic scene large model

Abstract

The present application relates to the technical field of computer vision, and provides a shallow pruning compression method based on a dynamic scene large model, which comprises the following steps: obtaining a visual token set of a dynamic scene video image and performing segmented perception causal mask processing to obtain a content-perception semantic segmented visual token stream; calculating attention scores of the segmented visual token stream through mask constraint attention; calculating systematic bias introduced by position encoding, and performing position bias calibration on the attention scores through the systematic bias to obtain calibrated attention scores; performing preliminary pruning compression according to the calibrated attention scores to obtain a register token set; and performing register token deduplication on the register token set to obtain pruned and compressed visual tokens. The present application solves the defects that the performance of a shallow decoder layer based on an attention-based compression method is severely reduced and robustness cannot be maintained at a high compression rate, and makes it possible to perform efficient pruning at the shallow layer of a model.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a shallow pruning and compression method based on a large dynamic scene model. Background Technology

[0002] Visual Large Models (VLLMs) understand and analyze dynamic scene content based on text cues. The input dynamic scene stream is converted into a large number of visual tokens by a visual encoder, and these tokens are processed during the model's pre-filling phase. This process incurs extremely high computational overhead. To mitigate this problem, pruning and compression strategies can be used to reduce the number of visual tokens.

[0003] Early pruning compression methods were primarily guided by visual cues, such as leveraging the sensitivity of the visual encoder itself to identify and retain visually dominant tokens, or by merging visually similar content that is temporally or spatially similar. These methods operate purely on the similarity of visual features and lack cross-modal semantic alignment relevant to text queries.

[0004] Attention-based pruning compression strategies leverage the model's inherent cross-modal reasoning ability (i.e., attention scores) to estimate token importance, thereby aligning the pruning process with semantic relevance. However, existing attention-based pruning methods still have the following limitations: poor performance of shallow pruning: when performed in shallow layers of the model, such as layer 1, it leads to a significant performance drop, especially under high compression ratios; key factors are ignored, and the root causes of shallow pruning failure include: Insufficient information interaction: In the shallow layer, the information interaction between visual tokens is insufficient, leading to "attention collapse", where some semantically unrelated tokens, such as background tokens, receive abnormally high attention scores.

[0005] Positional encoding bias: The model's positional encoding introduces a "positional bias" that is unrelated to the content, causing the model to unreasonably focus on tokens at the end of the sequence rather than tokens that are semantically important.

[0006] Importance-Redundancy Dilemma: Relying solely on attention scores leads to models retaining a large number of highly similar "register tokens," such as those from adjacent or similar backgrounds. These redundant tokens "crowd out" tokens with global contextual information that should be retained at high compression rates, resulting in performance degradation. Therefore, existing technologies urgently need a new pruning compression framework to overcome the aforementioned shallow pruning obstacles and achieve high-accuracy dynamic scene understanding even at high compression rates. Summary of the Invention

[0007] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides a shallow pruning compression method based on a large dynamic scene model, which solves the defects of attention-based compression methods, such as severe performance degradation in shallow decoder layers and inability to maintain robustness at high compression rates, making it possible to perform efficient pruning at the shallow layers of the model.

[0008] This invention provides a shallow pruning and compression method based on a large dynamic scene model, comprising: S1: Obtain the visual token set of dynamic scene video images, perform segmented perceptual causal masking on the visual token set, and obtain a content-aware semantic segmented visual token stream. S2: Calculate the attention score of the segmented visual token stream by masking the attention constraint; S3: Calculate the systematic bias introduced by the position coding, and use the systematic bias to calibrate the attention score to obtain the calibrated attention score; S4: Perform preliminary pruning and compression based on the calibrated attention scores to obtain the register token set; S5: Perform register token deduplication on the register token set to obtain pruned and compressed visual tokens.

[0009] Furthermore, step S1 includes: S11: Obtain the frame-level average token set by averaging the visual tokens from the same video frame; S12: Calculate the insertion segment boundary by evaluating the cosine similarity between consecutive frames in the average token set; S13: By inserting segment boundaries, the video token set is divided into multiple continuous non-overlapping segments to obtain a content-aware semantic segmented visual token stream.

[0010] Furthermore, step S3 includes: S31: Input the dynamic scene video composed of black frames into the large visual model, calculate the attention score deviation between the visual token and the last text token, and obtain the systematic bias; S32: Subtract systematic bias from the attention score to obtain the calibrated attention score.

[0011] Furthermore, in step S4, the Top-K visual tokens with the highest calibrated attention scores in the dynamic scene sequence are selected to form a register token set.

[0012] Furthermore, step S5 includes: S51: Calculate the cosine similarity between the non-registered token and the tokens in the registered token set. If the similarity is greater than the first similarity threshold, then classify the non-registered token into the nearest set of similar registered tokens. If the similarity is less than or equal to the first similarity threshold, then a modified non-register token set is formed; S52: Calculate the cosine similarity between the first and second tokens in the register token set; S53: If the cosine similarity between the first token and the second token is greater than the second similarity threshold, then merge the second token into the first token; If the cosine similarity between the first token and the second token is less than or equal to the second similarity threshold, then the cosine similarity between the second token and the third token is calculated. S54: Iteratively apply step S53 to deduplicate all tokens in the register token set; S55: Cluster the non-register tokens in the modified non-register token set, and supplement the register token set with the obtained clusters to obtain the pruned and compressed visual token set.

[0013] Furthermore, the clustering of non-registered tokens in the modified non-registered token set in step S55 includes: S551: Correct the non-registered token set to divide the tokens into multiple clusters, and take the cluster of the first token as the initial central cluster; S552: Calculate the cosine similarity between the cluster of the second token and the initial central cluster; If the cosine similarity between the cluster of the second token and the initial center cluster is greater than the third similarity threshold, then the cluster of the second token is added to the initial center cluster. If the cosine similarity between the second token's cluster and the initial central cluster is less than or equal to the third similarity threshold, then a new central cluster is created; S553: ​​Iteratively apply S552 to assign tokens in the modified non-register token set to the corresponding clusters, thus completing the clustering of non-register tokens in the modified non-register token set.

[0014] Furthermore, the diversity score of each central cluster is calculated, and the cluster center with the highest diversity score is selected as the supplementary register token.

[0015] This invention also provides a shallow pruning compression system based on a large dynamic scene model, used to execute the aforementioned shallow pruning compression method based on a large dynamic scene model, comprising: The acquisition module acquires a visual token set of dynamic scene video images, performs segmented perceptual causal masking on the visual token set, and obtains a content-aware semantic segmented visual token stream. The attention module calculates the attention score of the segmented visual token stream by masking the attention constraint. A deviation calibration module calculates the systematic deviation introduced by the position coding, and performs position deviation calibration on the attention score based on the systematic deviation to obtain the calibrated attention score. A preliminary pruning module performs preliminary pruning and compression based on the calibrated attention score to obtain a register token set; The deduplication module performs register token deduplication on the register token set to obtain pruned and compressed visual tokens.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a shallow pruning and compression method based on a dynamic scene large model as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a shallow pruning and compression method based on a dynamic scene large model as described above.

[0018] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: Achieving efficient shallow pruning compression and breaking deep layer dependencies: Existing pruning compression methods typically can only be performed at deep layers due to the unreliability of shallow attention, resulting in limited computational savings. This invention achieves aggressive pruning compression in the first layer decoder of a large visual language model by calibrating and enhancing shallow signals, thereby greatly reducing the computational cost of the entire model. After bias calibration, the shallow attention distribution can exhibit semantic relevance and effectiveness comparable to that of deep layers.

[0019] Segmented Causal Masking Solves Attention Collapse: Addressing the attention distraction problem caused by "high-norm, low-semantic" tokens commonly found in shallow layers, this invention divides dynamic scenes into semantically consistent segments and constrains the attention scope, forcing the model to focus on locally relevant information. This not only preserves causal decoding characteristics but also significantly enhances the quality of feature interactions in shallow layers.

[0020] Positional bias calibration ensures sorting reliability: This invention effectively eliminates the long-term decay bias introduced by positional encoding (such as RoPE). This allows attention-based token sorting to truly reflect semantic importance, avoiding the model erroneously discarding key information from the beginning or middle of dynamic scenes.

[0021] Register tokens balance importance and diversity: To address the problem that high-scoring tokens often have a lot of spatiotemporal redundancy, this invention uses a three-stage refining mechanism of "pre-filtering-deduplication-post-filling" to remove duplicate information and supplement it with representative diversity tokens while maintaining a fixed compression rate, thereby improving the overall entropy value of the retained information.

[0022] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a shallow pruning and compression method based on a large dynamic scene model provided by the present invention.

[0025] Figure 2 The overall flowchart of the framework of a shallow pruning and compression method based on a large dynamic scene model provided by the present invention is shown.

[0026] Figure 3 This is a schematic diagram comparing hierarchical attention scores according to an embodiment of the present invention.

[0027] Figure 4 This is a comparison chart of the position deviation calibration effect in an embodiment of the present invention.

[0028] Figure 5 This is a schematic diagram of a shallow pruning and compression system based on a large dynamic scene model provided by the present invention.

[0029] Figure 6 A block diagram of the electronic device provided by the present invention.

[0030] Figure label: 101. Acquisition Module; 102. Attention Module; 103. Deviation Calibration Module; 104. Preliminary Pruning Module; 105. Deduplication Module; 201. Processor; 202. Communication Bus; 203. Communication Interface; 204. Memory. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0032] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0033] The following is combined with Figures 1 to 6 This invention describes a shallow pruning and compression method based on a large dynamic scene model.

[0034] like Figure 1 and Figure 2 As shown, a shallow pruning and compression method based on a large dynamic scene model includes: S1: Obtain the visual token set of dynamic scene video images, perform segmented perceptual causal masking on the visual token set, and obtain a content-aware semantic segmented visual token stream. Dynamic scene video images include video images from edge devices such as mobile phones and smart glasses.

[0035] The dynamic scene stream is converted into a large number of visual tokens by a visual encoder, and the text input text segmenter that describes the video obtains text tokens; S11: Obtain the frame-level average token set by averaging the visual tokens from the same video frame; Segmented perceptual causal masking (SegM) such as Figure 2 As shown in Figure (a), given the set of visual tokens for the input. Average pooling is applied to all tokens from the same frame to obtain a frame-level average token set. , ,in, For the first Average tokens per frame; S12: Calculate the insertion segment boundary by evaluating the cosine similarity between consecutive frames in the average token set; In order to Divide into semantically consistent segments, and evaluate The cosine similarity between consecutive frames is used to insert segment boundaries, and the calculation expression is: in, For the inserted segment boundary, For the first Average tokens per frame For similarity threshold, To calculate cosine similarity, for Size; When the similarity between two adjacent frames is lower than the set threshold, it is considered that a semantic change has occurred. Therefore, a segmentation boundary point is inserted at this point to divide the entire dynamic scene sequence into several segments with consistent content. S13: By inserting segment boundaries, the video token set is divided into multiple continuous non-overlapping segments to obtain a content-aware semantic segmented visual token stream.

[0036] Inserted segment boundary Visual token set Divide into multiple continuous, non-overlapping segments. , , For the first Each segment; S2: Calculate the attention score of the segmented visual token stream by masking the attention constraint; Based on segmentation Construct a "segmented perception visual mask" with a block diagonal matrix structure, such as... Figure 2 As shown in Figure (a), during masked attention computation, visual tokens within the same segment remain mutually visible, while tokens from different segments are completely isolated. After applying the mask, one visual token is visible to other tokens. Attention score It follows the following distribution: in, For the first OK The effective attention score of the column, For the first OK The effective attention score of the column, This is the length of the segment.

[0037] The attention that was originally scattered across all segments will be "squeezed" into a single segment. This localization mask effectively reduces the scope of softmax operation, concentrating attention weights within semantically consistent regions, forming smaller triangular attention regions, and enhancing the information flow of visual feature aggregation.

[0038] The input dynamic scene visual tokens are processed by segment-aware causal masking, which automatically divides the token stream into content-aware semantic segments and constructs masks to constrain attention to be performed within each segment in order to enhance local information interaction.

[0039] S3: Calculate the systematic bias introduced by the position coding, and use the systematic bias to calibrate the attention score to obtain the calibrated attention score; Position coding calibration (PosC) such as Figure 2 As shown in Figure (b), Positional Encodings (PE) provide token order information for the Visual Large Model (VLLM), but also introduce inherent positional biases that tend to favor tokens at specific positions. A typical example is the long-term decay problem in Rotary Positional Encoding (RoPE), given a token index... and The attention score varies with the relative token distance. Increases and decreases. In the context of dynamic scene token pruning and compression, existing methods typically refer to the last text token. Visual tokens Attention score This is used to identify important tokens, and it is assumed that tokens with higher scores are more relevant to queries.

[0040] This positional bias can severely distort the distribution of attention scores and may lead to misunderstandings of the mechanism. For example... Figure 4 As shown in Figure (a), in the benchmark test, at a retention rate of 20%, most of the top-ranked tokens (orange area) clustered near the last few frames of the dynamic scene, which clearly reflects positional bias rather than true semantic relevance. Figure 4 Figure (b) further illustrates a case where the problem targets events in the middle of a dynamic scene, but attention is still biased towards later frames, resulting in semantically important regions (blue areas) not being adequately addressed.

[0041] This invention calibrates the PE bias before pruning and compression. Considering the diversity of PE techniques (such as APE and RPE) and the possible use of multiple PE strategies in VLLM, this invention proposes a simple, content-agnostic, and PE-type-independent method to estimate the inherent positional coding bias in VLLM.

[0042] Specifically, S31: Input the dynamic scene video composed of black frames into the visual large model, calculate the attention score deviation between the visual token and the last text token, and obtain the systematic bias; All visual tokens are homogeneous and contain no information. Therefore, the visual tokens computed by the model... and the last text token Any deviation in attention scores directly reflects the bias introduced by positional encoding; The larger the value, the stronger the bias, and the more it needs to be corrected. The more compensation you provide, the less you provide; S32: Subtract systematic bias from the attention score to obtain the calibrated attention score.

[0043] To remove this bias, this invention subtracts it from the attention score, calculated as follows: in, The calibrated attention score, To control the hyperparameters of the compensation intensity; The compensation is used only for the "ranking and pruning" of tokens, while the value of each token itself remains unchanged. This bias estimation requires only one lightweight inference using unbiased input and is performed once in each pruning step.

[0044] Position bias calibration of attention scores computed by shallow decoders allows attention scores to more accurately reflect the semantic importance of tokens by estimating and removing content-independent systematic biases introduced by position encoding (PE).

[0045] By inputting a dynamic scene devoid of any semantic information (e.g., completely black), a "baseline bias" score, purely due to positional encoding, is obtained. When processing real-world dynamic scenes, this "baseline bias" is subtracted from the raw attention score calculated by the model, resulting in a more accurate semantic importance score unaffected by positional interference. After bias calibration, shallow attention distributions can exhibit semantic relevance and effectiveness comparable to deeper ones. Figure 3 As shown.

[0046] S4: Perform preliminary pruning and compression based on the calibrated attention scores to obtain the register token set; Select the top-K visual tokens with the highest calibrated attention scores in the dynamic scene sequence to form a register token set; The reserved tokens are distributed across different segments. This constitutes a compact set of register tokens. ; S5: Perform register token deduplication on the register token set to obtain pruned and compressed visual tokens; Although highly attentional tokens are preserved after pruning and compression, many tokens remain highly similar in both spatial and temporal dimensions. To further compress the representation, this invention introduces a three-stage post-pruning refinement process, which is performed on register tokens within each segment to segment them. Register tokens within For example, that is The process includes: Register token deduplication (RegD) such as Figure 2 As shown in Figure (c); S51: Calculate the cosine similarity between the non-registered token and the tokens in the registered token set. If the similarity is greater than the first similarity threshold, the non-registered token is assigned to the nearest similar registered token set. If the similarity is less than or equal to the first similarity threshold, a modified non-registered token set is formed. In the "non-register tokens" (i.e., tokens that have been initially pruned), and... Any token within the string whose similarity exceeds the first similarity threshold The tokens that are similar to the nearest registered token are assigned to the same register; the remaining non-registered tokens are denoted as […]. This information is reserved for subsequent steps.

[0047] S52: Calculate the cosine similarity between the first and second tokens in the register token set; S53: If the cosine similarity between the first token and the second token is greater than the second similarity threshold, then merge the second token into the first token; If the cosine similarity between the first token and the second token is less than or equal to the second similarity threshold, then the cosine similarity between the second token and the third token is calculated. S54: Iteratively apply step S53 to deduplicate all tokens in the register token set; Register token The system performs iterative merging, sequentially comparing adjacent register tokens. If a token's similarity to its predecessor exceeds a second similarity threshold... It will be merged into the previous token; otherwise, it will become the new comparison benchmark. This process continues until the end of the sequence, resulting in a pruned, compressed, and deduplicated set of register tokens. .

[0048] Redundancy in register tokens is reduced by merging similar tokens.

[0049] S55: Cluster the non-register tokens in the modified non-register token set, and supplement the register token set with the obtained clusters to obtain the pruned and compressed visual token set.

[0050] S551: Correct the non-registered token set to divide the tokens into multiple clusters, and take the cluster of the first token as the initial central cluster; S552: Calculate the cosine similarity between the cluster of the second token and the initial central cluster; If the cosine similarity between the cluster of the second token and the initial center cluster is greater than the third similarity threshold, then the cluster of the second token is added to the initial center cluster. If the cosine similarity between the second token's cluster and the initial central cluster is less than or equal to the third similarity threshold, then a new central cluster is created; S553: ​​Iteratively apply S552 to assign tokens in the modified non-register token set to the corresponding clusters, thus completing the clustering of non-register tokens in the modified non-register token set; In order to maintain a fixed number of register tokens at a preset compression ratio, the present invention... Non-register tokens are clustered and supplemented with selected cluster centers. ,from Starting with the first token in the cluster, perform iterative clustering: if the similarity between a subsequent token and the current cluster exceeds a third similarity threshold... If the cluster is found to be in the cluster, then join the cluster; otherwise, start a new cluster.

[0051] Calculate the diversity score for each central cluster, and select the cluster center with the highest diversity score as the supplementary register token; the expression for calculating the diversity score is: in, For the first The diversity score of each central cluster, For the first Register tokens, for Size, for Size, To balance hyperparameters, For the central cluster set, It is the cluster center.

[0052] This score considers both the average similarity to existing register tokens (negative correlation) and the cluster size (positive correlation). It has the highest... Cluster center of score The selected tokens are the register tokens. "Non-reserved tokens" that are highly similar to the reserved tokens are merged; then redundant tokens already present in the "reserved tokens" set are iteratively merged; finally, to maintain the compression ratio, the most diverse representatives are selected from the remaining "non-reserved tokens" through clustering and added back to the "reserved tokens" set, balancing the importance and diversity of the token set.

[0053] like Figure 5 As shown, a shallow pruning compression system based on a large dynamic scene model is used to execute a shallow pruning compression method based on a large dynamic scene model, including: The acquisition module 101 acquires the visual token set of the dynamic scene video image, performs segmented perceptual causal masking on the visual token set, and obtains a content-aware semantic segmented visual token stream. Attention module 102 calculates the attention score of the segmented visual token stream by masking the attention constraint; The deviation calibration module 103 calculates the systematic deviation introduced by the position coding, and performs position deviation calibration on the attention score through the systematic deviation to obtain the calibrated attention score; The preliminary pruning module 104 performs preliminary pruning and compression based on the calibrated attention score to obtain a register token set; The deduplication module 105 performs register token deduplication on the register token set to obtain pruned and compressed visual tokens.

[0054] Through the collaborative work of the aforementioned modules, efficient shallow pruning compression is achieved, breaking the dependence on deep layers: existing pruning compression methods typically can only be performed at deep layers due to the unreliability of shallow attention, resulting in limited computational savings. This invention achieves aggressive pruning compression in the first-layer decoder of a large visual language model by calibrating and enhancing shallow signals, thereby greatly reducing the computational cost of the entire model. After bias calibration, the shallow attention distribution can exhibit semantic relevance and effectiveness comparable to that of deep layers.

[0055] Segmented Causal Masking Solves Attention Collapse: Addressing the attention distraction problem caused by "high-norm, low-semantic" tokens commonly found in shallow layers, this invention divides dynamic scenes into semantically consistent segments and constrains the attention scope, forcing the model to focus on locally relevant information. This not only preserves causal decoding characteristics but also significantly enhances the quality of feature interactions in shallow layers.

[0056] Positional bias calibration ensures sorting reliability: This invention effectively eliminates the long-term decay bias introduced by positional encoding (such as RoPE). This allows attention-based token sorting to truly reflect semantic importance, avoiding the model erroneously discarding key information from the beginning or middle of dynamic scenes.

[0057] Register tokens balance importance and diversity: To address the problem that high-scoring tokens often have a lot of spatiotemporal redundancy, this invention uses a three-stage refining mechanism of "pre-filtering-deduplication-post-filling" to remove duplicate information and supplement it with representative diversity tokens while maintaining a fixed compression rate, thereby improving the overall entropy value of the retained information.

[0058] This invention addresses the shortcomings of attention-based compression methods, such as severe performance degradation in shallow decoder layers and inability to maintain robustness at high compression rates, making efficient pruning of shallow layers of the model possible.

[0059] Table 1 shows a comparison of the present invention (ShaRP) with existing methods on multiple evaluation metrics in the dynamic scene understanding task using the LLaVAOneVision model. In Table 1, MVBench is a general video understanding benchmark, LongVideoBench is a video-language interaction understanding benchmark for long videos, MLVU is a multi-task long video understanding benchmark, the score is the average score, % is the performance retention rate relative to the original model, TFLOPS is the computational cost, and VideoMME is a comprehensive evaluation benchmark for video multimodal large language models. As shown in Table 1, under the same computational cost, the present invention significantly outperforms methods such as FastV, PDrop, Feather, and FrameFusion.

[0060] Table 1. Comparison of the present invention with existing methods on multiple evaluation metrics in the dynamic scene understanding task using the LLaVAOneVision model.

[0061] Table 2 shows the comparison of the cross-backbone network generalization ability of the present invention on the Qwen2-VL model. The results show that the present invention still maintains a leading advantage under different model structures.

[0062] Table 2 Comparison of the cross-backbone network generalization ability of the present invention on the Qwen2-VL model

[0063] Table 3 illustrates the inference efficiency of this invention. A 3x improvement in pre-filling speed is achieved while maintaining accuracy by less than 3% (97.2% performance retention). In Table 3, Attn. represents the attention module, Fltr. the filtering module, Dedpl. the deep processing module, Fil. the feature integration module, LLM Fw. the time consumed in the forward computation of the large language model, and Total the total pre-filling time.

[0064] Table 3. Reasoning efficiency of the present invention

[0065] To quantitatively evaluate the beneficial effects of this invention, it follows several benchmarks widely used in dynamic scene understanding tasks, including MVBemach, Long VideoBench, MLVU, and VideoMME. This invention primarily employs two core metrics: Average Score / Accuracy and Total FLOPs (TFLOPs). Average Score measures the model's ability to understand dynamic scene content after pruning and compression; TFLOPs measures the actual computational efficiency improvement brought about by pruning and compression. Tables (as shown in Tables 1 and 2) detail the comparison between this invention and existing pruning and compression methods (such as FastV, PDrop, Feather, etc.) at different pruning and compression ratios (R). The results show that this invention significantly outperforms all published pruning and compression methods across all pruning and compression ratios and all benchmark tests. Particularly at high compression ratios (e.g., R=14.0%), this invention maintains extremely high accuracy (97.2%), while other methods have experienced performance degradation. This invention presents the first training-free framework that effectively solves the problem of shallow pruning compression. In dynamic scene description tasks, traditional methods (FastV, Feather) lose crucial visual information due to incorrect pruning compression (e.g., affected by positional bias), leading to incorrect responses. In contrast, this invention accurately preserves the core semantic tokens in dynamic scenes, generating complete and accurate descriptions. Through quantitative and qualitative comparisons with different methods, the present invention demonstrates significant improvements in both maintaining high accuracy in dynamic scene understanding and achieving efficient pruning and compression (up to 3.0 times), fully showcasing the effectiveness and advancement of the present invention.

[0066] Figure 6 An example is a block diagram of an electronic device, such as... Figure 6As shown, the electronic device may include a processor 201, a communication interface 203, a memory 204, and a communication bus 202. The processor 201, communication interface 203, and memory 204 communicate with each other via the communication bus 202. The processor 201 can call logical instructions from the memory 204 to execute a shallow pruning compression method based on a large dynamic scene model.

[0067] Furthermore, the logical instructions in the aforementioned memory 204 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0068] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute a shallow pruning compression method based on a dynamic scene large model provided by the above methods.

[0069] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a shallow pruning compression method based on a large dynamic scene model provided by the above methods.

[0070] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A shallow pruning and compression method based on a large dynamic scene model, characterized in that, include: S1: Obtain the visual token set of dynamic scene video images, perform segmented perceptual causal masking on the visual token set, and obtain a content-aware semantic segmented visual token stream. S11: Obtain the frame-level average token set by averaging the visual tokens from the same video frame; S12: Calculate the insertion segment boundary by evaluating the cosine similarity between consecutive frames in the average token set; S13: By inserting segment boundaries, the video token set is divided into multiple continuous non-overlapping segments to obtain a content-aware semantic segmented visual token stream. S2: Calculate the attention score of the segmented visual token stream by masking the attention constraint; S3: Calculate the systematic bias introduced by the position coding, and use the systematic bias to calibrate the attention score to obtain the calibrated attention score; S31: Input the dynamic scene video composed of black frames into the large visual model, calculate the attention score deviation between the visual token and the last text token, and obtain the systematic bias; S32: Subtract systematic bias from the attention score to obtain the calibrated attention score; S4: Perform preliminary pruning and compression based on the calibrated attention scores to obtain the register token set; S5: Perform register token deduplication on the register token set to obtain pruned and compressed visual tokens; S51: Calculate the cosine similarity between the non-registered token and the tokens in the registered token set. If the similarity is greater than the first similarity threshold, then classify the non-registered token into the nearest set of similar registered tokens. If the similarity is less than or equal to the first similarity threshold, then a modified non-register token set is formed; S52: Calculate the cosine similarity between the first and second tokens in the register token set; S53: If the cosine similarity between the first token and the second token is greater than the second similarity threshold, then merge the second token into the first token; If the cosine similarity between the first token and the second token is less than or equal to the second similarity threshold, then the cosine similarity between the second token and the third token is calculated. S54: Iteratively apply step S53 to deduplicate all tokens in the register token set; S55: Cluster the non-register tokens in the modified non-register token set, and supplement the register token set with the obtained clusters to obtain the pruned and compressed visual token set.

2. The shallow pruning and compression method based on a large dynamic scene model according to claim 1, characterized in that, In step S4, the top-K visual tokens with the highest calibrated attention scores in the dynamic scene sequence are selected to form a register token set.

3. The shallow pruning and compression method based on a large dynamic scene model according to claim 1, characterized in that, The S55 step involves clustering the non-registered tokens in the modified non-registered token set, including: S551: Correct the non-registered token set to divide the tokens into multiple clusters, and take the cluster of the first token as the initial central cluster; S552: Calculate the cosine similarity between the cluster of the second token and the initial central cluster; If the cosine similarity between the cluster of the second token and the initial center cluster is greater than the third similarity threshold, then the cluster of the second token is added to the initial center cluster. If the cosine similarity between the second token's cluster and the initial central cluster is less than or equal to the third similarity threshold, then a new central cluster is created; S553: ​​Iteratively apply S552 to assign tokens in the modified non-register token set to the corresponding clusters, thus completing the clustering of non-register tokens in the modified non-register token set.

4. The shallow pruning and compression method based on a large dynamic scene model according to claim 1, characterized in that, Calculate the diversity score for each central cluster, and select the cluster center with the highest diversity score as the supplementary register token.

5. A shallow pruning and compression system based on a large dynamic scene model, characterized in that, To perform a shallow pruning and compression method based on a large dynamic scene model as described in any one of claims 1 to 4, comprising: The acquisition module acquires a visual token set of dynamic scene video images, performs segmented perceptual causal masking on the visual token set, and obtains a content-aware semantic segmented visual token stream. The attention module calculates the attention score of the segmented visual token stream by masking the attention constraint. A deviation calibration module calculates the systematic deviation introduced by the position coding, and calibrates the attention score by the systematic deviation to obtain the calibrated attention score. A preliminary pruning module performs preliminary pruning and compression based on the calibrated attention score to obtain a register token set; The deduplication module performs register token deduplication on the register token set to obtain pruned and compressed visual tokens.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the shallow pruning and compression method based on a large dynamic scene model as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the shallow pruning and compression method based on a large dynamic scene model as described in any one of claims 1 to 4.

Images