Large language model pruning method, electronic device, and program product

Abstract

The present disclosure provides a large language model pruning method, an electronic device and a program product, and relates to the technical field of computers. The large language model pruning method comprises: performing visual coding on an input video to obtain first visual markers of each video frame image; based on the overlap rate between adjacent video frame images, the input video is divided into multiple video segments; the feature vector and the final pruning rate of each video segment are determined; the processing process of the large language model on the input video is divided into multiple stages to obtain second visual markers of each video frame image; based on the first similarity between the second visual markers of adjacent video frame images, the original kernel matrix for the determinant point process of each stage is constructed, and the sub-kernel matrix is extracted; for each sub-kernel matrix, the optimal visual marker subset is obtained by maximum a posteriori inference; the optimal visual marker subset is combined as the pruning result of each stage, and the pruning result is taken as the input of the next stage.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to a large language model pruning method, electronic equipment, and program products. Background Technology

[0002] In recent years, with the surge in multimedia data and advancements in artificial intelligence, multimodal large language models (MLLMs) have demonstrated powerful capabilities in cross-modal understanding tasks. Their typical architecture consists of a visual encoder, a large language model (LLM), and a projector connecting the two. However, the high computational overhead caused by massive visual tokens severely restricts the deployment efficiency of multimodal large language models in scenarios such as real-time video analysis, mobile interaction, and edge computing. Therefore, a visual token pruning technique is urgently needed to dynamically identify and remove redundant or low-information visual tokens without significantly impairing the semantic understanding capabilities of multimodal large language models. This would effectively reduce computational complexity, memory usage, and inference latency, improving the deployment efficiency and real-time response capabilities of multimodal large language models in resource-constrained scenarios. Summary of the Invention

[0003] This disclosure provides a method for pruning large language models, an electronic device, and a program product.

[0004] According to one aspect of this disclosure, a large language model pruning method is provided, comprising: visually encoding each video frame image contained in an input video to obtain a first visual label for each video frame image; calculating the overlap rate between adjacent video frame images based on the first visual labels of each video frame image; segmenting the input video into multiple video segments based on the overlap rate, such that the overlap rate between adjacent video frame images belonging to different video segments is less than an overlap rate threshold; determining the feature vector of each video segment based on the first visual labels of each video frame image contained in each video segment; determining the final pruning rate of each video segment based on the similarity between the feature vector of each video segment and the feature vector of other video segments; and applying the large language model to the input video... The video processing is divided into multiple stages, resulting in second visual labels for each video frame image output at each stage. Based on the first similarity between the second visual labels of adjacent video frame images and the correlation between the second visual labels of each video frame image and the last instruction label corresponding to the input prompt, a final kernel matrix for the matrix-deterministic process is constructed for each stage, and a sub-kernel matrix corresponding to each video segment is extracted from the final kernel matrix. For each sub-kernel matrix, the optimal subset of visual labels that satisfies the final pruning rate of the corresponding video segment is selected through maximum a posteriori inference. The optimal subset of visual labels corresponding to each sub-kernel matrix is ​​merged as the pruning result of each stage, and the pruning result of the previous stage is used as the input of the next stage.

[0005] According to one technical solution, segmenting the input video based on the overlap rate of adjacent video frames effectively identifies content abrupt change boundaries and avoids semantic breaks caused by cross-segment pruning. By dynamically allocating the final pruning rate based on the similarity of feature vectors between each video segment and other segments, adaptive adjustment of the pruning budget can be achieved, preserving more details in high-information-density segments and increasing compression in redundant segments. Constructing the final kernel matrix by fusing the first similarity and relevance scores allows for simultaneous consideration of temporal diversity and language task orientation during the matrix-based pruning process. Extracting the sub-kernel matrices corresponding to each video segment from the global kernel matrix (final kernel matrix) enables localized, structure-aware pruning, avoiding cross-segment interference. Using the pruning results of the previous stage as input for the next stage enables progressive, cascaded visual tagging refinement, eliminating redundancy layer by layer, significantly reducing computational load without accumulating information loss.

[0006] According to at least one embodiment of the present disclosure, calculating the overlap rate between adjacent video frame images based on a first visual marker of each video frame image includes: obtaining sub-visual markers for each spatial location contained in each video frame image based on the first visual marker of each video frame image; calculating a second similarity of adjacent video frame images at each spatial location based on the sub-visual markers; and taking the average value of the second similarity of each spatial location contained in the adjacent video frame images as the overlap rate between the adjacent video frame images.

[0007] According to the technical solution of this embodiment, the accuracy and robustness of the overlap rate estimation between adjacent video frame images can be improved.

[0008] According to at least one embodiment of this disclosure, the input video is divided into multiple video segments based on the overlap rate, including: creating video segment boundaries between adjacent video frame images when the overlap rate between adjacent video frame images is less than the overlap rate threshold; and segmenting the input video based on the video segment boundaries to obtain multiple video segments.

[0009] According to the technical solution of this embodiment, accurate and adaptive segmentation of video segments can be achieved.

[0010] According to at least one embodiment of this disclosure, after dividing the input video into multiple video segments, the method further includes: if there are K consecutive video frame images in the video segment whose second similarity at a first spatial location belongs to the K largest among the second similarities of the video frame images contained in the video segment at the first spatial location, then the average value of the first visual markers of the K consecutive video frame images at the first spatial location is used as a third visual marker, where K is greater than 1; and the first visual markers of the K consecutive video frame images at the first spatial location are replaced with the third visual markers.

[0011] According to the technical solution of this embodiment, local redundancy can be effectively compressed while preserving key spatial semantics, thereby further improving the compactness and robustness of visual representation.

[0012] According to at least one embodiment of this disclosure, determining the feature vector of each video segment based on the first visual markers of each video frame image contained in each video segment includes: performing average pooling processing on the first visual markers of each video frame image contained in the video segment to obtain the feature vector of the video segment.

[0013] According to the technical solution of this embodiment, a feature vector representing the overall visual characteristics of the corresponding video segment can be generated, effectively preserving the overall semantic information of the video segment and suppressing local noise.

[0014] According to at least one embodiment of this disclosure, determining the final pruning rate of each video segment based on the similarity between the feature vectors of each video segment and the feature vectors of other video segments includes: calculating the sum of the feature vectors of each video segment to the left of the video segment to obtain the left-side feature vector of the video segment; calculating the sum of the feature vectors of each video segment to the right of the video segment to obtain the right-side feature vector of the video segment; calculating a third similarity between the feature vector of the video segment and the right-side feature vector as the right-side similarity of the video segment; calculating the difference between 1 and the third similarity between the feature vector of the video segment and the left-side feature vector as the left-side similarity of the video segment; performing a weighted summation of the left-side similarity and the right-side similarity to obtain the expected pruning rate of the video segment; standardizing the expected pruning rate to obtain the standard pruning rate of the video segment; and summing the products of the base pruning rate and the standard pruning rate and the pruning rate deviation as the final pruning rate of the video segment.

[0015] According to the technical solution of this embodiment, by standardizing the expected pruning rate, scale differences between different video segments can be eliminated. By combining the standard pruning rate with the basic pruning rate and the deviation term, the compression intensity of each video segment can be precisely controlled. While preserving key temporal semantics and cross-segment diversity, this effectively avoids the problems of over-pruning of high-information segments or under-pruning of redundant segments.

[0016] According to at least one embodiment of this disclosure, standardizing the expected pruning rate to obtain the standard pruning rate of the video segment includes: calculating the average and standard deviation of the expected pruning rates of each video segment included in the video; and dividing the difference between the expected pruning rate of the video segment and the average value by the standard deviation to obtain the standard pruning rate.

[0017] According to the technical solution of this embodiment, the pruning rate distribution shift caused by the difference in content complexity between different video segments can be eliminated.

[0018] According to at least one embodiment of this disclosure, the method further includes: constructing an original kernel matrix based on a first similarity between second visual tags of adjacent video frame images; performing word segmentation on the input prompts of the large language model to obtain an instruction tag sequence; extracting the last instruction tag from the instruction tag sequence; calculating the correlation between the last instruction tag and the second visual tags of each video frame image; constructing a diagonal matrix of the same size as the original kernel matrix based on the correlation, wherein the rows and columns of the diagonal matrix correspond to the second visual tags of each video frame image included in the video, and the diagonal elements in the diagonal matrix represent the correlation; performing a first matrix multiplication operation on the diagonal matrix and the original kernel matrix of each stage to obtain an intermediate kernel matrix of each stage; performing a second matrix multiplication operation on the intermediate kernel matrix of each stage and the diagonal matrix to obtain a final kernel matrix of each stage; and extracting a sub-kernel matrix corresponding to each video segment from the final kernel matrix of each stage.

[0019] According to the technical solution of this embodiment, by bidirectionally weighting the original kernel matrix with a diagonal matrix constructed based on relevance, the sampling probability of video frame images that are highly related to the current input prompt in the matrix-determinant process can be enhanced, while suppressing the influence of video frame images.

[0020] According to at least one embodiment of this disclosure, the process of calculating the relevance includes: calculating the dot product between the last instruction marker and each second visual marker to obtain multiple dot product results; dividing each dot product result by the square root of the dimension of the second visual marker to obtain a standardized dot product result; performing softmax normalization on each standardized dot product result to obtain a first normalization result; and performing min-max normalization on the first normalization result to obtain the relevance corresponding to each second visual marker.

[0021] According to the technical solution of this embodiment, the large language model can more clearly distinguish which video frame images are most relevant to the user input prompts, so as to prioritize the retention of these "key frames" during pruning.

[0022] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, such that the processor performs a large language model pruning method according to any embodiment of this disclosure.

[0023] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement the large language model pruning method of any embodiment of this disclosure.

[0024] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a large language model pruning method according to any embodiment of this disclosure.

[0025] This disclosure implements a progressive pruning method with three levels of linkage: segment-level, frame-level, and hierarchical. It can significantly reduce the number of redundant visual tags in large language models while effectively preserving key semantic information and cross-frame diversity. It avoids the performance degradation caused by traditional pruning methods that ignore temporal structure or the flow of multimodal information. Attached Figure Description

[0026] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0027] Figure 1 This is a flowchart illustrating a large language model pruning method according to one embodiment of the present disclosure.

[0028] Figure 2 This is a flowchart illustrating an overlap rate calculation method according to one embodiment of the present disclosure.

[0029] Figure 3 This is a flowchart illustrating a method for determining the final pruning rate according to one embodiment of this disclosure.

[0030] Figure 4 This is a flowchart illustrating the method corresponding to step S170 of one embodiment of the present disclosure.

[0031] Figure 5 This is a schematic block diagram of a large language model pruning device according to one embodiment of the present disclosure.

[0032] Figure 6 This is a schematic structural block diagram of an electronic device employing a processor-based hardware implementation according to one embodiment of the present disclosure. Detailed Implementation

[0033] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0034] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] Existing pruning methods based on multimodal large language models mainly fall into two categories: The first category compresses visual tags between the input and output of the large language model (i.e., pruning visual tags between the visual encoder and the large language model). While this method can effectively reduce spatial redundancy within a single frame (within a single video frame), it often neglects temporal redundancy across frames (adjacent video frames). The second category compresses visual tags after they are input into the large language model (i.e., pruning visual tags within the large language model). In this second category, redundant (unimportant) visual tags are typically removed after a fixed network layer (such as a shallower network) of the large language model, failing to consider the multimodal information flow patterns within the large language model.

[0036] To address this, considering that the semantic representation of visual tokens gradually converges as the number of layers in the large language model network increases, and that the similarity distribution between visual tokens exhibits a clear centralization trend during forward propagation, this disclosure proposes the following technical solution. In this solution, firstly, visual encoding is performed on each video frame of the input video to obtain the first visual tag. Then, based on the overlap rate of visual tags between adjacent video frames, the input video is segmented into multiple semantically coherent video segments. Subsequently, feature vectors are calculated for each video segment, and the final pruning rate is dynamically allocated based on the similarity of the feature vectors between video segments. Next, the inference process of the large language model is divided into multiple stages. Based on the similarity of the second visual tags output by adjacent frames at each stage, a deterministic kernel matrix is ​​constructed, from which the sub-kernel matrices corresponding to each video segment are extracted. Finally, for each sub-kernel matrix, the optimal subset of visual tags satisfying the final pruning rate of that video segment is selected through maximum a posteriori inference. The pruning results from each stage are passed down level by level, with the output of the previous stage serving as the input for the next stage, achieving progressive pruning through a three-level linkage: segment-level, frame-level, and hierarchical level. This disclosure can significantly reduce the number of redundant visual tags in large language models while effectively preserving key semantic information and cross-frame diversity. It avoids the performance degradation caused by traditional pruning methods that ignore temporal structure or the flow of multimodal information.

[0037] To facilitate description and make the technical solutions of this disclosure easier to understand, the terminology of this disclosure will be explained before describing the technical solutions of this disclosure.

[0038] The Determinantal Point Process (DPP) is a probabilistic model used to sample a subset from a set that is both highly representative and internally diverse. In the DPP, the kernel matrix is ​​a positive semi-definite matrix used to characterize the relationships of "importance" and "similarity" among all candidate elements. The kernel matrix is ​​essentially a two-dimensional table where each row and column corresponds to an element (e.g., a visual marker). The diagonal positions in the table represent the importance of the element itself. A larger value indicates a more critical and worthwhile element to retain. The off-diagonal positions in the table represent the similarity between two different elements. A higher value indicates greater similarity and more redundant information between the two different elements.

[0039] Maximum A Posteriori (MAP) inference is an inference method that, given prior knowledge and observational data, seeks the model parameters or latent variable values ​​that are most likely to explain an observation. In the determinant point process, MAP is used to find the subset of second visual markers that is most likely to be selected by the determinant point process and exactly satisfies the specified final pruning rate across all subsets of second visual markers.

[0040] Average pooling is a commonly used feature aggregation operation that reduces data dimensionality, smooths features, and preserves key semantic information by calculating the arithmetic mean of input data (such as first visual labels) over local regions or across the entire dimension.

[0041] Softmax normalization is a mathematical function that transforms a set of real numbers into a probability distribution. It works by exponentially operating on each value and then dividing by the sum of all exponents, ensuring that the output is always non-negative and sums to 1.

[0042] Min-Max normalization (also known as minimum-maximum scaling) is a normalization method that linearly transforms data to a fixed interval (usually [0, 1]).

[0043] This disclosure applies to multimodal reasoning scenarios (such as inputs containing text and video), and is particularly suitable for efficient multimodal reasoning in resource-constrained scenarios (such as real-time video analysis, mobile interaction, and edge computing).

[0044] Figure 1 A schematic diagram illustrating the overall flow of a large language model pruning method according to one embodiment of this disclosure is shown. Figure 1 The method shown includes steps S110 to S190. This method can be executed by electronic devices such as mobile phones and tablets.

[0045] In step S110, visual encoding is performed on each video frame image contained in the input video to obtain the first visual label of each video frame image.

[0046] Visual encoding can be implemented using a visual encoder in a multimodal large language model (such as ViT). After the input video is fed into the visual encoder, the visual encoder sequentially encodes each video frame image contained in the input video and uses the visual tags of each output video frame image as the first visual tag.

[0047] As an example, the first visual label obtained after visual encoding of a video frame image is represented as follows: , where the subscript t represents the frame number (i.e., the t-th video frame image); Represents the set of real numbers; Indicates the number of pixels contained in the height of a video frame; Indicates the width of a video frame and the number of pixels contained within it; This indicates the dimension of the first visual marker.

[0048] In step S120, the overlap rate between adjacent video frame images is calculated based on the first visual marker of each video frame image.

[0049] Adjacent video frames may have similar content at certain spatial locations (pixels). By calculating the overlap rate between adjacent video frames, we can characterize the degree of similarity between them. This helps prevent highly similar adjacent segments from being segmented into different video segments during subsequent video segmentation.

[0050] In step S130, the input video is divided into multiple video segments based on the overlap rate, such that the overlap rate between adjacent video frames belonging to different video segments is less than an overlap rate threshold. Adaptively segmenting video segments based on inter-frame overlap rate can effectively identify scene transitions or sudden viewpoint changes, improving the accuracy of video analysis.

[0051] As one possible implementation, the input video is segmented into multiple video segments based on the overlap rate, including: creating video segment boundaries between adjacent video frames when the overlap rate between adjacent video frames is less than an overlap rate threshold. The input video is then segmented based on these video segment boundaries to obtain multiple video segments. This implementation can achieve accurate and adaptive segmentation of video segments.

[0052] In step S140, the feature vector of each video segment is determined based on the first visual marker of each video frame image contained in each video segment.

[0053] The feature vector of a video segment can be a feature vector that integrates the first visual markers of each video frame image contained in the video segment. As one possible implementation, determining the feature vector of each video segment based on the first visual markers of each video frame image contained in the video segment includes: performing average pooling on the first visual markers of each video frame image contained in the video segment to obtain the feature vector of the video segment. By performing average pooling on the first visual markers of each video frame image, a feature vector representing the overall visual characteristics of the corresponding video segment can be generated. The feature vector obtained through average pooling can effectively preserve the overall semantic information of the video segment and suppress local noise.

[0054] In step S150, the final pruning rate of each video segment is determined based on the similarity between the feature vectors of each video segment and the feature vectors of other video segments.

[0055] "Other video segments" of a given video segment refer to all video segments in the input video other than the given video segment, including all preceding video segments in time and all subsequent video segments in time.

[0056] The final pruning rate of each video segment can be determined by analyzing the similarity between its feature vectors and those of other video segments. A high similarity between a video segment and other segments indicates lower representativeness and higher redundancy, thus justifying a higher final pruning rate. Conversely, a low similarity between a video segment and other segments indicates higher representativeness and lower redundancy, thus justifying a lower final pruning rate.

[0057] The final pruning rate of video segments can be used to measure the diversity and representativeness of each video segment within the entire input video. This allows for prioritizing the retention of information-rich and unique video segments during subsequent pruning.

[0058] In step S160, the processing of the input video by the large language model is divided into multiple stages to obtain the second visual labels of each video frame image output by each stage.

[0059] The forward inference process of a large language model consists of multiple consecutive stages (processing stages), which can be different network layers (or groups of layers) within the large language model. After the input video is encoded by a visual encoder, the first visual labels of each video frame output by the visual encoder are input into the large language model. The first stage of the large language model processes the first visual labels of each video frame and uses the visual labels output by the first stage as the second visual labels for the corresponding video frames. The large language model then uses the second visual labels of the video frames output by the previous stage as input for the next stage.

[0060] In step S170, based on the first similarity between the second visual markers of adjacent video frame images and the correlation between the second visual markers of each video frame image and the last instruction marker corresponding to the input prompt, the final kernel matrix for each stage of the matrix-deterministic point process is constructed, and the sub-kernel matrix corresponding to each video segment is extracted from the final kernel matrix.

[0061] The final kernel matrix integrates the first similarity between second visual tags reflecting temporal redundancy and the relevance between each second visual tag and the last instruction tag reflecting the task (input cue) guidance importance. This approach simultaneously considers the diversity of video frames within a video segment and their semantic relevance to the current task. This dual-guidance mechanism allows the matrix-based pruning process to effectively remove temporally repetitive or static redundant information while prioritizing the retention of visual content highly relevant to user input cue. This significantly reduces the loss of key task information while substantially compressing the number of visual tags, improving the accuracy and robustness of the pruned model in downstream tasks.

[0062] The final kernel matrix contains sub-kernel matrices corresponding to each video segment. In the process of extracting the sub-kernel matrices for each video segment from the final kernel matrix, the corresponding rows and columns are extracted from the final kernel matrix based on the indices of the video frames contained in each video segment, thus forming the sub-kernel matrices for each video segment.

[0063] In step S180, for each sub-core matrix, the optimal visual tag subset that satisfies the final pruning rate of the corresponding video segment is selected by maximum a posteriori inference.

[0064] During the maximum a posteriori inference process for each sub-kernel matrix, the subset of second visual tags corresponding to each sub-kernel matrix (the set of second visual tags for each video frame image contained in the video segment corresponding to the sub-kernel matrix) can be filtered. The subset of second visual tags that is most likely to be selected by the determinant point process and exactly satisfies the specified final pruning rate is retained as the optimal visual tag subset. The optimal visual tag subset obtained by filtering has been optimized in terms of quantity compared to the original second visual tag subset, removing redundant or second visual tags that are not highly relevant to the current input prompt.

[0065] In step S190, the optimal visual tag subsets corresponding to each sub-kernel matrix are merged and used as the pruning results of each stage, and the pruning results of the previous stage are used as the input of the next stage.

[0066] In each stage of the large language model, each video segment corresponds to an optimal visual tag subset. By merging the optimal visual tag subsets of each video segment contained in the input video, the pruning result of the input video at the corresponding stage can be obtained.

[0067] This disclosure segments the input video based on the overlap rate of adjacent video frames, effectively identifying content abrupt change boundaries and avoiding semantic breaks caused by cross-segment pruning. By dynamically allocating the final pruning rate based on the similarity of feature vectors between each video segment and other segments, adaptive adjustment of the pruning budget can be achieved, preserving more details in high-information-density segments and increasing compression in redundant segments. By fusing the first similarity and relevance to construct the final kernel matrix, temporal diversity and language task orientation can be considered simultaneously during the matrix-based pruning process. By extracting the sub-kernel matrices corresponding to each video segment from the global kernel matrix (final kernel matrix), localized and structure-aware pruning can be achieved, avoiding cross-video segment interference. By using the pruning results of the previous stage as input for the next stage, progressive and cascaded visual tagging refinement can be achieved, eliminating redundancy layer by layer, significantly reducing computational load without accumulating information loss.

[0068] Figure 2 A flowchart illustrating an overlap rate calculation method according to one embodiment of this disclosure is shown. Figure 2 The method shown includes steps S210 to S230.

[0069] In step S210, sub-visual labels for each spatial location contained in each video frame image are obtained based on the first visual label of each video frame image.

[0070] In step S220, based on the sub-visual tags, the second similarity of adjacent video frame images at each spatial location is calculated.

[0071] In step S230, the average value of the second similarity of each spatial location contained in adjacent video frame images is used as the overlap rate between adjacent video frame images.

[0072] The above steps, by employing fine-grained sub-visual labeling and spatial location-aware second similarity calculation, can improve the accuracy and robustness of overlap rate estimation between adjacent video frame images.

[0073] In the process of calculating the second similarity in step S230, existing similarity calculation methods can be used, such as calculating the cosine similarity between the first visual marker of the first video frame image at the first spatial location and the first visual marker of the second video frame image at the first spatial location, as the second similarity.

[0074] As a further implementation, after segmenting the input video into multiple video segments, the method further includes: if within a video segment there are K consecutive video frame images whose second similarity at a first spatial location is the largest among the K second similarities of all video frame images contained in the video segment at the first spatial location, then the average value of the first visual labels of the K consecutive video frame images at the first spatial location is used as a third visual label, where K is greater than 1. The first visual label of the K consecutive video frame images at the first spatial location is then replaced with the third visual label. The first spatial location can be any spatial location within the video frame images. This implementation, by introducing an averaged third visual label at a specific spatial location of highly similar consecutive frames and replacing the original first visual label, effectively compresses local redundancy while preserving key spatial semantics, further improving the compactness and robustness of the visual representation.

[0075] Figure 3 A flowchart illustrating a method for determining the final pruning rate according to one embodiment of this disclosure is shown. Figure 3 The method shown includes steps S310 to S350.

[0076] In step S310, the sum of the feature vectors of all video segments to the left of a video segment is calculated to obtain the left-side feature vector of the video segment. The sum of the feature vectors of all video segments to the right of a video segment is calculated to obtain the right-side feature vector of the video segment. The video segments to the left of a given video segment can be those preceding that video segment in time. The video segments to the right of a given video segment can be those following that video segment in time.

[0077] In step S320, the third similarity between the feature vector of the video segment and the feature vector on the right is calculated, and this is taken as the right-side similarity of the video segment. The difference between the third similarity between the feature vector of segment 1 and the feature vector on the left is calculated, and this is taken as the left-side similarity of the video segment.

[0078] In step S330, the left-side similarity and right-side similarity are weighted and summed to obtain the expected pruning rate of the video segment.

[0079] In step S340, the expected pruning rate is standardized to obtain the standard pruning rate of the video clip.

[0080] In step S350, the sum of the products of the base pruning rate and the standard pruning rate and the deviation of the pruning rate is taken as the final pruning rate of the video segment.

[0081] In the final pruning rate determination method in steps S310 to S350, the expected pruning rate of each video segment is dynamically generated by comprehensively considering the semantic similarity between each video segment and the video segments to its left and right. Standardizing the expected pruning rate eliminates scale differences between different video segments. Combining the standard pruning rate with the base pruning rate and a bias term allows for precise control of the compression intensity of each video segment, effectively avoiding over-pruning of high-information segments or under-pruning of redundant segments while preserving key temporal semantics and cross-segment diversity.

[0082] As one possible implementation, the cosine similarity algorithm can be used when calculating the third similarity.

[0083] As one possible implementation, in the process of obtaining the expected pruning rate of the video segment by weighted summing of the left and right similarities, the sum of the weights of the left and right similarities is set to 1. For example, the weight of the left similarity is the first weight value ( ), set the weight of the right-side similarity to the second weight value (1- The sum of the product of the left-side similarity and the first weight value, and the product of the right-side similarity and the second weight value, is used as the expected pruning rate of the video segment.

[0084] As one possible implementation, the expected pruning rate is standardized to obtain the standard pruning rate of the video segments. This includes calculating the average and standard deviation of the expected pruning rates of each video segment contained in the video. The difference between the expected pruning rate and the average rate of each video segment is divided by the standard deviation to obtain the standard pruning rate. The standardization method used in this implementation can eliminate the pruning rate distribution shift caused by differences in content complexity between different video segments.

[0085] Regarding step S170, in some embodiments of this disclosure, it may include, for example... Figure 4 Steps S1701 to S1708 are shown.

[0086] In step S1701, the original kernel matrix is ​​constructed based on the first similarity between the second visual tags of adjacent video frame images.

[0087] In step S1702, the input prompts of the large language model are segmented to obtain the instruction tag sequence.

[0088] In step S1703, the last instruction tag in the instruction tag sequence is extracted.

[0089] In step S1704, the correlation between the last instruction marker and the second visual marker of each video frame image is calculated.

[0090] In step S1705, a diagonal matrix of the same size as the original kernel matrix is ​​constructed based on the relevance. The rows and columns of the diagonal matrix correspond to the second visual labels of each video frame image contained in the video, and the diagonal elements in the diagonal matrix represent the relevance.

[0091] In step S1706, the diagonal matrix is ​​multiplied by the original kernel matrix of each stage for the first time to obtain the intermediate kernel matrix of each stage.

[0092] In step S1707, the intermediate kernel matrix of each stage is multiplied by the diagonal matrix for the second time to obtain the final kernel matrix of each stage.

[0093] In step S1708, the sub-kernel matrix corresponding to each video segment is extracted from the final kernel matrix of each stage.

[0094] In steps S1701 to S1708, the original kernel matrix preserves the local redundant structure between video frame images, which helps to remove static or repetitive content. By bidirectionally weighting the original kernel matrix with a diagonal matrix constructed based on relevance, the sampling probability of video frame images highly relevant to the current input prompt in the determinant point process can be enhanced, while suppressing the influence of video frame images.

[0095] In a specific example, the input video contains 6 video frames, with second visual labels F1, F2, F3, F4, F5, and F6 for each frame. The input prompt is "Describe what changes occurred in the video".

[0096] In step S1701, a 6×6 original kernel matrix L0 is constructed. The rows and columns of the original kernel matrix L0 represent the second visual labels F1, F2, F3, F4, F5, and F6 of the six video frame images, respectively. The off-diagonal elements of the original kernel matrix L0 represent the first similarity between the second visual labels of corresponding rows and columns; for example, the off-diagonal element in the first row and second column... s 12 represents the first similarity (e.g., cosine similarity) between the second visual label F1 of the first video frame and the second visual label F2 of the second video frame. The diagonal elements in the original kernel matrix L0 can typically be set to 1 (indicating perfect similarity). An example of the original kernel matrix L0 is shown below.

[0097] In step S1702, the input prompt "Describe what changes have occurred in the video" is processed by word segmentation, and the resulting instruction tag sequence can be: ["describe", "video", "in", "occurred", "already", "what", "change"].

[0098] In step S1703, the last extracted instruction is marked as "change".

[0099] In step S1704, the correlation between the last instruction marked "change" and the second visual label (F1, F2, F3, F4, F5, or F6) of each video frame image is calculated, resulting in six correlation values. For example, the six correlation values ​​are 0.3, 0.4, 0.8, 0.9, 0.7, and 0.2, respectively.

[0100] In step S1705, a 6×6 diagonal matrix D is constructed. The diagonal elements of the diagonal matrix D are the aforementioned correlation values ​​(0.3, 0.4, 0.8, 0.9, 0.7, 0.2) in sequence, and the values ​​of the off-diagonal elements are all 0.

[0101] In step S1706, the diagonal matrix D is multiplied by the original kernel matrix L0 to obtain the intermediate kernel matrix L1 = D × L0.

[0102] In step S1707, the intermediate kernel matrix L1 is multiplied by the diagonal matrix D to obtain the final kernel matrix L. final = L1× D = D × L0× D.

[0103] In step S1708, it is assumed that the six video frames are divided into two video segments: video segment A (F1–F3) and video segment B (F4–F6). Then, from the final kernel matrix L... final Extract the sub-matrices with row indices {1,2,3} and column indices {1,2,3} from the data to obtain the sub-kernel matrix L of video segment A. A From the final kernel matrix L final Extract the sub-matrices with row indices {4,5,6} and column indices {4,5,6} respectively to obtain the sub-kernel matrix L of video segment B. B .

[0104] As one possible implementation, the relevance calculation process includes: calculating the dot product between the last instruction marker and each second visual marker to obtain multiple dot product results; dividing each dot product result by the square root of the dimension of the second visual marker to obtain a standardized dot product result; performing softmax normalization on each standardized dot product result to obtain a first normalized result; and performing min-max normalization on the first normalized result to obtain the relevance corresponding to each second visual marker. For example, the formula for calculating the relevance can be expressed as: Where r represents the relevance. This represents the min-max normalization function. This represents the softmax normalization function. The last instruction marker can be a 1 × d row vector, where d represents the dimension of the second visual marker. This represents a matrix consisting of the second visual labels of each video frame image contained in the input video. It can be an N × d matrix, where N represents the number of video frame images contained in the input video. The superscript T denotes the transpose of the matrix. The result can be a 1 × N vector.

[0105] By using the above-mentioned relevance calculation method, large language models can more clearly distinguish which video frame images are most relevant to user input prompts, thus prioritizing the retention of these "keyframes" during pruning.

[0106] According to any of the above embodiments, this disclosure also provides a large language model pruning device 500. Figure 5 This is a schematic block diagram of a large language model pruning device 500 according to one embodiment of this disclosure. Figure 5 As shown, the large language model pruning device 500 includes a visual encoding module 510, an overlap rate calculation module 520, a video segmentation module 530, a feature vector determination module 540, a final pruning rate determination module 550, a second visual tag determination module 560, a sub-kernel matrix extraction module 570, an optimal visual tag subset determination module 580, and a pruning result determination module 590.

[0107] The visual encoding module 510 is used to perform visual encoding on each video frame image contained in the input video to obtain the first visual label of each video frame image.

[0108] The overlap rate calculation module 520 is used to calculate the overlap rate between adjacent video frame images based on the first visual marker of each video frame image.

[0109] The video segmentation module 530 is used to segment the input video into multiple video segments based on the overlap rate, such that the overlap rate between adjacent video frame images belonging to different video segments is less than the overlap rate threshold.

[0110] The feature vector determination module 540 is used to determine the feature vector of each video segment based on the first visual markers of each video frame image contained in each video segment.

[0111] The final pruning rate determination module 550 is used to determine the final pruning rate of each video segment based on the similarity between the feature vectors of each video segment and the feature vectors of other video segments.

[0112] The second visual label determination module 560 is used to divide the processing of the input video by the large language model into multiple stages and obtain the second visual labels of each video frame image output by each stage.

[0113] The sub-kernel matrix extraction module 570 is used to construct the final kernel matrix for each stage of the matrix-deterministic process based on the first similarity between the second visual markers of adjacent video frame images and the correlation between the second visual markers of each video frame image and the last instruction marker corresponding to the input prompt, and extract the sub-kernel matrix corresponding to each video segment from the final kernel matrix.

[0114] The optimal visual tag subset determination module 580 is used to select the optimal visual tag subset that satisfies the final pruning rate of the corresponding video segment for each sub-core matrix through maximum a posteriori inference.

[0115] The pruning result determination module 590 is used to merge the optimal visual label subsets corresponding to each sub-kernel matrix as the pruning result of each stage, and use the pruning result of the previous stage as the input of the next stage.

[0116] The large language model pruning device 500 disclosed herein can be implemented through a computer software architecture.

[0117] According to further embodiments of this disclosure, an electronic device is also provided. Figure 6 This diagram illustrates a schematic block diagram of an electronic device employing a processor-based hardware implementation according to an embodiment of the present disclosure. The hardware structure of the electronic device of the present disclosure can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one connecting line is used in this figure, but this does not indicate that there is only one bus or one type of bus.

[0118] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0119] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0120] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0121] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0125] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0126] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method for pruning large language models, characterized in that, include: Visual encoding is performed on each video frame image contained in the input video to obtain the first visual label of each video frame image; The overlap rate between adjacent video frames is calculated based on the first visual marker of each video frame image. Based on the overlap rate, the input video is divided into multiple video segments, such that the overlap rate between adjacent video frame images belonging to different video segments is less than the overlap rate threshold. Based on the first visual markers of each video frame image contained in each video segment, determine the feature vector of each video segment; The final pruning rate of each video segment is determined based on the similarity between the feature vectors of each video segment and the feature vectors of other video segments. The processing of the input video by the large language model is divided into multiple stages to obtain the second visual labels of each video frame image output at each stage. Based on the first similarity between the second visual markers of adjacent video frame images, and the correlation between the second visual markers of each video frame image and the last instruction marker corresponding to the input prompt, the final kernel matrix for each stage of the matrix-deterministic process is constructed, and the sub-kernel matrix corresponding to each video segment is extracted from the final kernel matrix; For each sub-core matrix, the optimal visual tag subset that satisfies the final pruning rate of the corresponding video segment is obtained by maximum a posteriori inference; as well as The optimal visual tag subsets corresponding to each sub-kernel matrix are merged and used as the pruning results of each stage, and the pruning results of the previous stage are used as the input of the next stage.

2. The large language model pruning method as described in claim 1, characterized in that, Based on the first visual markers of each video frame image, the overlap rate between adjacent video frame images is calculated, including: Based on the first visual marker of each video frame image, obtain the sub-visual markers of each spatial location contained in each video frame image; Based on the sub-visual tags, calculate the second similarity of adjacent video frame images at each spatial location; and The average value of the second similarity of each spatial location contained in the adjacent video frame images is taken as the overlap rate between the adjacent video frame images.

3. The large language model pruning method as described in claim 1, characterized in that, Based on the overlap rate, the input video is segmented into multiple video segments, including: If the overlap rate between adjacent video frames is less than the overlap rate threshold, a video segment boundary is created between the adjacent video frames; and Based on the boundaries of the video segments, the input video is segmented to obtain multiple video segments.

4. The large language model pruning method as described in claim 2, characterized in that, After dividing the input video into multiple video segments, the process further includes: If, within the video segment, there are K consecutive video frame images whose second similarity at a first spatial location belongs to the K largest among the video frame images contained in the video segment at the first spatial location, then the average value of the first visual markers of the K consecutive video frame images at the first spatial location is used as the third visual marker, where K is greater than 1; and The first visual marker at the first spatial location of the consecutive K-frame video image is replaced with the third visual marker.

5. The large language model pruning method as described in claim 1, characterized in that, Based on the first visual markers of each video frame image contained in each video segment, the feature vector of each video segment is determined, including: The first visual markers of each video frame image contained in the video segment are subjected to average pooling to obtain the feature vector of the video segment.

6. The large language model pruning method as described in claim 1, characterized in that, The final pruning rate of each video segment is determined based on the similarity between the feature vectors of each video segment and the feature vectors of other video segments, including: The feature vector of the video segment to the left is obtained by summing the feature vectors of the video segments to the left of the video segment; the feature vector of the video segment to the right of the video segment is obtained by summing the feature vectors of the video segments to the right of the video segment. Calculate the third similarity between the feature vector of the video segment and the right-side feature vector, as the right-side similarity of the video segment; calculate the difference between 1 and the third similarity between the feature vector of the video segment and the left-side feature vector, as the left-side similarity of the video segment; The expected pruning rate of the video segment is obtained by weighted summation of the left-side similarity and the right-side similarity. The expected pruning rate is standardized to obtain the standard pruning rate of the video segment; and The final pruning rate of the video segment is the sum of the products of the base pruning rate and the standard pruning rate and the deviation from the pruning rate.

7. The large language model pruning method as described in claim 1, characterized in that, Also includes: The original kernel matrix is ​​constructed based on the first similarity between the second visual tags of adjacent video frame images; The input prompts of the large language model are segmented to obtain a sequence of instruction tags; Extract the last instruction marker from the instruction marker sequence; Calculate the correlation between the last instruction marker and the second visual marker of each video frame image; Based on the relevance, a diagonal matrix with the same size as the original kernel matrix is ​​constructed. The rows and columns of the diagonal matrix correspond to the second visual labels of each video frame image contained in the video. The diagonal elements in the diagonal matrix represent the relevance. The diagonal matrix is ​​multiplied by the original kernel matrix of each stage to obtain the intermediate kernel matrix of each stage. Perform a second matrix multiplication operation between the intermediate kernel matrix and the diagonal matrix at each stage to obtain the final kernel matrix for each stage; and Extract the sub-kernel matrix corresponding to each video segment from the final kernel matrix of each stage.

8. The large language model pruning method as described in claim 7, characterized in that, The process of calculating the relevance includes: Calculate the dot product between the last instruction marker and each of the second visual markers to obtain multiple dot product results; Divide each dot product result by the square root of the dimension of the second visual label to obtain the standardized dot product result; Softmax normalization is performed on each standardized dot product result to obtain the first normalized result; and The first normalization result is subjected to min-max normalization to obtain the relevance corresponding to each second visual label.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes the execution instructions stored in the memory, causing the processor to perform the large language model pruning method according to any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the large language model pruning method as described in any one of claims 1 to 8.

Images