CLIP model structuring pruning method based on dependency graph and phased scheduling
By employing a dependency graph-based and staged scheduling approach, the CLIP model is structurally pruned, addressing the limitations of existing pruning methods in resource-constrained environments and the difficulty of engineering implementation. This approach achieves efficient model compression and performance recovery, making it suitable for edge devices and mobile terminals.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NORTHWEST UNIV
- Filing Date
- 2026-02-06
- Publication Date
- 2026-06-02
Smart Images
Figure CN122133712A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of deep learning model compression technology, specifically involving a CLIP model structured pruning method based on dependency graphs and staged scheduling. Background Technology
[0002] With the development of large-scale pre-trained cross-modal models, the CLIP model learns by comparing massive amounts of image and text data, effectively learning the alignment relationship between vision and text, enabling it to have both visual and text encoding capabilities, and achieving significant results in tasks such as zero-shot classification, image and text retrieval, and multimodal understanding.
[0003] Although CLIP models have been extensively studied, they typically contain hundreds of millions of parameters, resulting in enormous computational and storage overhead, which hinders their deployment in resource-constrained scenarios such as edge devices, mobile terminals, and online inference services. To reduce the complexity of CLIP models, existing technologies have proposed various model compression methods, such as distillation and quantization, unstructured pruning, and structured pruning, but all have certain limitations. For example, distillation and quantization methods reduce model size by using low-bit quantization or small model distillation, which reduces the number of parameters and inference costs to some extent, but this method usually depends on specific hardware and operator support, lacking versatility; unstructured pruning methods set sparse masks according to weight magnitudes, resetting some weights to zero while maintaining the tensor shape, achieving a high sparsity rate, but requiring specialized sparse computing libraries, resulting in limited acceleration in general GPU / CPU environments; structured pruning methods directly delete structural units such as convolutional kernels, channels, or attention heads, substantially reducing network width and computational cost, making it easier to achieve practical acceleration using existing computing libraries, but because it changes the tensor dimension, it easily introduces the problem of shape mismatch between upstream and downstream modules, increasing the difficulty of engineering implementation.
[0004] When specifically targeting CLIP models, structured pruning faces additional challenges: Firstly, the visual and text branches of CLIP map to the same contrastive learning space, and pruning either branch may disrupt cross-modal semantic alignment. Secondly, there are strict dimensionality consistency constraints between multi-head attention, residual connections, LayerNorm, and MLP layers within the Transformer; local channel pruning can easily lead to dimensionality non-closure, causing the model to fail to feed forward normally or become unstable during training. Furthermore, if the terminal LayerNorm and embedding layer are not adjusted simultaneously during intermediate layer pruning, it can easily cause inconsistencies in hidden dimensions, disrupting the geometric structure of the contrastive space.
[0005] Therefore, designing a configurable and phased structured channel pruning method while ensuring CLIP structure safety, cross-modal alignment stability, and training convergence remains a key problem that urgently needs to be solved in this field. Summary of the Invention
[0006] To address the aforementioned problems in existing technologies, this invention provides a structured pruning method for the CLIP model based on dependency graphs and staged scheduling. The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a structured pruning method for the CLIP model based on dependency graphs and staged scheduling, comprising: The prunable modules in the pre-trained CLIP model are abstracted as nodes in a dependency graph, and the channel dependencies and dimensional constraints between the prunable modules are represented by edges to form a dependency graph. According to preset rules, the pruning modules in the CLIP model are divided into multiple pruning groups; the pre-trained CLIP model includes a text branch and a visual branch, the text branch includes a text encoder, the visual branch includes a visual encoder, and both the visual encoder and the text encoder include a LayerNorm layer for normalization and an embedding layer for feature / Token representation. Using the text branch and the visual branch as target branches respectively, the first branch in the target branch is... Each channel in the pruning group is pruned and the dependency graph is updated by utilizing the first pruning group. The selection matrix of each pruning group updates the weights of the LayerNorm layer and the bias of the embedding layer in the target branch.
[0007] In one embodiment of the present invention, the visual branch further includes an image embedding layer and an image terminal projection layer, the text branch further includes a text embedding layer and a text terminal projection layer, and the visual encoder and the text encoder further include a multi-head attention layer and an MLP layer, wherein the multi-head attention layer contains... q / k / v The projection and output projection, the MLP layer, the text terminal projection layer and the image terminal projection layer are linear layers. The pruning module includes: the linear layer, the LayerNorm layer and the embedding layer in the visual encoder and the text encoder. The steps of abstracting prunable modules in a pre-trained CLIP model as nodes in a dependency graph, and forming the dependency graph by representing the channel dependencies and dimensional constraints between the prunable modules through edges, include: The prunable modules in the pre-trained CLIP model are abstracted as nodes in a dependency graph; If the output of the first pruning module is the input of the second pruning module, then add a channel consistency edge from the corresponding node of the first pruning module to the corresponding node of the second pruning module; both the first pruning module and the second pruning module belong to the linear layer.
[0008] In one embodiment of the present invention, if multiple pruning modules are located in the same multi-head attention layer and share the same input dimension or output dimension, then structurally shared edges are added between the nodes corresponding to the multiple pruning modules located in the same multi-head attention layer according to the data flow direction in the CLIP model.
[0009] In one embodiment of the present invention, the step of dividing the pruningable modules in the CLIP model into multiple pruning groups according to preset rules includes: Select any linear layer as a candidate pruning module, and calculate whether other pruning modules in the linear layer belong to the same level in the same branch as the candidate pruning module; if so, add the other pruning modules to the pruning group where the candidate pruning module is located.
[0010] In one embodiment of the present invention, the text branch and the visual branch are respectively used as target branches, and the first branch in the target branch is... Each channel in the pruning group is pruned and the dependency graph is updated by utilizing the first pruning group. The steps of updating the weights of the LayerNorm layer and the biases of the embedding layer in the target branch using the selection matrix of each pruning group include: Take any branch in the CLIP model as the target branch; For the target branch, the first Each pruning group contains pruning modules that can be pruned, and the dependency graph is updated. Generate the first Selection matrix of pruning groups The selection matrix The element is 0 or 1, representing the first... The corresponding channels in each pruning group are either pruned or retained; Using the selection matrix Update the weights of the LayerNorm layer and the biases of the embedding layer in the target branch:
[0011]
[0012] In the formula, , These represent the weights of the LayerNorm layer in the target branch before and after the update, respectively. , These represent the biases before and after the embedding layer update in the target branch, respectively; judge Is it less than , Indicates the number of pruning groups in the target branch; if so, then let And return the first branch in the target branch. The steps involve pruning the pruning modules contained in the pruning group; conversely, another branch in the CLIP model is taken as the target branch, and the steps involve pruning the first pruning module in the target branch. The steps for pruning the modules contained in a pruning group.
[0013] In one embodiment of the present invention, the first in the target branch The pruning process performed by the pruning modules contained in a pruning group includes: Calculate the first in the target branch The target sparsity of each pruning module:
[0014] In the formula, Indicates the first The weight of the layer where each pruning module resides. Indicates the first The preset base pruning rate of the branch where each pruning module is located. Indicates the first The target sparsity of a pruning module; Using the image-text pair dataset and the pre-trained CLIP model, the importance score of each channel in each pruning group in the target branch is calculated; Based on the importance score, the channels contained in each pruning group in the target branch are sorted in descending order; Based on the target sparsity, the Top-k algorithm is used to prune the channels contained in each pruning group in the target branch.
[0015] In one embodiment of the present invention, the step of calculating the importance score of each channel in each pruning group in the target branch using the image-text pair dataset and the pre-trained CLIP model includes: Obtain a preset number of image-text pairs from the image-text pair dataset and input them into the CLIP model. Each image-text pair includes image data and text data. After forward and backward propagation, the loss value of the preset loss function is calculated. ; Select the first branch in any pruning group within the target branch. Parameters of each channel Combined with the aforementioned loss value Calculate the first Importance of the main pathway in each channel:
[0016] In the formula, Expressing expectations, Indicates the first The parameters corresponding to each channel; Calculate the first Importance of backup paths for each channel ; Based on the importance of the main pathway Importance of backup pathways Calculate the first Importance score of each channel:
[0017] In the formula, Represents the weighting coefficients and .
[0018] In one embodiment of the present invention, the step of pruning the channels contained in each pruning group in the target branch using the Top-k algorithm according to the target sparsity includes: Based on the target sparsity, the Top-k algorithm is used to select the top-ranked... One channel is designated as a channel to be retained, and the remaining channels are designated as channels to be pruned. The channels to be pruned are pruned using either row pruning or column pruning. If the channel to be pruned is an output channel, then pruning is performed using row pruning, adjusting the weight of the channel to be pruned. Updated to If the channel to be pruned is an input channel, then pruning is performed using column pruning, adjusting the weight of the channel to be pruned. Updated to ; This represents the set of output channel indices retained by the pruning module to which the channel to be pruned belongs. This represents the set of input channels retained by the pruning module to which the channel to be pruned belongs.
[0019] In one embodiment of the present invention, if the channel to be pruned belongs to a multi-head attention layer, and the total hidden dimension of the multi-head attention layer is , The number of attention heads is Then, after pruning, the remaining number of channels in the multi-head attention layer to which the channel to be pruned belongs must still satisfy the following: ; In the formula, This represents the dimension of each attention head in the multi-head attention layer to which the channel to be pruned belongs. The channel index representing the remaining number of channels. Represents the set of positive integers.
[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention provides a structured pruning method for CLIP models based on dependency graphs and staged scheduling. By introducing dependency graphs to model the dependencies between pruneable modules and their channels, as well as multi-head dimension constraints and grouping constraints, and combining a staged sparsity scheduling mechanism with branch awareness and hierarchy awareness, and a dual-path channel importance evaluation strategy, structured channel compression of visual and text branches is achieved. This method improves the compression efficiency of CLIP models while ensuring the security of the network topology.
[0021] (2) To address the potential hidden dimension mismatch issue during pruning, this invention introduces a parameter alignment and rearrangement mechanism between the LayerNorm layer and the embedding layer to ensure the structural integrity of the pruned model. Furthermore, this invention performs distillation fine-tuning on the pruned CLIP model, enabling rapid recovery of model performance loss caused by pruning operations under limited training data and a small number of training epochs, significantly improving the practicality of the method.
[0022] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0023] Figure 1 This is a flowchart illustrating a CLIP model structured pruning method based on dependency graphs and staged scheduling provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a pre-trained CLIP model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the principle of the CLIP model structured pruning method based on dependency graphs and staged scheduling provided in this embodiment of the invention. Figure 4 This is a schematic diagram of the pruning process provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the calculation of importance scores provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the LayerNorm and Embedding alignment process provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the curves showing the changes in staged pruning and actual sparsity rate with different stages, provided in the embodiments of the present invention. Figure 8 This is a schematic diagram of the process of fine-tuning distillation after pruning provided in an embodiment of the present invention. Detailed Implementation
[0024] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.
[0025] Figure 1 This is a flowchart illustrating a structured pruning method for the CLIP model based on dependency graphs and staged scheduling provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a pre-trained CLIP model provided in an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the principle of the CLIP model structured pruning method based on dependency graphs and staged scheduling provided in this embodiment of the invention. Figures 1-3 As shown, this embodiment of the invention provides a structured pruning method for the CLIP model based on dependency graphs and staged scheduling, including: S1. The pruning modules in the pre-trained CLIP model are abstracted into nodes of a dependency graph, and the channel dependencies and dimensional constraints between the pruning modules are represented by edges to form a dependency graph.
[0026] S2. According to the preset rules, the pruning modules in the CLIP model are divided into multiple pruning groups. The pre-trained CLIP model includes a text branch and a visual branch. The text branch includes a text encoder, and the visual branch includes a visual encoder. Both the visual encoder and the text encoder include a LayerNorm layer for normalization and an embedding layer for feature / token representation.
[0027] In this embodiment, the pre-trained CLIP model includes a text branch and a vision branch. The vision branch includes an image embedding layer, a vision encoder, and an image terminal projection layer. The text branch includes a text embedding layer, a text encoder, and a text terminal projection layer. Both the vision encoder and the text encoder include: LayerNorm layers such as LayerNorm1 and LayerNorm2, an embedding layer, a multi-head attention layer, and an MLP layer. Optionally, the multi-head attention layer... q / k / v The projection and output projection, MLP layer, text terminal projection layer and image terminal projection layer are linear layers. The above-mentioned pruning modules refer to the linear layer, the LayerNorm layer and the embedding layer in the visual encoder and the text encoder.
[0028] For linear layers such as the first and second pruning modules, during edge connection, if the output of the first pruning module is the input of the second pruning module, a channel consistency edge is added from the corresponding node of the first pruning module to the corresponding node of the second pruning module. It is important to note that if multiple pruning modules are located within the same multi-head attention layer and share the same input or output dimension, then structurally shared edges are added between the corresponding nodes of the multiple pruning modules within the same multi-head attention layer, according to the data flow direction in the CLIP model.
[0029] Further, in step S2, any linear layer is selected as a candidate pruning module, and it is calculated whether other pruning modules in the linear layer belong to the same level in the same branch as the candidate pruning module; if so, the other pruning modules are added to the pruning group where the candidate pruning module is located.
[0030] S3. Using the text branch and the visual branch as target branches respectively, analyze the first... Each channel in the pruning group is pruned and the dependency graph is updated by utilizing the first pruning group. The selection matrix of each pruning group updates the weights of the LayerNorm layer and the bias of the embedding layer in the target branch.
[0031] Figure 4 This is a schematic diagram of the pruning process provided in an embodiment of the present invention. Figure 5 This is a schematic diagram illustrating the calculation of importance score provided in an embodiment of the present invention. Figure 6 This is a schematic diagram illustrating the alignment process of LayerNorm and Embedding provided in an embodiment of the present invention. For details, please refer to... Figures 4-6 Step S3 includes: S1. Take any branch in the CLIP model as the target branch.
[0032] S2, for the target branch, the first Each pruning group prunes the pruning modules it contains and updates the dependency graph.
[0033] First, calculate the first branch in the target branch. The target sparsity of each pruning module:
[0034] In the formula, Indicates the first The weight of the layer where each pruning module resides. Indicates the first The preset base pruning rate of the branch where each pruning module is located. Indicates the first The target sparsity of a pruning module.
[0035] Next, using the image-text pair dataset and the pre-trained CLIP model, the importance score of each channel in each pruned group in the target branch is calculated. Specifically, a predetermined number of image-text pairs are obtained from the image-text pair dataset and input into the CLIP model. Each image-text pair includes image data and text data, which are then fed into the visual branch and text branch, respectively. After forward and backward propagation, the loss value of the predetermined loss function is calculated. Select the first branch in any pruning group within the target branch. Parameters of each channel Combined with loss value Calculate the first Importance of the main pathway in each channel:
[0036] In the formula, Expressing expectations, Indicates the first The parameters corresponding to each channel.
[0037] Calculate the first Importance of backup paths for each channel And based on the importance of the main pathway Importance of backup pathways Calculate the first Importance score of each channel:
[0038] In the formula, Represents the weighting coefficients and .
[0039] Based on importance scores, the channels in each pruning group of the target branch are sorted in descending order. Then, according to the target sparsity, the Top-k algorithm is used to prune the channels in each pruning group of the target branch. For example, based on the target sparsity, the Top-k algorithm selects the top-ranked channels. One channel is designated as the channel to be retained, and the remaining channels are designated as the channels to be pruned; the channels to be pruned are pruned by row pruning or column pruning. It should be noted that if the channel to be pruned is an output channel, then pruning is performed using row pruning, reducing the weight of the channel to be pruned. Updated to If the channel to be pruned is an input channel, then pruning is performed using column pruning, adjusting the weights of the channels to be pruned. Updated to ; This represents the set of output channel indices retained by the pruning module to which the channel to be pruned belongs. This represents the set of input channels that the pruning module to which the channel to be pruned belongs retains.
[0040] Additionally, if the channel to be pruned belongs to a multi-head attention layer, and the total hidden dimension of the multi-head attention layer is... The number of attention heads is Then, after pruning, the remaining number of channels in the multi-head attention layer to which the pruning channel belongs must still satisfy the following: ; In the formula, This represents the dimension of each attention head in the multi-head attention layer to which the channel to be pruned belongs. Channel index representing the number of remaining channels. Represents the set of positive integers.
[0041] S3, Generate the first Selection matrix of pruning groups Select matrix The element is 0 or 1, representing the first element, respectively. The corresponding channels in each pruning group are either pruned or retained.
[0042] S4. Using the selection matrix Update the weights of the LayerNorm layer and the biases of the embedding layer in the target branch:
[0043]
[0044] In the formula, , These represent the weights of the LayerNorm layer before and after the update in the target branch, respectively. , These represent the biases before and after the embedding layer update in the target branch, respectively.
[0045] Steps S3-S4 utilize the selection matrix By aligning the LayerNorm layer in the target branch with the embedded layer, the consistency of the hidden space after pruning is ensured.
[0046] S5, Judgment Is it less than , Indicates the number of pruned groups in the target branch; if so, then let And return the first branch in the target branch. The steps for pruning the pruning modules contained in the pruning group are as follows: Conversely, another branch in the CLIP model is taken as the target branch, and the steps for pruning the pruning modules in the target branch are performed. The steps for pruning the modules contained in a pruning group.
[0047] Figure 7 This is a schematic diagram showing the curves of staged pruning and actual sparsity rate changing with different stages, provided in an embodiment of the present invention. As can be seen from the above process, the pruning process is divided into stages, with pruning groups as the unit. T Each stage, such as Figure 7 As shown, a monotonically increasing target sparsity can be set:
[0048] In the t Each stage, with Prune each pruning module to achieve the desired result; after each phase, recalculate the channel importance on the pruned structure. This approach captures the "secondary importance" brought about by structural changes. Channels that consistently score low across multiple stages are prioritized for removal, while the target sparsity is appropriately reduced for channels that consistently score high, thereby mitigating the risk of over-pruning key channels.
[0049] After pruning, the pruned CLIP model is further fine-tuned by distillation to restore cross-modal alignment performance and improve downstream retrieval results.
[0050] Figure 8 This is a schematic diagram of the post-pruning distillation fine-tuning process provided in an embodiment of the present invention. Figure 8 As shown, the pre-trained CLIP model is used as the teacher model, with its parameters frozen during the distillation phase. The pruned CLIP model is used as the student model, with its parameters updated under a predetermined compression structure. Depending on actual resources and application requirements, only a small number of low-rank adapter parameters inserted into the linear layer or multi-head attention layer can be updated, or the entire model can be briefly fine-tuned with a small learning rate and a limited number of training epochs to reduce the time and computational overhead of distillation.
[0051] On the same batch of image-text pairs, teachers and students output visual features and text features respectively, and construct their respective image-text similarity matrices. This invention employs a combined distillation method of "alignment-aware contrast loss + similarity matrix constraint + feature constraint," unifying them into a single total loss, which can be expressed as follows:
[0052] in, Used to maintain the student model's ability to learn through comparison of text and images. The image-text similarity relationship used to constrain the student model is close to that of the teacher. Used to maintain consistency between visual and textual representations in feature space. These are the weighting coefficients for the corresponding loss values.
[0053] In terms of training strategy, this invention selects only a small, representative dataset of images and text for distillation, reuses existing dual-tower forward computation and similarity calculation logic, and employs conventional optimizers such as AdamW to complete distillation in a short number of training epochs. Throughout the process, the teacher model remains in evaluation mode without updating its parameters, while the student model gradually approximates the teacher model's performance in cross-modal alignment and retrieval tasks while maintaining structural compression. This improves the practical performance of the pruned model without significantly increasing deployment costs.
[0054] As can be seen from the above embodiments, the beneficial effects of the present invention are as follows: (1) This invention provides a structured pruning method for CLIP models based on dependency graphs and staged scheduling. By introducing dependency graphs to model the dependencies between pruneable modules and their channels, as well as multi-head dimension constraints and grouping constraints, and combining a staged sparsity scheduling mechanism with branch awareness and hierarchy awareness, and a dual-path channel importance evaluation strategy, structured channel compression of visual and text branches is achieved. This method improves the compression efficiency of CLIP models while ensuring the security of the network topology.
[0055] (2) To address the potential hidden dimension mismatch issue during pruning, this invention introduces a parameter alignment and rearrangement mechanism between the LayerNorm layer and the embedding layer to ensure the structural integrity of the pruned model. Furthermore, this invention performs distillation fine-tuning on the pruned CLIP model, enabling rapid recovery of model performance loss caused by pruning operations under limited training data and a small number of training epochs, significantly improving the practicality of the method.
[0056] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0057] The use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples" indicates that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.
[0058] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.
Claims
1. A structured pruning method for the CLIP model based on dependency graphs and staged scheduling, characterized in that, include: The prunable modules in the pre-trained CLIP model are abstracted as nodes in a dependency graph, and the channel dependencies and dimensional constraints between the prunable modules are represented by edges to form a dependency graph. According to preset rules, the pruning modules in the CLIP model are divided into multiple pruning groups; the pre-trained CLIP model includes a text branch and a visual branch, the text branch includes a text encoder, the visual branch includes a visual encoder, and both the visual encoder and the text encoder include a LayerNorm layer for normalization and an embedding layer for feature / Token representation. Using the text branch and the visual branch as target branches respectively, the first branch in the target branch is... Each channel in the pruning group is pruned and the dependency graph is updated by utilizing the first pruning group. The selection matrix of each pruning group updates the weights of the LayerNorm layer and the bias of the embedding layer in the target branch.
2. The CLIP model structured pruning method based on dependency graphs and staged scheduling according to claim 1, characterized in that, The visual branch further includes an image embedding layer and an image terminal projection layer; the text branch further includes a text embedding layer and a text terminal projection layer; the visual encoder and the text encoder further include a multi-head attention layer and an MLP layer, wherein the multi-head attention layer contains... q / k / v The projection and output projection, the MLP layer, the text terminal projection layer and the image terminal projection layer are linear layers. The pruning module includes: the linear layer, the LayerNorm layer and the embedding layer in the visual encoder and the text encoder. The steps of abstracting prunable modules in a pre-trained CLIP model as nodes in a dependency graph, and forming the dependency graph by representing the channel dependencies and dimensional constraints between the prunable modules through edges, include: The prunable modules in the pre-trained CLIP model are abstracted as nodes in a dependency graph; If the output of the first pruning module is the input of the second pruning module, then add a channel consistency edge from the corresponding node of the first pruning module to the corresponding node of the second pruning module; both the first pruning module and the second pruning module belong to the linear layer.
3. The CLIP model structured pruning method based on dependency graphs and staged scheduling according to claim 2, characterized in that, If multiple pruning modules are located in the same multi-head attention layer and share the same input or output dimension, then structurally shared edges are added between the nodes corresponding to the multiple pruning modules located in the same multi-head attention layer according to the data flow direction in the CLIP model.
4. The CLIP model structured pruning method based on dependency graphs and staged scheduling according to claim 3, characterized in that, The step of dividing the pruningable modules in the CLIP model into multiple pruning groups according to preset rules includes: Select any linear layer as a candidate pruning module, and calculate whether other pruning modules in the linear layer belong to the same level in the same branch as the candidate pruning module; if so, add the other pruning modules to the pruning group where the candidate pruning module is located.
5. The CLIP model structured pruning method based on dependency graphs and staged scheduling according to claim 4, characterized in that, Using the text branch and the visual branch as target branches respectively, the first branch in the target branch is... Each channel in the pruning group is pruned and the dependency graph is updated by utilizing the first pruning group. The steps of updating the weights of the LayerNorm layer and the biases of the embedding layer in the target branch using the selection matrix of each pruning group include: Take any branch in the CLIP model as the target branch; For the target branch, the first Each pruning group contains pruning modules that can be pruned, and the dependency graph is updated. Generate the first Selection matrix of pruning groups The selection matrix The element is 0 or 1, representing the first... The corresponding channels in each pruning group are either pruned or retained; Using the selection matrix Update the weights of the LayerNorm layer and the biases of the embedding layer in the target branch: In the formula, , These represent the weights of the LayerNorm layer in the target branch before and after the update, respectively. , These represent the biases before and after the embedding layer update in the target branch, respectively; judge Is it less than , Indicates the number of pruning groups in the target branch; if so, then let And return the first branch in the target branch. The steps involve pruning the pruning modules contained in the pruning group; conversely, another branch in the CLIP model is taken as the target branch, and the steps involve pruning the first pruning module in the target branch. The steps for pruning the modules contained in a pruning group.
6. The CLIP model structured pruning method based on dependency graphs and staged scheduling according to claim 5, characterized in that, For the target branch, the first The pruning process performed by the pruning modules contained in a pruning group includes: Calculate the first in the target branch The target sparsity of each pruning module: In the formula, Indicates the first The weight of the layer where each pruning module resides. Indicates the first The preset base pruning rate of the branch where each pruning module is located. Indicates the first The target sparsity of a pruning module; Using the image-text pair dataset and the pre-trained CLIP model, the importance score of each channel in each pruning group in the target branch is calculated; Based on the importance score, the channels contained in each pruning group in the target branch are sorted in descending order; Based on the target sparsity, the Top-k algorithm is used to prune the channels contained in each pruning group in the target branch.
7. The CLIP model structured pruning method based on dependency graphs and staged scheduling according to claim 6, characterized in that, The steps for calculating the importance scores of each channel in each pruning group within the target branch using the image-text pair dataset and the pre-trained CLIP model include: A predetermined number of image-text pairs are obtained from the image-text pair dataset and input into the CLIP model. Each image-text pair includes image data and text data. After forward and backward propagation, the loss value of the preset loss function is calculated. ; Select the first branch in any pruning group within the target branch. Parameters of each channel Combined with the aforementioned loss value Calculate the first Importance of the main pathway in each channel: In the formula, Expressing expectations, Indicates the first The parameters corresponding to each channel; Calculate the first Importance of backup paths for each channel ; Based on the importance of the main pathway Importance of backup pathways Calculate the first Importance score of each channel: In the formula, Represents the weighting coefficients and .
8. The CLIP model structured pruning method based on dependency graphs and staged scheduling according to claim 6, characterized in that, Based on the target sparsity, the steps of pruning the channels contained in each pruning group in the target branch using the Top-k algorithm include: Based on the target sparsity, the Top-k algorithm is used to select the top-ranked... One channel is designated as a channel to be retained, and the remaining channels are designated as channels to be pruned. The channels to be pruned are pruned using either row pruning or column pruning. If the channel to be pruned is an output channel, then pruning is performed using row pruning, adjusting the weight of the channel to be pruned. Updated to If the channel to be pruned is an input channel, then pruning is performed using column pruning, adjusting the weight of the channel to be pruned. Updated to ; This represents the set of output channel indices retained by the pruning module to which the channel to be pruned belongs. This represents the set of input channels retained by the pruning module to which the channel to be pruned belongs.
9. The CLIP model structured pruning method based on dependency graphs and staged scheduling according to claim 8, characterized in that, If the channel to be pruned belongs to a multi-head attention layer, and the total hidden dimension of the multi-head attention layer is , The number of attention heads is Then, after pruning, the remaining number of channels in the multi-head attention layer to which the channel to be pruned belongs must still satisfy the following: ； In the formula, This represents the dimension of each attention head in the multi-head attention layer to which the channel to be pruned belongs. The channel index representing the remaining number of channels. Represents the set of positive integers.