Image segmentation method based on hierarchical pruning of modal collaborative visual language large model

Abstract

The application discloses an image segmentation method based on layered pruning of a modal collaborative visual language large model, comprising the following steps: acquiring a remote sensing image; converting the encoded visual information in each block into a visual representation with language information; grouping functionally similar layers in different modalities based on the output of each block; introducing KL divergence as an auxiliary target to support parameter importance estimation; calculating a modal collaborative importance score; calculating the sparsity of each weight; finding the optimal sparse weight that can maximize the performance of the model; performing layer-by-layer pruning of the visual language large model according to the layer sparsity; and performing image segmentation on the pruned visual language large model and outputting the image segmentation result. The application constructs cross-modal dependency relationships between VLM layers from different modalities, performs layer-by-layer pruning based on global cross-modal information, and utilizes the soft target of the original model to benefit from the original model and retain its behavior characteristics.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to an image segmentation method based on hierarchical pruning of a large modality collaborative visual language model. Background Art

[0002] Remote sensing image segmentation is the process of dividing an image into multiple regions or objects with similar characteristics. This approach has wide applications in areas such as land cover classification, urban planning, and environmental monitoring. Currently, techniques for remote sensing image segmentation primarily fall into two categories: traditional methods and deep learning-based approaches. Traditional methods include thresholding, edge detection, region growing, clustering, and watershed algorithms. With the development of convolutional neural networks (CNNs), numerous deep learning-based models have been applied to remote sensing image processing tasks, achieving remarkable results. These include fully convolutional networks (FCNs), U-Net, SegNet, the DeepLab series, and PSPNet (Pyramid Scene Parsing Network), which uses a pyramid pooling module to aggregate contextual information at different scales. Furthermore, there are techniques designed specifically for specific challenges, such as sliding window strategies and multi-scale fusion methods for processing high-resolution remote sensing imagery, as well as spatiotemporal modeling techniques that account for time series variation.

[0003] Although remote sensing image segmentation technology has made significant progress, there are still some challenges and shortcomings: data acquisition and annotation are difficult, and obtaining high-quality remote sensing images is costly; objects of different sizes may exist simultaneously in the same remote sensing image, such as buildings, roads, and farmland, and how to effectively deal with this multi-scale problem is currently a difficulty; images taken at different times will cause the same ground objects to exhibit different characteristics in the image due to changes in lighting conditions (such as morning and evening, seasonal changes); certain specific types of ground objects appear less frequently in the image, and it is difficult to fully learn the characteristics of these categories during model training, resulting in decreased prediction accuracy; the generalization ability is limited, and the trained model performs poorly on data collected in different regions or by different sensors, and has poor cross-domain adaptability. When encountering new types of ground objects that have never been seen before, the existing model may not be able to correctly identify them.

[0004] Neural symbolic approaches empower neural networks with symbolic reasoning capabilities, enabling the construction of more robust artificial intelligence (AI) systems. In recent years, with the increasing demand for multimodal data processing, vision-language models (VLMs) have become indispensable in advancing neural symbolic systems. Existing VLMs have demonstrated remarkable performance in image segmentation tasks by enabling effective multimodal perception and commonsense reasoning across multiple domains.

[0005] Researchers pre-train on large-scale image-text corpora to develop VLMs that can learn universal cross-modal representations for subsequent vision-language tasks. However, large-scale pre-training requires a lot of computing resources. At the same time, the significant progress in large language models (LLMs) has driven the exponential growth of VLMs, which further increases the cost of training VLMs from scratch. To cope with the huge demand for computing resources, researchers have developed modular pre-training techniques such as Q-former. Through a trainable lightweight projection component that maps the input or output of one modality to another, VLMs are able to leverage well-trained single-modality large models (LMs). Although modular pre-training techniques significantly reduce training costs, the computational and memory overhead of large-scale VLMs remains high.

[0006] In recent years, model compression has attracted considerable attention due to its ability to reduce the size of neural networks while maintaining their capabilities. To popularize powerful large models (LMs), researchers have pursued a wide range of approaches, including knowledge distillation, model pruning, quantization, and low-rank decomposition. Model pruning has received widespread attention as a key approach. Subsequently, researchers have proposed a variety of methods and achieved significant performance.

[0007] Current VLM pruning methods suffer from a significant imbalance in weight and gradient distribution between the language and vision submodules, making it difficult to determine the appropriate compression ratio for different components across modalities. To address this issue, researchers introduced iterative pruning, which performs pruning by learning pruning masks, which involves extensive computation of the inverse Hessian matrix and retraining. To enhance the overall understanding of layer-wise pruning, Sung et al. proposed an efficient layer-wise pruning (ECoFLaP) method that adaptively calculates the sparsity rate of each layer based on a global importance score. However, ECoFLaP focuses on calculating the importance of a single layer in the entire neural network, ignoring the correlation between layers of sub-networks of different modalities. Summary of the Invention

[0008] In view of this, the present invention constructs a modality-cooperative hierarchical pruning (MoCo) method for remote sensing image segmentation. Specifically, MoCo is designed for Q-former-based VLMs and estimates the importance of layers through cross-modal guidance. MoCo uses the original Q-former of VLMs as a query term to perform cross-modal alignment, effectively capturing the dependencies between sub-network layers of different modalities, thereby grouping layers of different modalities to collaboratively estimate the importance of layers. Therefore, the present application utilizes global cross-modal guidance to obtain sparsity and adaptively performs hierarchical unstructured weight pruning. Since MoCo does not introduce additional parameters, VLMs can be efficiently compressed in a single round of pruning.

[0009] In addition, existing layer-wise pruning methods usually evaluate the importance of layers by adding noise. Given a set of calibration data, the importance score of each layer can be calculated based on the change in the loss value after adding noise. However, this method cannot preserve the original knowledge of the model because it inevitably converges to a specific area. In order to retain knowledge and generalization ability during the model compression process, this application introduces a multi-perspective estimation mechanism to support the evaluation of layer importance. The method consists of two main steps. First, the samples of a given training dataset are divided into multiple clusters, and representative data points are selected from each cluster as calibration data, thereby providing a comprehensive perspective for the compression process. Second, soft targets of the original model are introduced to supplement the layer importance evaluation process, thereby utilizing the knowledge of the original model and forcing the compressed model to imitate its behavior.

[0010] To achieve the above objectives, the present application discloses an image segmentation method based on hierarchical pruning of a modality-cooperative visual language large model, comprising the following steps:

[0011] Acquisition of remote sensing images;

[0012] Given a calibration dataset and visual encoder ,in , N is the number of layers of the visual encoder, is the weight of the visual encoder, For visual input, is the language input, T is the number of data in the dataset, and the pre-trained Q-Former is used to convert the encoded visual information of each block in the image into a visual representation with language information;

[0013] Based on the output of each block, layers with similar functions in different modalities are grouped;

[0014] KL divergence is introduced as an auxiliary objective to support parameter importance estimation;

[0015] Calculate the modal co-importance score and assign the modal co-importance score to all weights in the same group;

[0016] After obtaining the importance score of each weight in each layer, calculate the sparsity of each weight;

[0017] In layer sparsity and calibration dataset Based on this, we find the optimal sparse weights that can maximize the performance of the model. ;

[0018] According to the layer sparsity and Perform hierarchical pruning of large visual language models;

[0019] The pruned visual language model performs image segmentation and outputs the image segmentation results.

[0020] Preferably, the optimal sparse weight The calculation is as follows:

[0021]

[0022] in is a function that calculates the importance score of each parameter in the weight matrix, express The weight after pruning is the sparse weight of the i-th layer, It is the full capacity model The weights of the layers, Indicates the Local target of the layer, Indicates the Target sparsity for the layer.

[0023] Preferably, the Q-Former-based visual language layer grouping utilizes a pre-trained Q-Former to convert the encoded visual information in each block into a visual representation with language information, including:

[0024]

[0025] in Represents a visual block The encoded visual representation of Represents a visual block The encoded visual representation of ,and , It is a language block The coding language representation is, It is a language block The encoding language representation of is, the weight is ,and Is weighted Q-Former, for converting visual blocks Extract visual information from each output and transform it into a representation with language information .

[0026] Preferably, the method of grouping functionally similar layers in different modalities based on the output of each block comprises: given a representation set with language information extracted from the output of the visual sub-model block and the set of encoded language representations of the language submodel block , cross attention is introduced to measure the correlation of different modal encoding representations, so that the layers of different modalities are grouped according to the cross attention scores of the encoding representations:

[0027]

[0028] in represents the correlation matrix between the vision and language sub-model layers in the vision-language model, Represents the visual sub-model block and the language sub-model nugget The correlation score of is a grouping set, where Indicates the Group layer, , M is the number of layers of the language encoder, for arrive of , the visual sub-model Blocks and their most related blocks Match, where and , for unmatched blocks, the layers in each block are grouped separately and marked as arrive .

[0029] Preferably, in order to provide parameter importance estimation during model compression, the given pre-training dataset is first The samples are divided into multiple clusters, namely ; Then, for each cluster Sort and select the data point closest to the center As representative samples, compose the multi-view calibration dataset .

[0030] Preferably, after obtaining the multi-view calibration dataset Finally, the output probability distribution of the original model is extracted as soft labels to capture the comprehensive knowledge of the model in various fields.

[0031] Preferably, the introduction of KL divergence as an auxiliary objective to support parameter importance estimation includes:

[0032] Given the weights of the full capacity model ,in For the The weight of the layer is calculated by targeting the global goal of the visual language model and consistency goals The zero-order approximate gradient of Importance:

[0033]

[0034] in It is The output of the layer contains the knowledge of the full capacity model; Indicates the The activation function or nonlinear transformation of the layer, is an additional parameter, is the objective function, is randomly generated Gaussian noise, d is the data in the standard data set, is the direction and scale of Gaussian noise, yes The importance score of is the output of layer i-1; by Add Gaussian noise and calculate weights Towards the target Impact ,in is the disturbance amplitude; therefore, Used as The importance score of the task is a combination of task performance and consistency with the full-capacity model.

[0035] Preferably, to achieve modal co-pruning of large visual language models, the modal co-importance score is calculated in three ways:

[0036] product: ,

[0037] Sum: ,

[0038] maximize: ;

[0039] is the importance score of weight W, Indicates the Group layer;

[0040] Then, the modal co-importance score is assigned to all weights within the same group: .

[0041] Preferably, after obtaining the importance score of each weight in each layer, the sparsity of each weight is calculated:

[0042]

[0043] in is the target sparsity of the full-capacity model, is the weight of the jth layer, |W| is the number of parameters, Based on The number of retained parameters calculated by the importance score of each layer is finally obtained. Sparsity .

[0044] Compared with the prior art, this application has the following beneficial effects:

[0045] This application is used to build cross-modal dependencies between VLM layers from different modalities, thereby supporting layer-by-layer pruning based on global cross-modal information.

[0046] This paper introduces a multi-view estimation mechanism to enhance the generation of compressed models. By leveraging the soft targets of the original model, the compressed model can benefit from the original model and preserve its behavioral characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Framework diagram of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention fall within the scope of protection of the present invention.

[0049] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0050] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0051] The technical solutions provided in the embodiments of this application involve technologies such as machine learning and natural language processing of artificial intelligence, and are specifically introduced and explained through the following embodiments.

[0052] Before introducing the embodiments of the present application, some terms involved in the present application are explained.

[0053] 1. Vision-Language Models (VLMs): These aim to build multimodal base models to process visual and linguistic information. Previous VLMs primarily learned multimodal base models through end-to-end pre-training on large-scale image-text paired datasets. To adapt to different tasks, researchers have developed a variety of architectures, including encoder-decoders, dual encoders, and fusion encoders. With the development of large models (LMs), Transformer-based VLMs have recently garnered widespread attention due to their outstanding performance in various vision-language tasks. However, the increasing parameter size of LMs inevitably leads to significant computational and memory overhead, making pre-training challenging.

[0054] To reduce costs and efficiently utilize existing pre-trained unimodal large models, the concept of modular pre-training was proposed. During training, separately pre-trained large vision and language models are typically fixed as modality encoders, decoders, or backbone networks. A trainable, lightweight projection component maps the input or output of one modality to the other, enabling large unimodal models to bridge the gap between modalities and perform multimodal tasks. The query transformer (Q-former), a classic architecture, is widely used to connect unimodal models. Specifically, it transfers the output of the large vision model to the input of the large language model by pre-training a set of learnable queries. While this reduces training costs, the deployment and inference of VLMs still require significant computational resources and memory. In this study, we investigated model pruning for Q-former-based vision-language models (VLMs) to reduce the deployment and inference costs of VLMs.

[0055] 2. Pruning of Large Transformer-Based Models: This approach aims to compress models by eliminating components or individual parameters, significantly reducing the model's memory and computational resource consumption. Previous pruning methods have primarily focused on unimodal models, with limited research addressing the pruning of multimodal models. With the rapid development of large models, visual language models (VLMs) have become an important approach for handling cross-modal tasks, and the demand for their efficiency is becoming increasingly urgent. Consequently, VLM pruning is gaining increasing attention. Existing pruning methods can be categorized as token-based and model-based.

[0056] Current pruning strategies are inherently influenced by the diverse architectural design of VLMs. This application is specifically designed for a modular multimodal learning framework based on Q-Former (Enhancing vision-language understanding with advanced large language models. In: The Twelfth International Conference on Learning Representations(2024)) for vision-language tasks, such as BLIP-2 (BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In: Krause, A., Brunskill, E., Cho, K., Engelhardt,B., Sabato, S., Scarlett, J. (eds.) Proceedings of the 40th International Conference on Machine Learning). This framework includes pre-trained FlanT5 and ViT for encoding textual and visual information, respectively, and a pre-trained Query Transformer (Q-Former) module to bridge the modality gap. BLIP-2, based on Q-Former, has achieved remarkable performance on a variety of vision-language tasks due to its strong zero-shot capability. Given the language encoder in BLIP-2 and visual encoder , whose weights are and ,in and Denotes the number of layers in each encoder. With these pre-trained backbones frozen, a lightweight query transformer module is introduced. , whose weight set is Q-Former extracts visual information and transforms it into a visual representation with language information while removing unimportant visual information. Let the language and visual input pair be , where the visual input Input the visual encoder and Q-Former in sequence to obtain a visual representation with language information . Subsequently, the output of BLIP-2 can be expressed as .

[0057] Current pruning strategies are essentially influenced by the diverse architectural design of VLMs. In this application, MoCo is designed for a Q-Former-based modular multimodal learning framework for vision-language tasks such as BLIP-2, which contains pre-trained FlanT5 and ViT for encoding textual and visual information, respectively, and a pre-trained query transformer (Q-Former) module to bridge the modality gap. BLIP-2 based on Q-Former has achieved remarkable performance on a variety of vision-language tasks due to its strong zero-shot capability. Given the language encoder in BLIP-2 and visual encoder , whose weights are and ,in and Represents the number of layers in each encoder. While keeping these pre-trained backbones, a lightweight query transformer module is introduced. , whose weight set is Q-Former extracts visual information and transforms it into a visual representation with language information while removing unimportant visual information. Let the language and visual input pair be , where the visual input Input the visual encoder and Q-Former in sequence to obtain a visual representation with language information . Subsequently, the output of BLIP-2 can be expressed as .

[0058] In this application, we aim to prune the BLIP-like visual language model based on Q-Former. Since the Q-Former architecture only accounts for a small part (about 5%) of the overall parameters of the VLM, this application keeps the Q-Former unchanged. Traditional layer-by-layer pruning methods try to and a set of calibration data Based on this, we find the optimal sparse weights that can maximize the performance of the model. ,Right now:

[0059]

[0060] in is a function that calculates the importance score of each parameter in the weight matrix, express The weight after pruning, Indicates the The local target of the layer, Indicates the The target sparsity of the layer. Typically, Setting the same value for all layers results in suboptimal performance in visual language models. Although subsequent researchers proposed determining layer sparsity based on the importance of layers in VLMs, little consideration was given to the interaction between modalities within visual language models. To address this problem, the present application proposes a modality coordination (MoCo) framework for performing layer-by-layer pruning of VLMs.

[0061] 1 Q-Former-based visual language layer grouping

[0062] Q-Former-based visual language layer grouping aims to capture the modal interaction in large-scale visual language models, thereby estimating the sparsity and parameter importance of layers in a modality-coordinated manner. Specifically, given a calibration dataset and a visual encoder wherein the calibration dataset is a remote sensing image dataset, including but not limited to AID (Aerial Image Dataset: an aerial image dataset provided by the Chinese Academy of Sciences, containing 10,000 images divided into 30 categories), DeepGlobe Road Extraction Challenge (a dataset focused on extracting road networks from high-resolution satellite images, which helps the research and application of transportation networks), etc. Q-Former-based visual language layer grouping utilizes a pre-trained Q-Former to convert the encoded visual information in each block into a visual representation with language information:

[0063]

[0064] wherein represents the encoded visual representation of the visual block , the weight is , and . is the encoded language representation of the language block , the weight is , and . is the Q-Former with weight for extracting visual information from each output of the visual block and converting it into a representation with language information.

[0065] Therefore, based on the output of each block, functionally similar layers in different modalities are grouped. Given the set of representations with language information extracted from the visual submodel block output and the set of encoded language representations of the language submodel block , cross attention is introduced to measure the correlation of different modal encoding representations, so that the layers of different modalities are grouped according to the cross attention scores of the encoding representations:

[0066]

[0067] in represents the correlation matrix between the vision and language sub-model layers in the VLM, Represents the visual sub-model block and the language sub-model nugget 's relevance score. is a grouping set, where Indicates the Group layer, For arrive of , we will visual sub-model Blocks and their most related blocks Match, where and For unmatched blocks, the layers in each block are grouped separately and marked as arrive .

[0068] 2 Multi-perspective evaluation mechanism

[0069] In order to maximize the performance of the compressed model, this application adopts a multi-view estimation mechanism to support the consistency between the compressed model and the original model. The parameter importance estimation of the traditional layer-by-layer pruning method relies on a set of calibration datasets randomly selected from the VLM pre-training dataset. However, random selection often ignores the knowledge domain involved in the samples, which may lead to overfitting of the compression model. In addition, the parameter estimation of layer-by-layer pruning in the past has placed too much emphasis on the performance of the compression model on the calibration data (i.e., the accuracy given hard labels), which inevitably leads to knowledge loss and limits the comprehensive understanding of the original model.

[0070] In order to provide a more comprehensive estimate of parameter importance during model compression, we first give a given pre-training dataset The samples are divided into multiple clusters, namely Then, for each cluster Sort and select the data point closest to the center As representative samples, compose the multi-view calibration dataset .

[0071] After obtaining the multi-view calibration dataset After that, the output probability distribution of the original model is extracted as a soft label to capture the comprehensive knowledge of the model in various fields. In order to enhance the consistency between the compressed model and the original model, KL divergence is introduced as an auxiliary target to support parameter importance estimation. Specifically, given the weight of the full capacity model ,in For the The weight of the layer is calculated by targeting the VLM global target and consistency goals The zero-order approximate gradient of Importance:

[0072]

[0073] in It is The output of the layer contains the knowledge of the full capacity model; Indicates the The activation function or nonlinear transformation of the layer, is an additional parameter. Add Gaussian noise and calculate weights Towards the target Impact ,in is the disturbance amplitude. Therefore, Used as The importance score of the task is a combination of task performance and consistency with the full-capacity model.

[0074] To implement modal co-pruning of VLM, modal co-importance scores are calculated in three ways: (1) Product: , (2) Sum: , (3) maximize: . Then, the modal co-importance score is assigned to all weights in the same group: .

[0075] After obtaining the importance score of each weight in each layer, calculate the sparsity of each weight:

[0076]

[0077] in is the target sparsity of the full-capacity model, Based on The number of retained parameters for calculating the importance score of . Finally, we get Sparsity , and substitute it into equation (1) to perform layer-by-layer pruning.

[0078] The pruned model is used to segment the remote sensing image and output the segmented image.

[0079] Compared with the prior art, this application has the following beneficial effects:

[0080] This application is used to build cross-modal dependencies between VLM layers from different modalities, thereby supporting layer-by-layer pruning based on global cross-modal information.

[0081] This paper introduces a multi-view estimation mechanism to enhance the generation of compressed models. By leveraging the soft targets of the original model, the compressed model can benefit from the original model and preserve its behavioral characteristics.

[0082] As used herein, the word "preferred" is intended to serve as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as advantageous over other aspects or designs. Rather, the use of the word "preferred" is intended to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any of the naturally inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing examples.

[0083] Moreover, although the present disclosure has been shown and described with respect to one or implementation, those skilled in the art will think of equivalent variations and modifications based on reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if structurally different from the disclosed structure that performs the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that can be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0084] The functional units in the embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or multiple or more units may be integrated into a single module. The aforementioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc. The aforementioned devices or systems may execute the storage method in the corresponding method embodiment.

[0085] In summary, the above embodiment is one implementation method of the present invention, but the implementation method of the present invention is not limited to the described embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. An image segmentation method based on hierarchical pruning of a large modal collaborative visual language model, characterized by: The following steps are involved: Acquisition of remote sensing images; Given a calibration dataset and visual encoder ,in , N is the number of layers of the visual encoder, is the weight of the visual encoder, For visual input, is the language input, T is the number of data in the dataset, and the pre-trained Q-Former is used to convert the encoded visual information of each block in the image into a visual representation with language information; Based on the output of each block, layers with similar functions in different modalities are grouped; KL divergence is introduced as an auxiliary objective to support parameter importance estimation; Calculate the modal co-importance score and assign the modal co-importance score to all weights in the same group; After obtaining the importance score of each weight in each layer, calculate the sparsity of each weight; In layer sparsity and calibration dataset Based on this, we find the optimal sparse weights that can maximize the performance of the model. ; According to the layer sparsity and Perform hierarchical pruning of large visual language models; The pruned visual language model performs image segmentation and outputs the image segmentation results.

2. The image segmentation method based on hierarchical pruning of a modal collaborative visual language large model according to claim 1 is characterized in that: The optimal sparse weight The calculation is as follows: in is a function that calculates the importance score of each parameter in the weight matrix, express The weight after pruning is the sparse weight of the i-th layer, It is the full capacity model The weights of the layers, Indicates the Local target of the layer, Indicates the Target sparsity for the layer.

3. The image segmentation method based on hierarchical pruning of a modality collaborative visual language large model according to claim 2 is characterized in that: The Q-Former-based visual-language layer grouping utilizes the pre-trained Q-Former to convert the encoded visual information in each block into a visual representation with language information, including: in Represents a visual block The encoded visual representation of Represents a visual block The encoded visual representation of ,and , It is a language block The coding language representation is, It is a language block The encoding language representation of is, the weight is ,and Is weighted Q-Former, for converting visual blocks Extract visual information from each output and transform it into a representation with language information .

4. The image segmentation method based on hierarchical pruning of a modality collaborative visual language large model according to claim 3 is characterized in that: Based on the output of each block, the functionally similar layers in different modalities are grouped, including: given a representation set with language information extracted from the output of the visual sub-model block and the set of encoded language representations of the language submodel block , cross attention is introduced to measure the correlation of different modal encoding representations, so that the layers of different modalities are grouped according to the cross attention scores of the encoding representations: in represents the correlation matrix between the vision and language sub-model layers in the vision-language model, Represents the visual sub-model block and the language sub-model nugget The correlation score of is a grouping set, where Indicates the Group layer, , M is the number of layers of the language encoder, for arrive of , the visual sub-model Blocks and their most related blocks Match, where and , for unmatched blocks, the layers in each block are grouped separately and marked as arrive .

5. The image segmentation method based on hierarchical pruning of a modality-cooperative visual language large model according to claim 4 is characterized in that: In order to provide parameter importance estimation during model compression, we first give a pre-training dataset The samples are divided into multiple clusters, namely ; Then, for each cluster Sort and select the data point closest to the center As representative samples, compose the multi-view calibration dataset .

6. The image segmentation method based on hierarchical pruning of a modality collaborative visual language large model according to claim 5 is characterized in that: After obtaining the multi-view calibration dataset Finally, the output probability distribution of the original model is extracted as soft labels to capture the comprehensive knowledge of the model in various fields.

7. The image segmentation method based on hierarchical pruning of a modality-cooperative visual language large model according to claim 6 is characterized in that: The introduction of KL divergence as an auxiliary objective to support parameter importance estimation includes: Given the weights of the full capacity model ,in For the The weight of the layer is calculated by targeting the global goal of the visual language model and consistency goals The zero-order approximate gradient of importance.

8. The image segmentation method based on hierarchical pruning of a modality-cooperative visual language large model according to claim 7 is characterized in that: To achieve modal co-pruning of large visual language models, the modal co-importance score is calculated in three ways: product: , Sum: , maximize: ; is the importance score of weight W, Indicates the Group layer; Then, the modal co-importance score is assigned to all weights within the same group: .

9. The image segmentation method based on hierarchical pruning of a modality-cooperative visual language large model according to claim 8 is characterized in that: After obtaining the importance score of each weight in each layer, calculate the sparsity of each weight: in is the target sparsity of the full-capacity model, is the weight of the jth layer, |W| is the number of parameters, Based on The number of retained parameters calculated by the importance score of each layer is finally obtained. Sparsity .

Images