Image segmentation method based on hierarchical pruning of modal collaborative visual language large model

Abstract

The invention discloses an image segmentation method based on hierarchical pruning of a modal collaborative visual language large model. The method comprises the following steps: acquiring a remote sensing image; converting the encoded visual information in each block into a visual representation with language information; based on the output of each block, grouping layers with similar functions in different modes; introducing KL divergence as auxiliary target support parameter importance estimation; calculating a modal collaboration importance score; calculating the sparseness of each weight; the optimal sparse weight capable of keeping the model performance to the maximum degree is found; executing layer-by-layer pruning of the visual language large model according to the layer sparseness; and carrying out image segmentation on the pruned visual language large model, and outputting an image segmentation result. According to the method, a cross-modal dependency relationship between VLM layers from different modals is constructed, and layer-by-layer pruning is carried out based on global cross-modal information; by utilizing the soft target of the original model, benefits can be obtained from the original model, and the behavior characteristics can be reserved.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing, and in particular relates to an image segmentation method based on hierarchical pruning of a large modality collaborative visual language model. Background Art

[0002] Remote sensing image segmentation refers to the process of dividing a remote sensing image into multiple regions or objects with similar characteristics, which has a wide range of applications in land cover classification, urban planning, environmental monitoring and other fields. At present, the techniques used for remote sensing image segmentation mainly include two categories: traditional methods and deep learning-based methods: traditional methods include thresholding, edge detection, region growing, clustering algorithms, watershed algorithms, etc. With the development of convolutional neural networks (CNNs), many deep learning-based models have been applied to remote sensing image processing tasks and have achieved remarkable results, such as fully convolutional networks (FCNs), U-Net, SegNet, DeepLab series, PSPNet (Pyramid Scene Parsing Network): using pyramid pooling modules to aggregate contextual information at different scales. In addition, there are technologies designed specifically for specific challenges, such as sliding window strategies or multi-scale fusion methods used when processing high-resolution remote sensing images, as well as spatiotemporal modeling techniques that consider time series changes.

[0003] Although remote sensing image segmentation technology has made significant progress, there are still some challenges and shortcomings: data acquisition and annotation are difficult, and it is costly to obtain high-quality remote sensing images; targets of different sizes may exist simultaneously in the same remote sensing image, such as buildings, roads, and farmland, and how to effectively deal with this multi-scale problem is currently a difficulty; images taken at different times will cause the same objects to show different characteristics in the image due to changes in lighting conditions (such as morning and evening, seasonal changes); certain specific types of objects appear less frequently in the image, and it is difficult to fully learn the characteristics of these categories during model training, resulting in decreased prediction accuracy; the generalization ability is limited, and the trained model performs poorly on data collected in different regions or by different sensors, and has poor cross-domain adaptability. When encountering new types of objects that have never been seen before, the existing model may not be able to correctly identify them.

[0004] Neural symbolic methods endow neural networks with symbolic reasoning capabilities, enabling them to build more robust artificial intelligence (AI) systems. In recent years, with the increasing demand for multimodal data processing, vision-language models (VLMs) have become indispensable in advancing neural symbolic systems. Existing VLMs have demonstrated remarkable performance in image segmentation tasks by enabling effective multimodal perception and commonsense reasoning in multiple domains.

[0005] Researchers pre-train on large-scale image-text corpora to develop VLMs that can learn universal cross-modal representations for subsequent vision-language tasks. However, large-scale pre-training requires a lot of computational resources. At the same time, the remarkable progress of large language models (LLMs) has driven the exponential growth of the size of VLMs, which further increases the cost of training VLMs from scratch. To cope with the huge demand for computational resources, researchers have developed modular pre-training techniques such as Q-former. Through a trainable lightweight projection component that maps the input or output of one modality to another modality, VLMs are able to leverage well-trained single-modal large models (LMs). Although modular pre-training techniques significantly reduce training costs, the computational and memory overhead of large-scale VLMs is still high.

[0006] In recent years, model compression has attracted much attention due to its ability to reduce the size of neural networks while maintaining their capabilities. To popularize powerful large models (LMs), researchers have made extensive efforts, including knowledge distillation, model pruning, quantization, and low-rank decomposition. As a major approach, model pruning has received widespread attention. Researchers have subsequently proposed a variety of methods and achieved remarkable performance.

[0007] Current VLM pruning methods suffer from a significant imbalance in weight and gradient distribution between language and vision submodules, making it difficult to determine the appropriate compression ratio for different components across modalities. To address this issue, researchers introduced iterative pruning, which performs pruning by learning pruning masks, which involves a lot of computation and retraining of the inverse Hessian matrix. To enhance the overall understanding of layer-wise pruning, Sung et al. proposed an efficient layer-wise pruning (ECoFLaP) method that adaptively calculates the sparsity rate of each layer based on the global importance score. However, ECoFLaP focuses on calculating the importance of a single layer in the entire neural network, ignoring the correlation between sub-network layers of different modalities. Summary of the invention

[0008] In view of this, the present invention constructs a modality collaborative hierarchical pruning (MoCo) method for remote sensing image segmentation. Specifically, MoCo is designed for Q-former-based VLMs, and estimates the importance of layers through cross-modal guidance. MoCo uses the original Q-former of VLMs as a query item to perform cross-modal alignment, effectively capturing the dependencies between sub-network layers of different modalities, thereby grouping layers of different modalities to collaboratively estimate the importance of layers. Therefore, the present application utilizes global cross-modal guidance to obtain sparsity rate and adaptively performs hierarchical unstructured weight pruning. Since MoCo does not introduce additional parameters, VLMs can be efficiently compressed in a single round of pruning.

[0009] In addition, existing layered pruning methods usually evaluate the importance of layers by adding noise. Given a set of calibration data, the importance score of each layer can be calculated based on the change in the loss value after adding noise. However, this method cannot preserve the original knowledge of the model because it inevitably converges to a specific area. In order to retain knowledge and generalization ability during model compression, this application introduces a multi-view estimation mechanism to support the evaluation of layer importance. The method consists of two main steps. First, the samples of a given training dataset are divided into multiple clusters, and representative data points are selected from each cluster as calibration data, thereby providing a comprehensive perspective for the compression process. Secondly, a soft target of the original model is introduced to supplement the layer importance evaluation process, thereby utilizing the knowledge of the original model and forcing the compressed model to imitate its behavior.

[0010] To achieve the above purpose, the image segmentation method based on hierarchical pruning of a modality collaborative visual language large model disclosed in the present application includes the following steps:

[0011] Acquisition of remote sensing images;

[0012] Given a calibration dataset and the visual encoder E v (·; W v ),in N is the number of layers of the visual encoder, is the weight of the visual encoder, For visual input, is the language input, T is the number of data in the dataset, and the pre-trained Q-Former is used to convert the encoded visual information of each block in the image into a visual representation with language information;

[0013] Based on the output of each block, layers with similar functions in different modalities are grouped;

[0014] Introducing KL divergence as an auxiliary objective to support parameter importance estimation;

[0015] Calculate the modal co-importance score and assign the modal co-importance score to all weights in the same group;

[0016] After obtaining the importance score of each weight in each layer, calculate the sparsity of each weight;

[0017] In the layer sparsity p i Based on the calibration dataset D, find the optimal sparse weight that can maximize the model performance

[0018] According to the layer sparsity p i and Perform hierarchical pruning of large visual language models;

[0019] The pruned visual language model performs image segmentation and outputs the image segmentation result.

[0020] Preferably, the optimal sparse weight The calculation is as follows:

[0021]

[0022] Among them, F imp is a function that calculates the importance score of each parameter in the weight matrix, W i The weight after pruning is the sparse weight of the i-th layer, W i is the weight of the i-th layer of the full-capacity model, represents the local target of the i-th layer, p i represents the target sparsity of the i-th layer.

[0023] Preferably, the Q-Former-based visual-language layer grouping utilizes a pre-trained Q-Former to convert the encoded visual information in each block into a visual representation with language information, including:

[0024]

[0025] in represents the encoded visual representation of visual block i, represents the encoded visual representation of visual block i-1, and the weight is and is the coded language representation of language chunk j, is the coded language representation of language block j-1, and its weight is and E q is a weight W q Q-Former is used to extract visual information from each output of visual block i and transform it into a representation with language information

[0026] Preferably, the method of grouping functionally similar layers in different modalities based on the output of each block comprises: given a representation set with language information extracted from the output of the visual sub-model block and the set of encoded language representations of the language submodel block Cross-attention is introduced to measure the correlation of encoding representations of different modalities, so that layers of different modalities are grouped according to the cross-attention scores of the encoding representations:

[0027]

[0028] Where A represents the correlation matrix between the visual and language sub-model layers in the visual-language model, A i，j represents the correlation score between the visual sub-model block i and the language sub-model block j, G is a grouping set, where G k denotes the kth group of layers, k∈[1,min(N,M)], M is the number of layers of the language encoder, and for i from N to 1, matches the i-th block of the visual submodel with its most relevant block j, where j∈J and J={1,2,...,M}. For unmatched blocks, group the layers in each block separately and label them as min(N,M)+1 To G max(N,M) .

[0029] Preferably, in order to provide parameter importance estimation in the model compression process, firstly, a given pre-training dataset D * The samples are divided into multiple clusters, namely Then, for each cluster Sort and select the data point X closest to the center i As representative samples, compose a multi-view calibration dataset

[0030] Preferably, after obtaining the multi-view calibration dataset D′, the output probability distribution of the original model is extracted as soft labels to capture the comprehensive knowledge of the model in various fields.

[0031] Preferably, the introduction of KL divergence as an auxiliary objective to support parameter importance estimation includes:

[0032] Given the weights of the full capacity model Where W i is the weight of the i-th layer, and is calculated for the global goal of the visual language model and consistency goals The zero-order approximate gradient of the weight W is estimated i Importance:

[0033] h i =f i (W i ·h (i-1) θ i ),

[0034]

[0035] where h i is the output of the i-th layer, containing the knowledge of the full capacity model; f i represents the activation function or nonlinear transformation of the i-th layer, θ i is an additional parameter, is the objective function, η is the randomly generated Gaussian noise, d is the data in the standard data set, δ is the direction and scale of the Gaussian noise, It is W i The importance score, h (i―1) is the output of layer i-1; by applying W in two opposite directions i Add Gaussian noise and calculate weight W i To target The impact of W i , where ∈ is the disturbance amplitude; therefore, Used as W i The importance score of the task is a combination of task performance and consistency with the full-capacity model.

[0036] Preferably, to achieve modal co-pruning of large visual language models, modal co-importance scores are calculated in three ways:

[0037] product:

[0038] Sum:

[0039] maximize:

[0040] is the importance score of weight W, G k represents the kth group layer;

[0041] Then, the modal co-importance score is assigned to all weights within the same group:

[0042]

[0043] Preferably, after obtaining the importance score of each weight in each layer, the sparsity of each weight is calculated:

[0044]

[0045] Where p is the target sparsity of the full capacity model, W j is the weight of the jth layer, |W| is the number of parameters, N i According to W i The number of retained parameters calculated by the importance score, finally, the sparsity p of each layer i is obtained i .

[0046] Compared with the prior art, this application has the following beneficial effects:

[0047] This application is used to build cross-modal dependencies between VLM layers from different modalities, thereby supporting layer-by-layer pruning based on global cross-modal information.

[0048] This application introduces a multi-view estimation mechanism to enhance the generation of compressed models. By using the soft targets of the original model, the compressed model can benefit from the original model and preserve its behavioral characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Framework diagram of the present invention. DETAILED DESCRIPTION

[0050] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0051] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0052] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0053] The technical solutions provided in the embodiments of the present application involve technologies such as machine learning and natural language processing of artificial intelligence, which are specifically introduced and explained through the following embodiments.

[0054] Before introducing the embodiments of the present application, some terms involved in the present application are first explained.

[0055] 1. Visual Language Models (VLMs): Aims to build a multimodal base model to process visual and language information. Previous VLMs mainly learn multimodal base models by end-to-end pre-training on large-scale image-text pairing datasets. In order to adapt to different tasks, researchers have developed a variety of structures, including encoder-decoder, dual encoder, and fusion encoder. With the development of large models (LMs), Transformer-based VLMs have recently received widespread attention due to their outstanding performance in various visual language tasks. However, with the increase in the size of LMs parameters, it will inevitably bring a lot of computational and memory overhead, making pre-training challenging.

[0056] In order to reduce the cost and efficiently utilize the existing pre-trained unimodal large models, the concept of modular pre-training is proposed. During the training process, the separately pre-trained vision and language large models are usually fixed as modality encoders, decoders or backbone networks. Through a trainable lightweight projection component, the input or output of one modality is mapped to the other modality, so that the unimodal large model can bridge the gap between modalities and perform multimodal tasks. As one of the classic architectures, the query transformer (Q-former) is widely used to connect unimodal models. Specifically, it transfers the output of the vision large model to the input of the language large model by pre-training a set of learnable queries. Despite the reduced training cost, the deployment and inference of VLMs still require a lot of computing resources and memory. In this study, we studied model pruning for Q-former-based visual language models (VLMs) to reduce the deployment and inference costs of VLMs.

[0057] 2. Pruning of large models based on Transformer architecture: aims to compress the model by eliminating components or individual parameters, thereby significantly reducing the model's memory and computing resource consumption. Previous pruning methods mainly focused on unimodal models, while few studies have focused on pruning multimodal models. With the rapid development of large models, visual language models (VLMs) have become an important method for handling cross-modal tasks, and the demand for their efficiency has become increasingly urgent. Therefore, the pruning of VLMs has received more and more attention. Existing pruning methods can be divided into token-based pruning methods and model-based pruning methods.

[0058] Current pruning strategies are essentially influenced by the diverse architectural design of VLMs. This application is designed for a modular multimodal learning framework based on Q-Former (Enhancing vision-language understanding with advanced large language models. In: The Twelfth International Conference on Learning Representations (2024). https: / / openreview.net / forum?id=1tZbq88f27) for visual-language tasks such as BLIP-2 (BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (eds.) Proceed-ings of the 40th International Conference on Machine Learning), which contains pre-trained FlanT5 and ViT for encoding text and visual information, respectively, and a pre-trained query transformer (Q-Former) module to bridge the modality gap. BLIP-2 based on Q-Former has achieved remarkable performance on a variety of visual language tasks due to its strong zero-shot capability. Given the language encoder E in BLIP-2 l (·; W l ) and the visual encoder Ev ( ·; W v), and their weights are and Where M and N denote the number of layers in each encoder. With these pre-trained backbones frozen, a lightweight query transformer module E is introduced. q (·; W q ), whose weight set is W q Q-Former extracts visual information and transforms it into a visual representation with language information while removing unimportant visual information. Let the language and visual input pair be (x l ,x v ), where the visual input x v Input the visual encoder and Q-Former in sequence to obtain a visual representation with language information. v =Eq (E v (x v ; W v );W q ). Then, the output of BLIP-2 can be expressed as E l ([o v , x l ]; W l ).

[0059] Current pruning strategies are inherently influenced by the diverse architectural design of VLMs. In this application, MoCo is designed for a Q-Former-based modular multimodal learning framework for visual-language tasks such as BLIP-2, which contains pre-trained FlanT5 and ViT for encoding textual and visual information, respectively, and a pre-trained query transformer (Q-Former) module to bridge the modality gap. BLIP-2 based on Q-Former has achieved remarkable performance on multiple visual-language tasks due to its strong zero-shot capability. Given the language encoder E in BLIP-2 l (·; W l ) and the visual encoder E v (·; W v ), and their weights are and Where M and N denote the number of layers in each encoder. While keeping these pre-trained backbones, a lightweight query transformer module E is introduced. q (·; W q ), whose weight set is W q Q-Former extracts visual information and transforms it into a visual representation with language information while removing unimportant visual information. Let the language and visual input pair be (x l ,x v ), where the visual input x v Input the visual encoder and Q-Former in sequence to obtain a visual representation with language information. v =E q (E v (x v ; W v );W q ). Then, the output of BLIP-2 can be expressed as E l ([o v , x l ]; W l ).

[0060] In this application, we aim to prune the BLIP-like visual language model based on Q-Former. Given that the Q-Former architecture accounts for only a small portion (about 5%) of the overall parameters of the VLM, this application keeps Q-Former unchanged. Traditional layer-by-layer pruning methods attempt to reduce the number of layers to a certain level within a predetermined sparsity p. i and a set of calibration data Based on this, we find the optimal sparse weights that can maximize the model performance. Right now:

[0061]

[0062] Among them, F imp is a function that calculates the importance score of each parameter in the weight matrix, W i The weight after pruning, represents the local target of the i-th layer, and p i represents the target sparsity of the i-th layer. Usually, p i Setting the same value for all layers can lead to suboptimal performance in visual language models. Although subsequent researchers have proposed to determine layer sparsity based on the importance of layers in VLM, few consider the interaction between modalities within visual language models. To address this problem, this application proposes a modality coordination (MoCo) framework to perform layer-by-layer pruning of VLM.

[0063] 1. Visual language layer grouping based on Q-Former

[0064] The Q-Former-based visual-language layer grouping aims to capture the modal interactions in large-scale visual-language models, thereby estimating the sparsity and parameter importance of layers in a modality-cooperative manner. Specifically, given a calibration dataset and the visual encoder E v (·; W v ),in The calibration dataset is a remote sensing image dataset, including but not limited to AID (Aerial Image Dataset: an aerial image dataset provided by the Chinese Academy of Sciences, containing 10,000 images divided into 30 categories), DeepGlobe Road Extraction Challenge (a dataset focused on extracting road networks from high-resolution satellite images, which is helpful for the research and application of transportation networks), etc. Q-Former-based visual language layer grouping uses pre-trained Q-Former to convert the encoded visual information in each block into a visual representation with language information:

[0065]

[0066] in represents the encoded visual representation of visual block i, and the weight is and is the encoded language representation of language block j, and the weight is and E q is a weight W q Q-Former is used to extract visual information from each output of visual block i and transform it into a representation with language information

[0067] Therefore, based on the output of each block, layers with similar functions in different modalities are grouped. Given a set of representations with language information extracted from the output of the visual sub-model block and the set of encoded language representations of the language submodel block Cross-attention is introduced to measure the correlation of encoding representations of different modalities, so that layers of different modalities are grouped according to the cross-attention scores of the encoding representations:

[0068]

[0069] where A represents the correlation matrix between the visual and language sub-model layers in the VLM, and A i，j represents the correlation score between the visual sub-model block i and the language sub-model block j. G is a grouping set, where G k denotes the kth group of layers, k∈[1,min(N,M)]. For i from N to 1, we match the i-th block of the visual submodel with its most relevant block j, where j∈J and J={1,2,...,M}. For unmatched blocks, the layers in each block are grouped separately and labeled as min(N,M)+1 To G max(N,M) .

[0070] 2. Multi-perspective evaluation mechanism

[0071] In order to retain the performance of the compression model to the maximum extent, this application adopts a multi-view estimation mechanism to support the consistency between the compression model and the original model. The parameter importance estimation of the traditional layer-by-layer pruning method relies on a set of calibration data sets D randomly selected from the VLM pre-training data set, but the random selection often ignores the knowledge domain involved in the sample, which may lead to overfitting of the compression model. In addition, the parameter estimation of layer-by-layer pruning in the past focused too much on the performance of the compression model on the calibration data (i.e., the accuracy of the given hard labels), which inevitably caused knowledge loss and limited the comprehensive understanding of the original model.

[0072] In order to provide a more comprehensive estimate of parameter importance during model compression, we firstly assign a given pre-training dataset D *The samples are divided into multiple clusters, namely Then, for each cluster Sort and select the data point X closest to the center i As representative samples, compose a multi-view calibration dataset

[0073] After obtaining the multi-view calibration dataset D′, the output probability distribution of the original model is extracted as a soft label to capture the comprehensive knowledge of the model in various fields. In order to enhance the consistency between the compressed model and the original model, the KL divergence is introduced as an auxiliary objective to support parameter importance estimation. Specifically, given the weights of the full-capacity model Where W i is the weight of the i-th layer, and is calculated for the VLM global target and consistency goals The zero-order approximate gradient of the weight W is estimated i Importance:

[0074] h i =f i (W i ·h (i-1) θ i ), (7)

[0075]

[0076] where h i is the output of the i-th layer, containing the knowledge of the full capacity model; f i represents the activation function or nonlinear transformation of the i-th layer, θ i is an additional parameter. By i Add Gaussian noise and calculate weight W i To target The impact of W i , where ∈ is the disturbance amplitude. Therefore, Used as W i The importance score of the task is a combination of task performance and consistency with the full-capacity model.

[0077] To implement modal co-pruning of VLM, the modal co-importance score is calculated in three ways: (1) Product: (2) Sum: (3) Maximization: Then, the modal co-importance score is assigned to all weights within the same group:

[0078] After obtaining the importance score of each weight in each layer, calculate the sparsity of each weight:

[0079]

[0080] Where p is the target sparsity of the full capacity model, N i According to W i The number of retained parameters for calculating the importance score of . Finally, the sparsity p of each layer i is obtained i , and substitute it into equation (1) to perform layer-by-layer pruning.

[0081] The pruned model is used to perform image segmentation on the remote sensing image and output the segmented image.

[0082] Compared with the prior art, this application has the following beneficial effects:

[0083] This application is used to build cross-modal dependencies between VLM layers from different modalities, thereby supporting layer-by-layer pruning based on global cross-modal information.

[0084] This application introduces a multi-view estimation mechanism to enhance the generation of compressed models. By using the soft targets of the original model, the compressed model can benefit from the original model and preserve its behavioral characteristics.

[0085] As used herein, the word "preferred" is intended to be used as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" is intended to present concepts in a specific way. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the naturally included permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0086] Moreover, although the present disclosure has been shown and described with respect to one or implementations, those skilled in the art will think of equivalent variations and modifications based on the reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations, and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if the structure is not equivalent to the disclosed structure of the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that may be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0087] The functional units in the embodiments of the present invention may be integrated into a processing module, or each unit may exist physically separately, or multiple or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc. The above-mentioned devices or systems may execute the storage method in the corresponding method embodiment.

[0088] To sum up, the above embodiment is an implementation mode of the present invention, but the implementation mode of the present invention is not limited by the embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. An image segmentation method based on hierarchical pruning of a large modality collaborative visual language model, characterized in that: The following steps are involved: Acquisition of remote sensing images; Given a calibration dataset and the visual encoder E v (·; W v ),in N is the number of layers of the visual encoder, is the weight of the visual encoder, For visual input, is the language input, T is the number of data in the dataset, and the pre-trained Q-Former is used to convert the encoded visual information of each block in the image into a visual representation with language information; Based on the output of each block, layers with similar functions in different modalities are grouped; Introducing KL divergence as an auxiliary objective to support parameter importance estimation; Calculate the modal co-importance score and assign the modal co-importance score to all weights in the same group; After obtaining the importance score of each weight in each layer, calculate the sparsity of each weight; In the layer sparsity p i Based on the calibration dataset D, find the optimal sparse weight that can maximize the model performance According to the layer sparsity p i and Perform hierarchical pruning of large visual language models; The pruned visual language model performs image segmentation and outputs the image segmentation result.

2. The image segmentation method based on hierarchical pruning of a modality collaborative visual language large model according to claim 1 is characterized in that: The optimal sparse weight The calculation is as follows: where F imp is a function that calculates the importance score of each parameter in the weight matrix, W i The weight after pruning is the sparse weight of the i-th layer, W i is the weight of the i-th layer of the full-capacity model, represents the local target of the i-th layer, p i represents the target sparsity of the i-th layer.

3. The image segmentation method based on hierarchical pruning of a modal collaborative visual language large model according to claim 2 is characterized in that: The Q-Former-based visual-language layer grouping utilizes a pre-trained Q-Former to convert the encoded visual information in each block into a visual representation with language information, including: in represents the encoded visual representation of visual block i, represents the encoded visual representation of visual block i-1, and the weight is and is the coded language representation of language chunk j, is the coded language representation of language block j-1, and its weight is and E q is a weight W q Q-Former is used to extract visual information from each output of visual block i and transform it into a representation with language information 4. The image segmentation method based on hierarchical pruning of a modality collaborative visual language large model according to claim 3 is characterized in that: Based on the output of each block, the functionally similar layers in different modalities are grouped, including: given a representation set with language information extracted from the output of the visual sub-model block and the set of encoded language representations of the language submodel block Cross-attention is introduced to measure the correlation of encoding representations of different modalities, so that layers of different modalities are grouped according to the cross-attention scores of the encoding representations: Where A represents the correlation matrix between the visual and language sub-model layers in the visual-language model, A i，j represents the correlation score between the visual sub-model block i and the language sub-model block j, G is a grouping set, where G k denotes the kth group of layers, k∈[1,min(N,N)], M is the number of layers of the language encoder, and for i from N to 1, the i-th block of the visual submodel is matched with its most relevant block j, where j∈J and J={1,2,...,M}. For unmatched blocks, the layers in each block are grouped separately and labeled as min(N,M)+1 To G max(N,M) .

5. The image segmentation method based on hierarchical pruning of a modality collaborative visual language large model according to claim 4 is characterized in that: In order to provide parameter importance estimation in the model compression process, we first give a pre-training dataset D * The samples are divided into multiple clusters, namely Then, for each cluster Sort and select the data point X closest to the center i As representative samples, compose a multi-view calibration dataset 6. The image segmentation method based on hierarchical pruning of a modality collaborative visual language large model according to claim 5 is characterized in that: After obtaining the multi-view calibration dataset D′, the output probability distribution of the original model is extracted as soft labels to capture the comprehensive knowledge of the model in various fields.

7. The image segmentation method based on hierarchical pruning of a modality collaborative visual language large model according to claim 6 is characterized in that: The introduction of KL divergence as an auxiliary objective to support parameter importance estimation includes: Given the weights of the full capacity model Where W i is the weight of the i-th layer, and is calculated for the global goal of the visual language model and consistency goals The zero-order approximate gradient of the weight W is estimated i Importance: h i =f i (W i ·h (i-1 );θ i ), where h i is the output of the i-th layer, containing the knowledge of the full capacity model; f i represents the activation function or nonlinear transformation of the i-th layer, θ i is an additional parameter, is the objective function, η is the randomly generated Gaussian noise, d is the data in the standard data set, δ is the direction and scale of the Gaussian noise, It is W i The importance score, h (i―1) is the output of layer i-1; by applying W in two opposite directions i Add Gaussian noise and calculate weight W i Impact W on target L i , where ∈ is the disturbance amplitude; therefore, Used as W i The importance score of the task is a combination of task performance and consistency with the full-capacity model.

8. The image segmentation method based on hierarchical pruning of a modality collaborative visual language large model according to claim 7 is characterized in that: To achieve modal co-pruning of large visual language models, the modal co-importance score is calculated in three ways: product: Sum: maximize: is the importance score of weight W, G k represents the kth group layer; Then, the modal co-importance score is assigned to all weights within the same group:

9. The image segmentation method based on hierarchical pruning of a modality collaborative visual language large model according to claim 8 is characterized in that: After obtaining the importance score of each weight in each layer, calculate the sparsity of each weight: Where p is the target sparsity of the full capacity model, W j is the weight of the jth layer, |W| is the number of parameters, N i According to W i The number of retained parameters calculated by the importance score, finally, the sparsity p of each layer i is obtained i .

Images