Attention clipping method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an attention clipping method and device, electronic equipment and a storage medium, and relates to the technical field of cross-modal large model optimizing.The method comprises the steps that a multi-modal sample pair is collected to serve as a verification set, and a harmful attention head is positioned according to a resource condition determination algorithm of a current scene; if the genetic algorithm is adopted, coding all attention heads into binary vectors, and optimizing the binary vectors on the verification set by using the genetic algorithm and a fitness function to obtain positioning information; if a back propagation algorithm is adopted, a suppression coefficient is set for each attention head, and a comparison loss function is used for training on the verification set until a set condition is met to obtain positioning information; and adjusting numerical values in the attention weight matrix according to the positioning information of the harmful attention head to obtain an adjusted attention weight matrix. The problems that in the prior art, harmful attention heads cannot be automatically and flexibly cut, and downstream task performance is poor are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal large model optimization, and particularly to an attention pruning method, device, electronic device, and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, vision-language cross-modal large models (such as CLIP) have demonstrated powerful cross-modal representation capabilities in fields such as image-text retrieval, zero-shot classification, and visual question answering. However, due to the wide range and uneven quality of the training data sources for such models, the image encoder contains attention heads that are irrelevant or harmful to the target task, and these attention heads encode biases or noises, thus affecting the performance of downstream tasks.

[0003] However, traditional methods lack a systematic attention pruning mechanism, especially an automated pruning strategy for large-scale vision-language cross-modal models, and it is difficult to quickly and correctly locate and prune harmful attention heads to improve the performance of the model in downstream tasks.

[0004] In addition, different application scenarios have different requirements for computing resources and data resources. Traditional methods cannot flexibly adapt to environments with limited floating-point computing capabilities, scarce computing resources, or insufficient high-quality data, resulting in challenges in model optimization and deployment.

[0005] Therefore, there is an urgent need for an attention pruning method that can prune harmful attention heads without modifying model parameters, improve model performance, and at the same time adapt to different resource limitations. Summary of the Invention

[0006] Embodiments of the present invention provide an attention pruning method to solve the problem that the existing technology lacks an automated and flexible attention pruning mechanism, resulting in the inability of cross-modal large models to prune harmful attention heads and poor performance of downstream tasks. The technical solutions are as follows: According to one aspect of the present invention, an attention pruning method includes: collecting multi-modal sample pairs as a validation set, and determining whether to use a genetic algorithm or a backpropagation algorithm to locate harmful attention heads according to the resource conditions of the current scenario; if the genetic algorithm is used, encoding all attention heads as binary vectors, and using the genetic algorithm and a fitness function to optimize the binary vectors on the validation set to obtain the location information of harmful attention heads; if the backpropagation algorithm is used, setting an inhibition coefficient for each attention head, and using a contrast loss function to train on the validation set until a set condition is reached to obtain the location information of harmful attention heads; adjusting the values in the attention weight matrix according to the location information of the harmful attention heads to obtain an adjusted attention weight matrix.

[0007] In one embodiment, determining to use a genetic algorithm or a backpropagation algorithm for locating harmful attention heads according to the resource conditions of the current scenario is achieved through the following steps: If the current scenario does not have floating-point operation capabilities, the genetic algorithm is used for locating harmful attention heads; if the current scenario has floating-point operation capabilities, the backpropagation algorithm is used for locating harmful attention heads.

[0008] In one embodiment, encoding all attention heads as binary vectors is achieved through the following steps: Count the total number of attention heads in the image encoder of the cross-modal model to be optimized, and assign a unique number to each of the attention heads; Generate a binary vector according to the total number of the attention heads; A value of 1 for an element in the binary vector indicates that the corresponding attention head is a harmful attention head to be pruned.

[0009] In one embodiment, using a genetic algorithm and a fitness function to optimize the binary vector on the validation set to obtain the location information of harmful attention heads is achieved through the following steps: Randomly generate a set number of different binary vectors as the initial population; The set number is 15 - 30% of the total number of attention heads; Define a fitness function, and use the genetic algorithm and the tournament selection strategy to iteratively perform crossover and mutation on the initial population; The fitness function is used to evaluate the performance of each binary vector on the validation set; Determine the optimal binary vector according to the fitness function, and obtain the location information of harmful attention heads according to the optimal binary vector.

[0010] In one embodiment, setting an inhibition coefficient for each of the attention heads is achieved through the following steps: Set a learnable parameter for each attention head in the image encoder of the cross-modal model to be optimized as the inhibition coefficient of the attention head; The initial value of the inhibition coefficient is set to 1 indicating no pruning; Define an update formula for the inhibition coefficient according to the requirements of the cross-modal model to be optimized; The update formula is used to control the change of the inhibition coefficient between 0 representing pruning and 1 representing no pruning.

[0011] In one embodiment, setting an inhibition coefficient for each of the attention heads and using a contrast loss function to train on the validation set until a set condition is reached to obtain the location information of harmful attention heads is achieved through the following steps: Use the same contrast loss function as the cross-modal model to be optimized and set a convergence condition, and perform backpropagation training on the inhibition coefficient using the validation set and the contrast loss function; After the contrast loss function reaches the convergence condition, obtain the optimal inhibition coefficient, and obtain the location information of harmful attention heads according to the optimal inhibition coefficient; During the training process, control the parameters of the cross-modal model to be optimized to remain unchanged.

[0012] In one embodiment, adjusting the values in the attention weight matrix according to the positioning information of the harmful attention head to obtain an adjusted attention weight matrix is achieved through the following steps: According to the positioning information of the harmful attention head, performing an inhibition operation on the values corresponding to the harmful attention head in the attention weight matrix of the cross-modal model to be optimized; the inhibition operation includes multiplying by a coefficient less than 1; performing row normalization on the inhibited attention weight matrix so that the output of the pruned attention head conforms to the row normalization constraint of softmax, obtaining a new attention weight matrix.

[0013] According to one aspect of the present invention, an attention pruning device includes: a validation set preparation module for collecting multimodal sample pairs as a validation set and determining whether to use a genetic algorithm or a backpropagation algorithm for positioning harmful attention heads according to the resource conditions of the current scenario; a genetic algorithm positioning module for, if the genetic algorithm is used, encoding all attention heads as binary vectors and using the genetic algorithm and a fitness function to optimize the binary vectors on the validation set to obtain the positioning information of the harmful attention heads; a backpropagation positioning module for, if the backpropagation algorithm is used, setting an inhibition coefficient for each attention head and using a contrastive loss function to train on the validation set until a set condition is reached to obtain the positioning information of the harmful attention heads; an attention head pruning module for adjusting the values in the attention weight matrix according to the positioning information of the harmful attention heads to obtain an adjusted attention weight matrix.

[0014] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, where a computer-readable instruction is stored on the memory; the computer-readable instruction is executed by one or more of the processors, enabling the electronic device to implement the attention pruning method as described above.

[0015] According to one aspect of the present invention, a storage medium stores a computer-readable instruction, and the computer-readable instruction is executed by one or more processors to implement the attention pruning method as described above.

[0016] The beneficial effects brought by the technical solution provided by the present invention are: In the above technical solution, the present invention first collects multi-modal sample pairs as the validation set, and selects a genetic algorithm or a backpropagation algorithm to locate harmful attention heads according to the current scenario resource conditions. If the scenario does not have floating-point operation capabilities, all attention heads are encoded as binary vectors, and a set number of different binary vectors are randomly generated as the initial population. By defining a fitness function, genetic algorithm and tournament selection strategy are used to iteratively cross-mutate, and the optimal binary vector is determined according to the fitness function, and then the harmful attention head location information is obtained. If the scenario has floating-point operation capabilities, learnable parameters are set for each attention head as the suppression coefficient, the same contrast loss function as the cross-modal model to be optimized is used and the convergence condition is set, and the suppression coefficient is trained by backpropagation on the validation set. When the contrast loss function reaches the convergence condition, the optimal suppression coefficient is obtained, so as to determine the harmful attention head location information. Finally, according to the harmful attention head location information, the corresponding values of the harmful attention heads in the attention weight matrix are suppressed to obtain the adjusted attention weight matrix. By reasonably pruning harmful attention heads, the performance and efficiency of the cross-modal model are effectively improved, providing a new idea for model optimization, and effectively solving the problem in the prior art that due to the lack of an automated and flexible attention pruning mechanism, harmful attention heads cannot be pruned in cross-modal large models, resulting in poor performance of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments of the present invention. Obviously, the following drawings are only some embodiments of the present invention, and those skilled in the art can obtain other drawings without creative efforts based on these drawings.

[0018] Figure 1 is a flowchart of an attention pruning method shown according to an exemplary embodiment; Figure 2 is a schematic structural diagram of an attention pruning method in an application scenario; Figure 3 is a block diagram of an attention pruning device shown according to an exemplary embodiment; Figure 4 is a hardware structure diagram of an electronic device shown according to an exemplary embodiment; Figure 5 is a block diagram of an electronic device shown according to an exemplary embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] Embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present invention, and should not be construed as a limitation of the present invention.

[0020] Those skilled in the art of the present technology can understand that, unless specifically stated otherwise, the singular forms "a", "an", "the", and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure means the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any unit and all combinations of one or more related listed items.

[0021] The present invention provides an attention pruning method. By analyzing the influence of each attention head in the multi-head attention module on the output features, it locates and suppresses the attention heads that are harmful to performance, thereby improving the quality of the output features. It effectively solves the problem in the prior art that there is a lack of an automated and flexible attention pruning mechanism, resulting in the inability of cross-modal large models to prune harmful attention heads and poor performance of downstream tasks. The attention pruning method is applicable to an attention pruning device, and the attention pruning device can be an electronic device. The attention pruning method in the embodiments of the present invention can be applied to various scenarios, such as the optimization of visual-language cross-modal large models, etc.

[0022] Please refer to Figure 1 , embodiments of the present invention provide an attention pruning method, and this method is applicable to an electronic device.

[0023] In the following method embodiments, for the convenience of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation thereto.

[0024] As Figure 1 shown, the method may include the following steps: Step 110, collect multi-modal sample pairs as a validation set, and determine to use a genetic algorithm or a backpropagation algorithm to locate harmful attention heads according to the resource conditions of the current scenario.

[0025] In a possible implementation, if the current scenario does not have floating-point operation capabilities, the genetic algorithm is used to locate the harmful attention heads. If the current scenario has floating-point operation capabilities, the backpropagation algorithm is used to locate the harmful attention heads.

[0026] Specifically, the method of locating harmful attention heads based on the genetic algorithm only depends on model inference and does not require training. Therefore, it is suitable for deployment on some edge devices with low-precision integer computing power, such as INT8 / INT16 chips, so as to achieve efficient attention pruning under limited computing resources.

[0027] Specifically, the method of locating harmful attention heads based on the backpropagation algorithm requires very few parameters to be trained and is suitable for lightweight edge devices with certain floating-point operation capabilities, such as mobile devices and embedded GPUs. This method introduces learnable parameters for each attention head and uses a contrastive loss function for training to obtain an independent pruning intensity coefficient for each attention head, so as to achieve more accurate attention pruning with certain floating-point operation capabilities.

[0028] In the above process, the embodiments of the present invention lay a solid foundation for the subsequent attention pruning process through reasonable data preparation and algorithm selection. Selecting a suitable algorithm according to the resource conditions of different scenarios can ensure the efficiency and applicability of the pruning process, and at the same time meet the requirements of different application scenarios.

[0029] Step 120, if the genetic algorithm is adopted, all attention heads are encoded as binary vectors, and the genetic algorithm and fitness function are used to optimize the binary vectors on the validation set to obtain the location information of the harmful attention heads.

[0030] In a possible implementation, the total number of attention heads in the image encoder of the cross-modal model to be optimized is counted, and a unique number is assigned to each attention head, and a binary vector is generated according to the total number of attention heads.

[0031] Among them, the value of the element in the binary vector being 1 indicates that the corresponding attention head is a harmful attention head to be pruned, and the value of the element being 0 indicates that the corresponding attention head does not need to be pruned.

[0032] In a possible implementation, a set number of different binary vectors are randomly generated as the initial population, a fitness function is defined, and the genetic algorithm and tournament selection strategy are used to iteratively perform crossover and mutation on the initial population. The optimal binary vector is determined according to the fitness function, and the location information of the harmful attention heads is obtained according to the optimal binary vector.

[0033] Among them, the set quantity is 15-30% of the total number of attention heads, and the fitness function is used to evaluate the performance of each binary vector on the validation set, such as the accuracy or recall rate in the downstream task of image-text matching.

[0034] Among them, in the genetic algorithm, the crossover operation is to select two binary vectors (parents), exchange them at a randomly selected crossover point to generate two new binary vectors (offspring). The crossover operation helps to increase the diversity of the population and explore more potential solutions.

[0035] Among them, the mutation operation is to randomly flip some bits of the binary vector (0 to 1, 1 to 0) to generate a new binary vector. The mutation operation helps to prevent the algorithm from falling into a local optimal solution and increases the possibility of finding the global optimal solution.

[0036] Among them, selection and iteration are to determine the optimal binary vector according to the fitness function, eliminate individuals with low fitness, and retain individuals with high fitness (the elite individual retention strategy uses a tournament selection strategy with a scale of 3), so as to obtain a new generation of population. The number of evolutionary generations is set to 100, and the crossover, mutation, and selection operations are repeated until the exit condition is met, so as to obtain the location information of the harmful attention heads.

[0037] In the above process, the embodiment of the present invention can find the optimal solution in a large search space through the genetic algorithm by simulating the natural evolution process. The fitness function can accurately evaluate the performance of each binary vector on the validation set, so as to effectively locate the harmful attention heads.

[0038] Step 130, if the backpropagation algorithm is adopted, a suppression coefficient is set for each attention head, and the contrast loss function is used to train on the validation set until the set condition is reached to obtain the location information of the harmful attention heads.

[0039] In a possible implementation, a learnable parameter is set for each attention head in the image encoder of the cross-modal model to be optimized as the suppression coefficient of the attention head, and the update formula of the suppression coefficient is defined according to the requirements of the cross-modal model to be optimized.

[0040] Among them, the initial value of the suppression coefficient is set to 1 indicating no clipping, and the update formula is used to control the change of the suppression coefficient between 0 indicating clipping and 1 indicating no clipping.

[0041] In a possible implementation, the same contrast loss function as the cross-modal model to be optimized is used and the convergence condition is set. The suppression coefficient is trained by backpropagation using the validation set and the contrast loss function. When the contrast loss function reaches the convergence condition, the optimal suppression coefficient is obtained, and the location information of the harmful attention heads is obtained according to the optimal suppression coefficient.

[0042] During the training process, the parameters of the cross-modal model to be optimized are controlled to remain unchanged.

[0043] In the above process, the embodiments of the present invention can quickly adjust the suppression coefficient by using gradient information through the backpropagation algorithm, and can better measure the performance of the model through the comparison loss function, so as to accurately find the harmful attention heads.

[0044] Step 140: Adjust the values in the attention weight matrix according to the positioning information of the harmful attention heads to obtain an adjusted attention weight matrix.

[0045] In a possible implementation, according to the positioning information of the harmful attention heads, perform a suppression operation on the values corresponding to the harmful attention heads in the attention weight matrix of the cross-modal model to be optimized, and perform row normalization on the suppressed attention weight matrix to make the output of the pruned attention heads conform to the row normalization constraint of softmax, so as to obtain a new attention weight matrix.

[0046] Among them, the suppression operation includes multiplying by a coefficient less than 1, etc., which is not limited here.

[0047] In the above process, the embodiments of the present invention obtain an adjusted attention weight matrix by suppressing the values corresponding to the harmful attention heads and performing row normalization, which can effectively reduce the influence of the harmful attention heads on the output features, improve the performance of the model, and at the same time keep the model architecture unchanged, only replacing the attention weight matrix, which is convenient for the deployment and integration of the model.

[0048] Through the above process, the present invention realizes the systematic pruning of the attention heads in the cross-modal large model through four steps: data preparation and algorithm selection, harmful attention head positioning, attention weight matrix adjustment, and model verification and effect evaluation. Without modifying the model parameters, it hardly increases the inference overhead, and can significantly improve the performance of the model in downstream tasks. Through reasonable data preparation and algorithm selection, accurate harmful attention head positioning, effective attention weight matrix adjustment, and actual verification and effect evaluation, it provides new ideas and methods for the optimization of cross-modal large models, and is applicable to the needs of different application scenarios.

[0049] In an application scenario, for an image-text cross-modal model, attention pruning can specifically include the following steps: Step 1: Data preparation and algorithm selection.

[0050] Specifically, randomly sample 1000 pieces of data from an image-text dataset (such as MS COCO) as the validation set D for subsequent algorithm optimization.

[0051] Further, select an algorithm according to the resource conditions of the current scenario. If the current scenario does not have floating-point operation capabilities (such as edge devices), then select the attention pruning strategy based on genetic algorithm (AAT-GA); if it has floating-point operation capabilities (such as mobile devices, embedded GPUs), then select the attention pruning strategy based on backpropagation (AAT-BP).

[0052] In the above process, the embodiment of the present invention samples 1000 pieces of data as the validation set, which not only ensures the effectiveness of algorithm optimization but also reduces the computational amount, and is applicable to scenarios with limited data. Selecting the algorithm flexibly according to the resource conditions enables the technology to be widely applied to devices with different computing capabilities, improving the applicability of the technology.

[0053] Step 2: Locate harmful attention heads.

[0054] Specifically, if the attention pruning strategy based on genetic algorithm (AAT-GA) is selected: First, encode all the attention heads in the image encoder of the cross-modal model to be optimized into a binary vector of length N, where each digit represents the state of the corresponding attention head (1 represents pruning, 0 represents retention), and then use the genetic algorithm to search for the optimal 0-1 combination of the binary vector.

[0055] Among them, the search process of the genetic algorithm includes steps such as randomly generating the initial population, calculating the fitness, selection, crossover, and mutation, and finally obtaining the optimal binary vector.

[0056] In the above process, the embodiment of the present invention can find the optimal solution in a large search space through the genetic algorithm by simulating the natural evolution process, and the fitness function can accurately evaluate the performance of each binary vector on the validation set, thereby effectively locating harmful attention heads.

[0057] Specifically, if the attention pruning strategy based on backpropagation (AAT-BP) is selected: First, set a learnable parameter as the suppression coefficient for each attention head in the image encoder of the cross-modal model to be optimized, with the initial value set to 1, and then use the same contrast loss function as the cross-modal model to be optimized to perform backpropagation training on the validation set D. During the training process, control the model parameters unchanged. When the contrast loss function reaches the convergence condition, obtain the optimal suppression coefficient, and then obtain the harmful attention head location information.

[0058] In the above process, the embodiment of the present invention can quickly adjust the suppression coefficient by using the gradient information through the backpropagation algorithm, and can better measure the performance of the model through the contrast loss function, thereby accurately finding harmful attention heads.

[0059] Step 3: Adjust the attention weight matrix.

[0060] Specifically, according to the positioning information of the harmful attention heads, perform an inhibition operation on the values corresponding to the harmful attention heads in the attention weight matrix of the cross-modal model to be optimized, such as multiplying by a coefficient less than 1 (such as 0.1), and then perform row normalization on the inhibited attention weight matrix to make the output of the pruned attention heads conform to the row normalization constraint of softmax, obtaining a new attention weight matrix.

[0061] In the above process, through the inhibition and row normalization processing of the values corresponding to the harmful attention heads, the embodiments of the present invention can adjust the attention weight matrix, enabling the model to pay more attention to important attention heads, reducing the influence of noise and bias, and improving the performance and stability of the model.

[0062] Step 4: Model verification and performance evaluation.

[0063] Specifically, use the pruned model to verify the performance on downstream tasks (such as cross-modal retrieval, zero-shot classification), compare the performance metrics (such as recall rate, accuracy) of the model before and after pruning on various tasks, and evaluate the effectiveness of the attention pruning method. Through actual verification and performance evaluation, the improvement effect of the attention pruning method on the model performance can be intuitively understood, providing strong support for subsequent model optimization and application.

[0064] Through the above process, the embodiments of the present invention have described in detail the implementation process of the attention pruning technology. Through four steps of data preparation and algorithm selection, harmful attention head positioning, attention weight matrix adjustment, and model verification and performance evaluation, the systematic pruning of attention heads in the cross-modal large model is realized. The present invention does not require modifying model parameters, hardly increases the inference overhead, and can significantly improve the performance of the model in downstream tasks. In various cross-modal retrieval and classification tasks, the recall rate can be increased by up to 11.1%, and the accuracy in zero-shot classification tasks can be increased by 1.0% - 1.5%. The present invention has certain effectiveness and applicability, providing new ideas and methods for the optimization of cross-modal large models.

[0065] In an application scenario, Figure 2 shows a schematic structural diagram of optimizing the model using the attention pruning strategy (AAT) in the present invention.

[0066] As Figure 2 shown, use the attention ablation technique to optimize the attention heads of the large model based on the CLIP framework for processing text and image information. Figure 2 shows the input data of two modalities: text information and image information corresponding to the text.

[0067] Figure 2The large model in the CLIP framework contains two encoders: a text encoder for encoding the input text information and converting the text into a feature vector representation; an image encoder for encoding the input image information and converting the image into a feature vector representation.

[0068] Among them, an arrow between the two encoders indicates that the encoded feature vectors can be used for text-to-image retrieval tasks, that is, finding the corresponding image according to the text description.

[0069] Figure 2 The internal structure of the attention clipping technique is shown in detail below the image encoder. The input data is first processed in chunks, splitting the image or feature map into multiple small pieces, and then the attention weights are calculated through the multi-head attention module. The multi-head attention module contains multiple layers, and each layer contains multiple attention heads (attention head 1, attention head 2, attention head 3, attention head 4). Each attention head is used to calculate the attention weights between different parts of the input data; after each attention head, there is a feed-forward network (FFN) for further non-linear transformation of the features after attention calculation.

[0070] Among them, the attention clipping indicator is below each layer, used to indicate whether the attention head is retained or ablated (that is, removed or its effect is suppressed). In this way, certain attention heads can be selectively retained or removed to optimize the model. Each multi-head attention mechanism MHA module contains multiple attention heads, and the calculation results of multiple attention heads are integrated to obtain a richer feature representation.

[0071] Furthermore, after attention head clipping on the multi-modal large model of the CLIP framework, an optimized CLIP model is obtained. Using this optimized CLIP model for text-to-image and image-to-text tasks on mainstream English datasets such as MS COCO and Flickr30k, the mean recall rate (mean-R) is increased by 1% - 5%. On mainstream Chinese datasets such as COCO-CN and Flickr30k, the improved Chinese-CLIP model using the method of the present invention is significantly better than the original model in text-to-image and image-to-text tasks, and the mean recall rate (mean-R) is increased by 1% - 11%.

[0072] Furthermore, after pruning the attention heads of the CLIP model, an optimized CLIP model is obtained. The zero-shot classification experiment conducted using this optimized CLIP model on the ImageNet-1k dataset shows that the accuracy has increased by 1.0% - 1.5%, and at the same time, the improvement in retrieval ability is also maintained.

[0073] Among them, when pruning the attention heads of the CLIP model on an RTX 4090 GPU, the time taken to optimize the model by selecting the attention pruning strategy AAT-BP based on backpropagation is 2 - 8 minutes, and the time taken to optimize the model by selecting the attention pruning strategy AAT-GA of the genetic algorithm is 18 - 50 minutes.

[0074] Through the above process, in the embodiments of the present invention, by directly manipulating the attention weight matrix, the attention heads that have a negative impact on the output features are systematically located and pruned, thereby improving the performance of cross-modal downstream tasks. It can effectively screen out the attention heads beneficial to the task, improve the performance and efficiency of the model, while reducing unnecessary consumption of computing resources. It is applicable to scenarios with limited computing resources and data resources, and shows good applicability, improvement effect, and deployment feasibility under various model structures, languages, tasks, and device conditions.

[0075] In an application scenario, the attention pruning based on the genetic algorithm (AAT-GA) is used to optimize the model.

[0076] Specifically, first, 1000 pieces of data are randomly sampled from the image-text dataset MS COCO, that is, 1000 image-text pair samples are used as the validation set D for subsequent optimization of the genetic algorithm. Taking the CLIP-Base model as an example, its image encoder has 12 Transformer layers, and each Transformer layer has 12 attention heads, so there are a total of 144 attention heads. These 144 attention heads are encoded into a binary vector with a length of 144, and each number represents the state of the corresponding attention head (1 represents pruning, 0 represents retention).

[0077] Furthermore, 216 different binary vectors (15% of 144) are randomly generated as the initial population. The binary vector is generally set to 15% - 30% of the total number of attention heads. For each binary vector, according to its pruning / retention state of the attention heads, the attention heads in the image encoder of the CLIP-Base model are pruned.

[0078] Specifically, for the pruned attention heads, the values in their attention weight matrices are multiplied by 0.1 (inhibition coefficient). After pruning, each model is inferred on the validation set D, and the fitness corresponding to each binary vector is calculated.

[0079] Among them, the fitness function calculates the fitness value based on the cosine similarity between the $i$-th text in the validation set $D$ and its corresponding real image, and the cosine similarity between the $i$-th text and the $j$-th hard negative sample image in the corresponding hard negative sample set.

[0080] Furthermore, select the individual with the maximum fitness as the optimal individual, and determine whether the number of generations of evolution has reached the upper limit (100 generations). If so, exit and output the optimal individual; if not, continue; use the tournament selection strategy (select 1 out of every 3) to retain the individuals with high fitness and eliminate the individuals with low fitness.

[0081] Among them, the crossover probability is set to 0.9, the mutation probability is set to 0.5, perform the crossover and mutation operations, and expand the number of individuals in the population to the initial population size of 216; during the evolution process, update the hard negative sample set in the fitness function until the set exit condition is met.

[0082] Finally, obtain the optimal combination of attention head pruning - retention through the genetic algorithm, that is, locate which attention heads should be pruned. Prune the attention heads that should be pruned located by the genetic algorithm, and the optimization is completed after pruning. Use this model to perform downstream tasks (such as cross-modal retrieval, zero-shot classification, etc.) to verify its performance.

[0083] Through the above process, in this embodiment, the attention heads of the CLIP-Base model are optimized and pruned by the genetic algorithm. Through the crossover, mutation operations and fitness evaluation of the genetic algorithm, the optimal combination of attention head pruning is gradually selected. The fitness function is designed based on the cosine similarity between text and image and the similarity of hard negative samples, ensuring that the optimized model has better performance in downstream tasks. Finally, the optimal pruning scheme is obtained through the genetic algorithm, and the performance improvement of the optimized model in tasks such as cross-modal retrieval and zero-shot classification is verified.

[0084] In another application scenario, use attention pruning based on backpropagation (AAT-BP) for model optimization.

[0085] Specifically, randomly sample 1000 pieces of data in the image-text dataset MS COCO, that is, 1000 image-text pair samples as the validation set $D$ for subsequent optimization of the backpropagation algorithm. For each attention head in the image encoder of the CLIP-Base model, a learnable parameter is added to the model.

[0086] Specifically, the suppression coefficient is obtained through the following formula: ; Among them, the temperature coefficient is default set to 5.0. Each attention head is pruned to obtain the pruned model under the current suppression coefficient. The initial values of all learnable parameters are set to 1.0 to ensure that all attention heads are retained initially.

[0087] Furthermore, the same contrast loss function as in CLIP training is used, that is, minimizing the similarity score of incorrect matches and simultaneously maximizing the similarity score of correct matches. The learnable parameters are trained using the validation set D, and the contrast loss function is used for training on the validation set D; except for the learnable parameters, the remaining parameters of the model are fixed; during the training process, the values of the learnable parameters will gradually converge to 0 or 1, indicating that the corresponding attention head is completely pruned or retained.

[0088] Furthermore, the model is trained until the loss function converges. This model is the model after attention pruning, and the optimization is completed. Use this model to perform downstream tasks (such as cross-modal retrieval, zero-shot classification, etc.) to verify its performance.

[0089] Through the above process, this embodiment uses the backpropagation algorithm to optimize and prune the attention heads of the CLIP-Base model. Add learnable parameters for each attention head, and adjust the attention weights through the suppression coefficient formula. Use the same contrast loss function as in CLIP training to train the learnable parameters and fix the remaining parameters of the model. As the training progresses, the learnable parameters gradually converge to 0 or 1, realizing the soft pruning of the attention heads. Finally, the model after attention pruning is trained, and its performance improvement in tasks such as cross-modal retrieval and zero-shot classification is verified. The flexible pruning of the attention heads is achieved through the backpropagation algorithm, improving the performance of the model in downstream tasks.

[0090] The following is the device embodiment of the present invention, which can be used to execute the attention pruning method involved in the present invention. For the details not disclosed in the device embodiment of the present invention, please refer to the method embodiment of the attention pruning method involved in the present invention.

[0091] Please refer to Figure 3 , an attention pruning device 8 in the embodiment of the present invention00.

[0092] The attention pruning device 800 includes but is not limited to: a validation set preparation module 810, a genetic algorithm positioning module 830, a backpropagation positioning module 850, and an attention head pruning module 870.

[0093] Among them, the validation set preparation module 810 is used to collect multi-modal sample pairs as the validation set and determine whether to use the genetic algorithm or the backpropagation algorithm to locate harmful attention heads according to the resource conditions of the current scenario.

[0094] The genetic algorithm positioning module 830 is used to, if the genetic algorithm is adopted, encode all attention heads into binary vectors, and use the genetic algorithm and the fitness function to optimize the binary vectors on the validation set to obtain the positioning information of harmful attention heads.

[0095] The backpropagation positioning module 850 is used to, if the backpropagation algorithm is adopted, set an inhibition coefficient for each attention head, and use the contrast loss function to train on the validation set until the set conditions are met to obtain the positioning information of harmful attention heads.

[0096] The attention head pruning module 870 is used to adjust the values in the attention weight matrix according to the positioning information of harmful attention heads to obtain an adjusted attention weight matrix.

[0097] It should be noted that when the above-mentioned embodiments provide attention pruning, only the division of the above functional modules is used for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the attention pruning device will be divided into different functional modules to complete all or part of the functions described above.

[0098] In addition, the attention pruning device provided by the above embodiments and the embodiments of the attention pruning method belong to the same concept. The specific ways in which each module performs operations have been described in detail in the method embodiments, and will not be repeated here.

[0099] Figure 4 The structural schematic diagram of an electronic device shown according to an exemplary embodiment.

[0100] It should be noted that this electronic device is only an example adapted to the present invention and cannot be considered as providing any limitation to the scope of use of the present invention. This electronic device cannot be interpreted as requiring dependence on or necessarily having Figure 4 one or more components shown in the exemplary electronic device 2000.

[0101] The hardware structure of the electronic device 2000 may vary greatly due to different configurations or performances. For example, Figure 4 as shown, the electronic device 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0102] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the electronic device 2000.

[0103] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted to the present invention, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, at least one USB interface 237, etc., as Figure 4 shown, and specific limitations are not constituted herein.

[0104] The memory 250, as a carrier for resource storage, can be a read-only memory, a random access memory, a magnetic disk, an optical disc, etc. The resources stored thereon include an operating system 251, application programs 253, data 255, etc., and the storage method can be transient storage or permanent storage.

[0105] Among them, the operating system 251 is used to manage and control each hardware device and application program 253 on the electronic device 2000, so as to realize the operation and processing of the massive data 255 in the memory 250 by the central processing unit 270. It can be Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSD TM, etc.

[0106] The application program 253 is a computer-readable instruction that completes at least one specific task based on the operating system 251. It may include at least one module ( Figure 4 not shown), and each module can separately contain computer-readable instructions for the electronic device 2000. For example, the attention clipping device can be regarded as an application program 253 deployed on the electronic device 2000.

[0107] The data 255 can be signal information, etc., and is stored in the memory 250.

[0108] The central processing unit 270 may include one or more processors, and is configured to communicate with the memory 250 through at least one communication bus, so as to read the computer-readable instructions stored in the memory 250, and then realize the operation and processing of the massive data 255 in the memory 250. For example, the attention clipping method is completed in the form of reading a series of computer-readable instructions stored in the memory 250 by the central processing unit 270.

[0109] In addition, the present invention can also be realized by hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of the present invention is not limited to any specific hardware circuit, software, and the combination of the two.

[0110] Please refer to Figure 5 , in the embodiments of the present invention, an electronic device 4000 is provided. The electronic device 4000 may include: a desktop computer, a notebook computer, a server, etc. with sensor recognition capabilities.

[0111] InFigure 5 In this case, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0112] Among them, the data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 5 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0113] Optionally, the electronic device 4000 may further include a transceiver 4004, and the transceiver 4004 can be used for data interaction between the electronic device and other electronic devices, such as data sending and / or data receiving, etc. It should be noted that in practical applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0114] The processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in combination with the disclosure of the present invention. The processor 4001 can also be a combination that realizes computing functions, such as a combination including one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0115] The memory 4003 can be a ROM (Read Only Memory), or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory), or other types of dynamic storage devices that can store information and instructions. It can also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium that can be used to carry or store desired program instructions or code in the form of instructions or data structures and can be accessed by the electronic device 4000, but not limited to this.

[0116] Computer-readable instructions are stored on the memory 4003, and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002.

[0117] The one or more processors 4001 execute the computer-readable instructions to implement the attention clipping method in the above embodiments.

[0118] In addition, an embodiment of the present invention provides a storage medium on which computer-readable instructions are stored, and the computer-readable instructions are executed by one or more processors to implement the attention clipping method as described above.

[0119] An embodiment of the present invention provides a computer program product. The computer program product includes computer-readable instructions. The computer-readable instructions are stored in a storage medium, and one or more processors of the electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the attention clipping method as described above.

[0120] Compared with the related art, the beneficial effects of the present invention are: 1. The present invention can significantly improve the model performance; by accurately locating and clipping the harmful attention heads in the cross-modal large model, the feature representation ability of the model is optimized, and a significant improvement in performance in downstream tasks such as cross-modal retrieval and image-text matching is achieved, solving the problem of performance degradation caused by attention head noise or bias.

[0121] 2. The present invention can maintain the original architecture of the model; by only operating on the attention weight matrix, without modifying the model structure or parameters, it realizes optimization while maintaining the original architecture and generalization of the model, and solves the problem that traditional optimization methods may damage the original characteristics of the model.

[0122] 3. The present invention can be flexibly adapted to different scenarios; it provides two attention pruning strategies based on genetic algorithms and backpropagation algorithms, which are respectively applicable to edge inference devices with limited floating-point computing capabilities and devices with certain floating-point training computing power, realizing flexible adaptation in different application scenarios, and solving the optimization problems caused by resource limitations.

[0123] 4. The present invention can reduce the computational overhead; since the attention pruning is only performed in the inference stage and no additional training process or model parameter adjustment is required, it realizes optimization with almost no increase in the inference overhead, and solves the problem of increased computational burden that may be brought by traditional optimization methods.

[0124] 5. The present invention can improve the data utilization efficiency; only a small-scale validation set is required to complete the optimization of the attention pruning strategy, realizing efficient optimization under the condition of limited data resources, solving the problem of dependence on large-scale data training, and reducing the cost of data collection and processing.

[0125] It should be understood that although the steps in the flowchart of the accompanying drawings are shown sequentially according to the indication of the arrows, these steps do not necessarily have to be executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps does not have a strict order restriction, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times, and their execution order does not necessarily have to be sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0126] The above are only some embodiments of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. An attention clipping method, characterized in that, The method includes: Collecting multimodal sample pairs as a validation set, and determining to use a genetic algorithm or a backpropagation algorithm to locate harmful attention heads according to the resource conditions of the current scenario; If the genetic algorithm is used, all attention heads are encoded as binary vectors, and the genetic algorithm and a fitness function are used to optimize the binary vectors on the validation set to obtain the location information of harmful attention heads; If the backpropagation algorithm is used, an inhibition coefficient is set for each attention head, and a contrast loss function is used to train on the validation set until a set condition is reached to obtain the location information of harmful attention heads; Adjust the values in the attention weight matrix according to the location information of the harmful attention heads to obtain an adjusted attention weight matrix.

2. The attention clipping method according to claim 1, wherein The determining to use a genetic algorithm or a backpropagation algorithm to locate harmful attention heads according to the resource conditions of the current scenario includes: If the current scenario does not have floating-point operation capabilities, use the genetic algorithm to locate harmful attention heads. If the current scenario has floating-point operation capabilities, use the backpropagation algorithm to locate harmful attention heads.

3. The attention clipping method according to claim 1, characterized in that The encoding all attention heads as binary vectors includes: Counting the total number of attention heads in the image encoder of the cross-modal model to be optimized, and assigning a unique number to each attention head; Generating a binary vector according to the total number of attention heads; the value of an element in the binary vector being 1 indicates that the corresponding attention head is a harmful attention head to be pruned.

4. The attention clipping method according to claim 3, wherein The using the genetic algorithm and a fitness function to optimize the binary vectors on the validation set to obtain the location information of harmful attention heads includes: Randomly generating a set number of different binary vectors as an initial population; the set number being 15 - 30% of the total number of attention heads; Defining a fitness function, and using the genetic algorithm and a tournament selection strategy to iteratively perform crossover and mutation on the initial population; the fitness function is used to evaluate the performance of each binary vector on the validation set; Determining the optimal binary vector according to the fitness function, and obtaining the location information of harmful attention heads according to the optimal binary vector.

5. The attention clipping method according to claim 1, characterized in that The setting an inhibition coefficient for each attention head includes: Setting a learnable parameter for each attention head in the image encoder of the cross-modal model to be optimized as the inhibition coefficient of the attention head; the initial value of the inhibition coefficient is set to 1 indicating no pruning; Defining an update formula for the inhibition coefficient according to the requirements of the cross-modal model to be optimized; the update formula is used to control the change of the inhibition coefficient between 0 representing pruning and 1 representing no pruning.

6. The attention clipping method according to claim 5, characterized in that The setting an inhibition coefficient for each attention head, and using a contrast loss function to train on the validation set until a set condition is reached to obtain the location information of harmful attention heads includes: Using the same contrast loss function as the cross-modal model to be optimized and setting a convergence condition, and performing backpropagation training on the inhibition coefficient using the validation set and the contrast loss function; After the contrast loss function reaches the convergence condition, an optimal suppression coefficient is obtained, and the localization information of the harmful attention head is obtained according to the optimal suppression coefficient; during the training process, the parameters of the cross-modal model to be optimized are controlled to remain unchanged.

7. The attention clipping method according to claim 1, wherein The adjusting the values in the attention weight matrix according to the localization information of the harmful attention head to obtain an adjusted attention weight matrix includes: According to the localization information of the harmful attention head, a suppression operation is performed on the values corresponding to the harmful attention head in the attention weight matrix of the cross-modal model to be optimized; the suppression operation includes multiplying by a coefficient less than 1; Performing row normalization on the suppressed attention weight matrix so that the output of the pruned attention head conforms to the row normalization constraint of softmax to obtain a new attention weight matrix.

8. An attention clipping device, characterized in that, The device includes: A validation set preparation module, configured to collect multi-modal sample pairs as a validation set, and determine to use a genetic algorithm or a backpropagation algorithm to locate harmful attention heads according to the resource conditions of the current scenario; A genetic algorithm localization module, configured to, if a genetic algorithm is used, encode all attention heads as binary vectors, and use the genetic algorithm and a fitness function to optimize the binary vectors on the validation set to obtain the localization information of the harmful attention heads; A backpropagation localization module, configured to, if a backpropagation algorithm is used, set a suppression coefficient for each of the attention heads, and use a contrast loss function to train on the validation set until a set condition is reached to obtain the localization information of the harmful attention heads; An attention head pruning module, configured to adjust the values in the attention weight matrix according to the localization information of the harmful attention heads to obtain an adjusted attention weight matrix.

9. An electronic device, characterized in that, including: At least one processor and at least one memory, wherein, The memory stores computer-readable instructions; The computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the attention pruning method according to any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that, The computer-readable instructions are executed by one or more processors to implement the attention pruning method according to any one of claims 1 to 7.

Images