An attention pruning method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a kind of attention pruning method, device, electronic equipment and storage medium, involve cross-modal big model optimization technical field, wherein the method includes: collecting multi-modal sample pair as verification set, according to the resource condition of current scene to determine the positioning of harmful attention head;If using genetic algorithm, all attention heads are coded into binary vector, and the positioning information is obtained by optimizing binary vector on the verification set using genetic algorithm and fitness function;If using back propagation algorithm, set inhibition coefficient for each attention head, and train on the verification set using contrast loss function until the set condition is reached to obtain the positioning information;According to the positioning information of harmful attention head, the values in the attention weight matrix are adjusted to obtain the adjusted attention weight matrix.The present application solves the problem that the harmful attention head cannot be automatically and flexibly pruned in the prior art, and the downstream task performance is poor.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal large model optimization, and in particular to an attention tailoring method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI), large-scale cross-modal vision-language models (such as CLIP) have demonstrated powerful cross-modal representation capabilities in areas such as image-text retrieval, zero-shot classification, and visual question answering. However, because these models rely on a wide variety of training data sources and varying quality, image encoders often contain attention heads that are irrelevant or harmful to the target task. These attention heads encode bias or noise, impacting the performance of downstream tasks.

[0003] However, traditional methods lack a systematic attention pruning mechanism, especially automated pruning strategies for large-scale vision-language cross-modal models, making it difficult to quickly and correctly locate and prune harmful attention heads to improve the model's performance in downstream tasks.

[0004] In addition, different application scenarios have different requirements for computing resources and data resources. Traditional methods cannot flexibly adapt to environments with limited floating-point computing capabilities, scarce computing resources, or insufficient high-quality data, resulting in challenges in model optimization and deployment.

[0005] Therefore, there is an urgent need for an attention pruning method that can prune harmful attention heads without modifying model parameters, improve model performance, and adapt to the limitations of different resources. Summary of the Invention

[0006] The embodiments of the present invention provide an attention pruning method to address the problem that the existing technology lacks an automated and flexible attention pruning mechanism, resulting in the inability to prune harmful attention heads in large cross-modal models and poor performance on downstream tasks. The technical solution is as follows:

[0007] According to one aspect of the present invention, a method for attention tailoring includes: collecting multimodal sample pairs as a verification set, and determining whether to use a genetic algorithm or a back propagation algorithm to locate harmful attention heads based on the resource conditions of the current scene; if a genetic algorithm is used, encoding all attention heads into binary vectors, and using a genetic algorithm and a fitness function to optimize the binary vectors on the verification set to obtain positioning information of the harmful attention heads; if a back propagation algorithm is used, setting a suppression coefficient for each of the attention heads, and using a contrast loss function to train on the verification set until the set conditions are met to obtain positioning information of the harmful attention heads; and adjusting the values ​​in the attention weight matrix according to the positioning information of the harmful attention heads to obtain an adjusted attention weight matrix.

[0008] In one embodiment, determining whether to use a genetic algorithm or a back propagation algorithm to locate harmful attention heads is achieved through the following steps based on the resource conditions of the current scene: if the current scene does not have floating-point computing capabilities, a genetic algorithm is used to locate harmful attention heads; if the current scene has floating-point computing capabilities, a back propagation algorithm is used to locate harmful attention heads.

[0009] In one embodiment, encoding all attention heads into binary vectors is achieved by the following steps: counting the total number of attention heads in the image encoder of the cross-modal model to be optimized and assigning a unique number to each attention head; generating a binary vector based on the total number of attention heads; the value of an element in the binary vector being 1 indicates that the corresponding attention head is a harmful attention head that needs to be pruned.

[0010] In one embodiment, optimizing the binary vector on the validation set using a genetic algorithm and a fitness function to obtain the location information of the harmful attention heads is achieved by the following steps: randomly generating a set number of different binary vectors as an initial population; the set number is 15-30% of the total number of attention heads;

[0011] A fitness function is defined, and a genetic algorithm and a tournament selection strategy are used to iteratively perform crossover and mutation on the initial population; the fitness function is used to evaluate the performance of each binary vector on the validation set; an optimal binary vector is determined based on the fitness function, and positioning information of the harmful attention head is obtained based on the optimal binary vector.

[0012] In one embodiment, setting a suppression coefficient for each of the attention heads is achieved by the following steps: setting a learnable parameter for each attention head in the image encoder of the cross-modal model to be optimized as the suppression coefficient of the attention head; setting the initial value of the suppression coefficient to 1, which indicates no cropping; defining an update formula for the suppression coefficient according to the requirements of the cross-modal model to be optimized; and using the update formula to control the suppression coefficient to vary between 0, which indicates cropping, and 1, which indicates no cropping.

[0013] In one embodiment, a suppression coefficient is set for each of the attention heads, and a contrastive loss function is used to perform training on the validation set until the set conditions are met to obtain the positioning information of the harmful attention heads. This is achieved by the following steps: using the same contrastive loss function as the cross-modal model to be optimized and setting convergence conditions, and using the validation set and the contrastive loss function to perform back-propagation training on the suppression coefficient; when the contrastive loss function reaches the convergence conditions, the optimal suppression coefficient is obtained, and the positioning information of the harmful attention heads is obtained based on the optimal suppression coefficient; and during the training process, the parameters of the cross-modal model to be optimized are controlled to remain unchanged.

[0014] In one embodiment, the values ​​in the attention weight matrix are adjusted according to the positioning information of the harmful attention head, and the adjusted attention weight matrix is ​​obtained by the following steps: according to the positioning information of the harmful attention head, the values ​​corresponding to the harmful attention heads in the attention weight matrix of the cross-modal model to be optimized are suppressed; the suppression operation includes multiplying by a coefficient less than 1; the suppressed attention weight matrix is ​​row-normalized so that the pruned attention head output meets the row normalization constraint of softmax, and a new attention weight matrix is ​​obtained.

[0015] According to one aspect of the present invention, an attention trimming device includes: a verification set preparation module, which is used to collect multimodal sample pairs as a verification set, and determine whether to use a genetic algorithm or a back propagation algorithm to locate harmful attention heads based on the resource conditions of the current scene; a genetic algorithm positioning module, which is used to encode all attention heads into binary vectors if a genetic algorithm is used, and use a genetic algorithm and a fitness function to optimize the binary vectors on the verification set to obtain positioning information of harmful attention heads; a back propagation positioning module, which is used to set a suppression coefficient for each attention head if a back propagation algorithm is used, and use a contrast loss function to train on the verification set until the set conditions are met to obtain positioning information of harmful attention heads; an attention head trimming module, which is used to adjust the values ​​in the attention weight matrix according to the positioning information of the harmful attention heads to obtain an adjusted attention weight matrix.

[0016] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the attention clipping method as described above.

[0017] According to one aspect of the present invention, a storage medium stores computer-readable instructions thereon, wherein the computer-readable instructions are executed by one or more processors to implement the attention tailoring method as described above.

[0018] The beneficial effects brought about by the technical solution provided by the present invention are:

[0019] In the above technical solution, the present invention first collects multimodal sample pairs as a verification set, selects a genetic algorithm or a back propagation algorithm to locate harmful attention heads according to the current scene resource conditions, and if the scene does not have floating-point computing capabilities, encodes all attention heads into binary vectors, randomly generates a set number of different binary vectors as the initial population, and defines a fitness function. Iterative cross-mutation is performed using a genetic algorithm and a tournament selection strategy, and the optimal binary vector is determined according to the fitness function to obtain the harmful attention head positioning information; if the scene has floating-point computing capabilities, a learnable parameter is set for each attention head as a suppression coefficient, and the same contrast loss function as the cross-modal model to be optimized is used and the convergence condition is set. The suppression coefficient is back-propagated on the verification set. When the contrast loss function reaches the convergence condition, the optimal suppression coefficient is obtained, thereby determining the harmful attention head positioning information. Finally, according to the harmful attention head positioning information, the corresponding values ​​of the harmful attention heads in the attention weight matrix are suppressed to obtain the adjusted attention weight matrix. By rationally trimming harmful attention heads, the performance and efficiency of the cross-modal model are effectively improved, providing new ideas for model optimization. This effectively solves the problem of the lack of automated and flexible attention trimming mechanisms in existing technologies, which results in the inability of large cross-modal models to trim harmful attention heads and poor performance of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0021] Figure 1 is a flow chart of an attention tailoring method according to an exemplary embodiment;

[0022] Figure 2 This is a structural diagram of the attention clipping method in an application scenario;

[0023] Figure 3 is a block diagram of an attention tailoring device according to an exemplary embodiment;

[0024] Figure 4 is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0025] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0026] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0027] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0028] The present invention provides an attention trimming method that analyzes the impact of each attention head in a multi-head attention module on the output features, locates and suppresses attention heads that are detrimental to performance, thereby improving the quality of the output features. This method effectively addresses the problem in the prior art of lacking an automated and flexible attention trimming mechanism, which results in the inability of large cross-modal models to trim harmful attention heads and poor downstream task performance. The attention trimming method is suitable for use in an attention trimming device, which can be an electronic device. The attention trimming method in the embodiments of the present invention can be applied to a variety of scenarios, such as the optimization of large cross-modal vision-language models.

[0029] See also Figure 1 , an embodiment of the present invention provides an attention clipping method, which is applicable to electronic devices.

[0030] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0031] like Figure 1 As shown, the method may include the following steps:

[0032] In step 110 , multimodal sample pairs are collected as a validation set, and the genetic algorithm or back propagation algorithm is used to locate harmful attention heads based on the resource conditions of the current scenario.

[0033] In one possible implementation, if the current scene does not have floating-point computing capabilities, a genetic algorithm is used to locate harmful attention heads. If the current scene has floating-point computing capabilities, a back-propagation algorithm is used to locate harmful attention heads.

[0034] Specifically, the genetic algorithm-based method for locating harmful attention heads only relies on model inference and does not require training. Therefore, it is suitable for deployment on some edge devices with low-precision integer computing power, such as INT8 / INT16 chips, thereby achieving efficient attention cropping when computing resources are limited.

[0035] Specifically, the backpropagation-based approach to localizing harmful attention heads requires minimal parameter training and is suitable for lightweight edge devices with sufficient floating-point computing power, such as mobile devices and embedded GPUs. This approach introduces learnable parameters for each attention head and trains it using a contrastive loss function to derive an independent clipping strength coefficient for each attention head, thereby achieving more precise attention clipping while maintaining sufficient floating-point computing power.

[0036] In the above process, the embodiments of the present invention lay a solid foundation for the subsequent attention cropping process through reasonable data preparation and algorithm selection. Selecting the appropriate algorithm based on the resource conditions of different scenarios can ensure the efficiency and applicability of the cropping process while meeting the needs of different application scenarios.

[0037] In step 120, if a genetic algorithm is used, all attention heads are encoded into binary vectors, and the binary vectors are optimized on the validation set using the genetic algorithm and the fitness function to obtain the positioning information of the harmful attention heads.

[0038] In one possible implementation, the total number of attention heads in the image encoder of the cross-modal model to be optimized is counted, a unique number is assigned to each attention head, and a binary vector is generated according to the total number of attention heads.

[0039] Among them, the value of the element in the binary vector is 1, which means that the corresponding attention head is a harmful attention head that needs to be pruned, and the value of the element is 0, which means that the corresponding attention head does not need to be pruned.

[0040] In one possible implementation, a set number of different binary vectors are randomly generated as the initial population, a fitness function is defined, a genetic algorithm and a tournament selection strategy are used to iteratively perform crossover and mutation on the initial population, the optimal binary vector is determined according to the fitness function, and the positioning information of the harmful attention head is obtained based on the optimal binary vector.

[0041] The number of attention heads is set to 15-30% of the total number of attention heads, and the fitness function is used to evaluate the performance of each binary vector on the validation set, such as the accuracy or recall rate on the downstream image-text matching task.

[0042] Among them, the crossover operation in the genetic algorithm is to select two binary vectors (parents), exchange them at a randomly selected crossover point, and generate two new binary vectors (offspring). The crossover operation helps to increase the diversity of the population and explore more potential solutions.

[0043] Among them, the mutation operation is to randomly flip certain bits of the binary vector (0 to 1, 1 to 0) to generate a new binary vector. The mutation operation helps prevent the algorithm from falling into the local optimal solution and increases the possibility of finding the global optimal solution.

[0044] Selection and iteration determine the optimal binary vector based on the fitness function, eliminate individuals with low fitness, and retain individuals with high fitness (a tournament selection strategy with a size of 3 is used to retain elite individuals), thereby generating a new generation of the population. The number of evolutionary generations is set to 100, and crossover, mutation, and selection operations are repeated until the exit condition is met, thereby obtaining the location information of the harmful attention heads.

[0045] In the above process, the embodiment of the present invention can find the optimal solution in a larger search space by simulating the natural evolution process through the genetic algorithm. The fitness function can accurately evaluate the performance of each binary vector on the verification set, thereby effectively locating harmful attention heads.

[0046] In step 130, if the back-propagation algorithm is used, a suppression coefficient is set for each attention head, and a contrast loss function is used to train on the validation set until the set conditions are met to obtain the positioning information of the harmful attention heads.

[0047] In one possible implementation, a learnable parameter is set for each attention head in the image encoder of the cross-modal model to be optimized as the suppression coefficient of the attention head, and an update formula for the suppression coefficient is defined according to the requirements of the cross-modal model to be optimized.

[0048] The initial value of the suppression coefficient is set to 1, which indicates no clipping, and the update formula is used to control the suppression coefficient to change between 0, which indicates clipping, and 1, which indicates no clipping.

[0049] In one possible implementation, the same contrastive loss function as that of the cross-modal model to be optimized is used and the convergence condition is set. The suppression coefficient is back-propagated and trained using the validation set and the contrastive loss function. When the contrastive loss function reaches the convergence condition, the optimal suppression coefficient is obtained, and the positioning information of the harmful attention head is obtained based on the optimal suppression coefficient.

[0050] During the training process, the parameters of the cross-modal model to be optimized are controlled to remain unchanged.

[0051] In the above process, the embodiment of the present invention can use the gradient information to quickly adjust the suppression coefficient through the back-propagation algorithm, and can better measure the performance of the model by comparing the loss function, so as to accurately find the harmful attention heads.

[0052] Step 140: Adjust the values ​​in the attention weight matrix according to the positioning information of the harmful attention head to obtain an adjusted attention weight matrix.

[0053] In one possible implementation, based on the positioning information of the harmful attention heads, the values ​​corresponding to the harmful attention heads in the attention weight matrix of the optimized cross-modal model are suppressed, and the suppressed attention weight matrix is ​​row-normalized so that the pruned attention head output meets the row normalization constraint of softmax, thereby obtaining a new attention weight matrix.

[0054] The suppression operation includes multiplying by a coefficient less than 1, etc., which is not limited here.

[0055] In the above process, the embodiment of the present invention obtains the adjusted attention weight matrix by suppressing the numerical values ​​corresponding to the harmful attention heads and performing row normalization processing, which can effectively reduce the impact of harmful attention heads on the output features and improve the performance of the model. At the same time, the model architecture remains unchanged and only the attention weight matrix is ​​replaced to facilitate the deployment and integration of the model.

[0056] Through the above process, the present invention achieves systematic pruning of attention heads in large cross-modal models through four steps: data preparation and algorithm selection, localization of harmful attention heads, adjustment of the attention weight matrix, and model verification and effect evaluation. This method does not require modification of model parameters, adds almost no inference overhead, and significantly improves the model's performance in downstream tasks. Through reasonable data preparation and algorithm selection, accurate localization of harmful attention heads, effective adjustment of the attention weight matrix, and practical verification and effect evaluation, it provides new ideas and methods for optimizing large cross-modal models, suitable for the needs of different application scenarios.

[0057] In one application scenario, attention tailoring for an image-text cross-modal model can include the following steps:

[0058] Step 1: Data preparation and algorithm selection.

[0059] Specifically, 1,000 pieces of data are randomly sampled from an image-text dataset (such as MS COCO) as a validation set D for subsequent algorithm optimization.

[0060] Furthermore, the algorithm is selected based on the resource conditions of the current scenario. If the current scenario does not have floating-point computing capabilities (such as edge devices), the attention pruning strategy based on the genetic algorithm (AAT-GA) is selected; if floating-point computing capabilities are available (such as mobile terminals and embedded GPUs), the attention pruning strategy based on backpropagation (AAT-BP) is selected.

[0061] In the above process, the embodiment of the present invention uses 1000 data samples as a validation set, ensuring the effectiveness of algorithm optimization while reducing computational effort, making it suitable for scenarios with limited data. Flexible algorithm selection based on resource constraints allows the technology to be widely applied to devices with varying computing power, improving its applicability.

[0062] Step 2: Localization of harmful attention heads.

[0063] Specifically, if the attention cropping strategy based on genetic algorithm (AAT-GA) is selected: first, all attention heads in the image encoder of the cross-modal model to be optimized are encoded into a binary vector of length N, where each number represents the state of the corresponding attention head (1 represents cropping, 0 represents retention), and then a genetic algorithm is used to search for the optimal 0-1 combination of the binary vector.

[0064] The search process of the genetic algorithm includes the steps of randomly generating the initial population, calculating fitness, selection, crossover, mutation, etc., and finally obtaining the optimal binary vector.

[0065] In the above process, the embodiment of the present invention can find the optimal solution in a larger search space by simulating the natural evolution process through the genetic algorithm. The fitness function can accurately evaluate the performance of each binary vector on the verification set, thereby effectively locating harmful attention heads.

[0066] Specifically, if the attention cropping strategy based on backpropagation (AAT-BP) is selected: first, a learnable parameter is set as the suppression coefficient for each attention head in the image encoder of the cross-modal model to be optimized, and the initial value is set to 1. Then, the same contrast loss function as the cross-modal model to be optimized is used to perform backpropagation training on the validation set D. During the training process, the model parameters are kept unchanged. When the contrast loss function reaches the convergence condition, the optimal suppression coefficient is obtained, and then the positioning information of the harmful attention heads is obtained.

[0067] In the above process, the embodiment of the present invention can use the gradient information to quickly adjust the suppression coefficient through the back-propagation algorithm, and can better measure the performance of the model by comparing the loss function, so as to accurately find the harmful attention heads.

[0068] Step 3: Attention weight matrix adjustment.

[0069] Specifically, according to the positioning information of the harmful attention heads, the values ​​corresponding to the harmful attention heads in the attention weight matrix of the optimized cross-modal model are suppressed, such as multiplying them by a coefficient less than 1 (such as 0.1), and then the suppressed attention weight matrix is ​​row-normalized so that the pruned attention head output meets the row normalization constraint of softmax, and a new attention weight matrix is ​​obtained.

[0070] In the above process, the embodiment of the present invention can adjust the attention weight matrix by suppressing and row normalizing the values ​​corresponding to harmful attention heads, so that the model pays more attention to important attention heads, reduces the impact of noise and bias, and improves the performance and stability of the model.

[0071] Step 4: Model verification and effect evaluation.

[0072] Specifically, the pruned model was used to verify its performance on downstream tasks (such as cross-modal retrieval and zero-shot classification). The performance metrics (such as recall and precision) of the pruned and unpruned models on various tasks were compared to evaluate the effectiveness of the attention pruning method. Through practical verification and effect evaluation, we can intuitively understand the effect of attention pruning on model performance, providing strong support for subsequent model optimization and application.

[0073] Through the above process, the embodiment of the present invention describes in detail the implementation process of the attention clipping technology. Through the four steps of data preparation and algorithm selection, harmful attention head positioning, attention weight matrix adjustment, and model verification and effect evaluation, systematic clipping of attention heads in cross-modal large models is achieved. The present invention does not require modification of model parameters, hardly increases inference overhead, and can significantly improve the performance of the model in downstream tasks. In a variety of cross-modal retrieval and classification tasks, the recall rate can be increased by up to 11.1%, and the accuracy rate in zero-sample classification tasks can be increased by 1.0%-1.5%. The present invention has certain effectiveness and applicability, and provides new ideas and methods for the optimization of cross-modal large models.

[0074] In one application scenario, Figure 2 A schematic diagram of the structure of the model optimized using the attention trimming strategy (AAT) in this invention is shown.

[0075] like Figure 2 As shown in the figure, the Attention Ablation Technique is used to optimize the attention head of a large model based on the CLIP framework for processing text and image information. Figure 2 The figure shows two types of input data: text information and image information corresponding to the text.

[0076] Figure 2The large model of the CLIP framework contains two encoders: the text encoder is used to encode the input text information and convert the text into a feature vector representation; the image encoder is used to encode the input image information and convert the image into a feature vector representation.

[0077] Among them, there is an arrow between the two encoders, which means that the encoded feature vector can be used to perform text-to-image retrieval tasks, that is, to find the corresponding image based on the text description.

[0078] Figure 2 The internal structure of the attention cropping technology is shown in detail below the image encoder. The input data is first processed in blocks to divide the image or feature map into multiple small blocks, and then the attention weights are calculated through the multi-head attention module. The multi-head attention module contains a multi-layer structure, and each layer contains multiple attention heads (attention head 1, attention head 2, attention head 3, attention head 4). Each attention head is used to calculate the attention weights between different parts of the input data; after each attention head, there is a feedforward network (FFN) for further nonlinear transformation of the features after attention calculation.

[0079] An attention clipping indicator, located below each layer, indicates whether the attention head should be retained or ablated (i.e., its effect removed or suppressed). This allows for selective retention or removal of attention heads to optimize the model. Each multi-head attention (MHA) module contains multiple attention heads, combining their computational results to obtain a richer feature representation.

[0080] Furthermore, after performing attention head pruning on the CLIP framework's large multimodal model, an optimized CLIP model was obtained. This optimized CLIP model was used to perform text-to-image and image-to-text retrieval tasks on mainstream English datasets such as MS COCO and Flickr30k, improving the average recall rate (mean-R) by 1%-5%. On mainstream Chinese datasets such as COCO-CN and Flickr30k, the improved Chinese-CLIP model using this inventive method significantly outperformed the original model in both text-to-image and image-to-text tasks, with an average recall rate (mean-R) improvement of 1%-11%.

[0081] Furthermore, after attention head pruning, the CLIP model was optimized to obtain the model. The zero-shot classification experiment on the ImageNet-1k dataset using the optimized CLIP model showed an accuracy improvement of 1.0%-1.5%, while also maintaining the improvement in retrieval capabilities.

[0082] Among them, when performing attention head pruning on the CLIP model on the RTX 4090 GPU, it takes 2 to 8 minutes to optimize the model using the backpropagation-based attention pruning strategy AAT-BP, and 18 to 50 minutes to optimize the model using the genetic algorithm-based attention pruning strategy AAT-GA.

[0083] Through the above process, the embodiment of the present invention directly manipulates the attention weight matrix to systematically locate and trim attention heads that have a negative impact on output features, thereby improving the performance of cross-modal downstream tasks. It can effectively screen out attention heads that are beneficial to the task, improve the performance and efficiency of the model, and reduce unnecessary computing resource consumption. It is suitable for scenarios with limited computing resources and limited data resources, and shows good applicability, improvement effect and deployment feasibility under various model structures, languages, tasks and equipment conditions.

[0084] In one application scenario, attention cropping based on genetic algorithm (AAT-GA) is used for model optimization.

[0085] Specifically, 1000 data points, or 1000 image-text pairs, were randomly sampled from the MS COCO image-text dataset as the validation set D for subsequent optimization of the genetic algorithm. Taking the CLIP-Base model as an example, its image encoder has 12 Transformer layers, each with 12 attention heads, for a total of 144 attention heads. These 144 attention heads are encoded into a 144-length binary vector, where each digit represents the state of the corresponding attention head (1 for cropping, 0 for retaining).

[0086] Furthermore, 216 (15% of 144) different binary vectors are randomly generated as the initial population. The binary vectors are generally set to 15-30% of the total number of attention heads. For each binary vector, the attention heads in the image encoder of the CLIP-Base model are pruned according to its pruned / retained status of the attention heads.

[0087] Specifically, for the pruned attention head, the value in its attention weight matrix is ​​multiplied by 0.1 (suppression coefficient). After the pruned attention head, each model is inferred on the validation set D to calculate the fitness corresponding to each binary vector.

[0088] The fitness function calculates the fitness value based on the cosine similarity between the i-th text in the verification set D and its corresponding real image, and the cosine similarity between the i-th text and the j-th difficult negative sample image in the corresponding difficult negative sample set.

[0089] Furthermore, the individual with the largest fitness is selected as the optimal individual, and it is determined whether the number of evolutionary generations has reached the upper limit (100 generations). If so, the algorithm exits and outputs the optimal individual; if not, the algorithm continues; a tournament selection strategy (select 1 out of every 3) is used to retain individuals with high fitness and eliminate individuals with low fitness.

[0090] Among them, the crossover probability is set to 0.9 and the mutation probability is set to 0.5. Crossover and mutation operations are performed to expand the number of individuals in the population to the initial population size of 216; during the evolution process, the difficult negative sample set in the fitness function is updated until the set exit condition is met.

[0091] Finally, a genetic algorithm is used to determine the optimal combination of attention heads to prune and retain, effectively identifying which attention heads should be pruned. The optimization is then completed by pruning the attention heads identified by the genetic algorithm. This model is then used to perform downstream tasks (such as cross-modal retrieval and zero-shot classification) to validate its performance.

[0092] Through the above process, this example uses a genetic algorithm to optimize and prune the attention heads of the CLIP-Base model. Through the genetic algorithm's crossover, mutation, and fitness evaluation, the optimal pruning combination of attention heads is gradually selected. The fitness function is designed based on the cosine similarity between text and image and the similarity of difficult negative samples, ensuring that the optimized model performs better in downstream tasks. Ultimately, the genetic algorithm obtains the optimal pruning solution, and the optimized model's performance is verified in tasks such as cross-modal retrieval and zero-shot classification.

[0093] In another application scenario, attention pruning based on backpropagation (AAT-BP) is used for model optimization.

[0094] Specifically, we randomly sampled 1,000 data points from the MS COCO image-text dataset, representing 1,000 image-text pairs, as the validation set D, which we used to optimize the backpropagation algorithm. For each attention head in the CLIP-Base model's image encoder, a learnable parameter was added to the model.

[0095] Specifically, the suppression coefficient is obtained by the following formula:

[0096] ;

[0097] The temperature coefficient is set to 5.0 by default. Each attention head is pruned to obtain the pruned model under the current suppression coefficient. The initial values ​​of all learnable parameters are set to 1.0 to ensure that all attention heads are retained at the beginning.

[0098] Furthermore, the same contrastive loss function as used in CLIP training is used, minimizing the similarity score of incorrect matches while maximizing the similarity score of correct matches. The learnable parameters are trained on the validation set D using the contrastive loss function. Except for the learnable parameters, the remaining parameters of the model are fixed. During training, the values ​​of the learnable parameters gradually converge to 0 or 1, indicating that the corresponding attention head is completely pruned or retained.

[0099] The model is then trained until the loss function converges. This model is then optimized after attention pruning. This model is then used to perform downstream tasks (such as cross-modal retrieval and zero-shot classification) to verify its performance.

[0100] Through the above process, this embodiment uses the back-propagation algorithm to optimize and trim the attention heads of the CLIP-Base model. New learnable parameters are added to each attention head, and the attention weight is adjusted using the suppression coefficient formula. The learnable parameters are trained using the same contrast loss function as CLIP training, and the remaining model parameters are fixed. As the training progresses, the learnable parameters gradually converge to 0 or 1, achieving soft trimming of the attention heads. Finally, the model after attention trimming is trained, and its performance improvement in tasks such as cross-modal retrieval and zero-shot classification is verified. Flexible trimming of attention heads is achieved through the back-propagation algorithm, which improves the performance of the model in downstream tasks.

[0101] The following is an embodiment of the device of the present invention, which can be used to perform the attention tailoring method involved in the present invention. For details not disclosed in the embodiment of the device of the present invention, please refer to the method embodiment of the attention tailoring method involved in the present invention.

[0102] See also Figure 3 , an embodiment of the present invention provides an attention cropping device 800.

[0103] The attention trimming device 800 includes but is not limited to: a verification set preparation module 810, a genetic algorithm positioning module 830, a back-propagation positioning module 850 and an attention head trimming module 870.

[0104] Among them, the verification set preparation module 810 is used to collect multimodal sample pairs as a verification set, and determine whether to use a genetic algorithm or a back propagation algorithm to locate harmful attention heads based on the resource conditions of the current scene.

[0105] The genetic algorithm positioning module 830 is used to encode all attention heads into binary vectors if a genetic algorithm is used, and use the genetic algorithm and fitness function to optimize the binary vectors on the validation set to obtain the positioning information of the harmful attention heads.

[0106] The back-propagation positioning module 850 is used to set a suppression coefficient for each attention head if the back-propagation algorithm is adopted, and to train on the validation set using a contrast loss function until the set conditions are met to obtain the positioning information of the harmful attention head.

[0107] The attention head trimming module 870 is used to adjust the values ​​in the attention weight matrix according to the positioning information of the harmful attention heads to obtain an adjusted attention weight matrix.

[0108] It should be noted that the attention clipping provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the attention clipping device will be divided into different functional modules to complete all or part of the functions described above.

[0109] In addition, the attention cropping device and the attention cropping method provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.

[0110] Figure 4 The following is a schematic diagram of the structure of an electronic device according to an exemplary embodiment.

[0111] It should be noted that the electronic device is only an example adapted to the present invention and cannot be considered to provide any limitation on the scope of use of the present invention. The electronic device cannot be interpreted as needing to rely on or must have Figure 4 One or more components of exemplary electronic device 2000 are shown.

[0112] The hardware structure of the electronic device 2000 may vary greatly due to different configurations or performances, such as Figure 4 As shown, the electronic device 2000 includes a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0113] Specifically, the power supply 210 is used to provide operating voltage for various hardware devices on the electronic device 2000 .

[0114] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted by the present invention, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 4 As shown, this does not constitute a specific limitation.

[0115] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include an operating system 251, application 253 and data 255, etc. The storage method can be temporary storage or permanent storage.

[0116] Among them, the operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000, so as to enable the central processing unit 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0117] The application 253 is a computer-readable instruction that performs at least one specific task based on the operating system 251, and may include at least one module ( Figure 4 (not shown), each module may respectively include computer-readable instructions for the electronic device 2000. For example, the attention tailoring device may be considered as an application 253 deployed on the electronic device 2000.

[0118] The data 255 may be signal information, etc., and is stored in the memory 250 .

[0119] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby performing operations and processing on the massive data 255 in the memory 250. For example, the attention tailoring method is implemented by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0120] In addition, the present invention can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present invention is not limited to any specific hardware circuits, software, or combination thereof.

[0121] See also Figure 5 In an embodiment of the present invention, an electronic device 4000 is provided. The electronic device 4000 may include: a desktop computer, a laptop computer, a server, etc. with sensor recognition capabilities.

[0122] exist Figure 5 In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0123] The data exchange between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0124] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0125] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0126] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 4000, but is not limited to these.

[0127] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0128] The computer-readable instructions are executed by one or more processors 4001 to implement the attention clipping method in the above-mentioned embodiments.

[0129] In addition, an embodiment of the present invention provides a storage medium on which computer-readable instructions are stored. The computer-readable instructions are executed by one or more processors to implement the attention clipping method described above.

[0130] A computer program product is provided in an embodiment of the present invention. The computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the attention clipping method as described above.

[0131] Compared with the related art, the present invention has the following beneficial effects:

[0132] 1. This invention can significantly improve model performance. By precisely locating and pruning harmful attention heads in large cross-modal models, it optimizes the model's feature representation capabilities, achieves significant performance improvements in downstream tasks such as cross-modal retrieval and image-text matching, and solves the problem of performance degradation caused by attention head noise or bias.

[0133] 2. The present invention can maintain the original architecture of the model; by operating only on the attention weight matrix without modifying the model structure or parameters, it can achieve optimization while maintaining the original architecture and generalization of the model, solving the problem that traditional optimization methods may destroy the original characteristics of the model.

[0134] 3. The present invention can flexibly adapt to different scenarios; it provides two attention clipping strategies based on genetic algorithms and backpropagation algorithms, which are respectively suitable for edge inference devices with limited floating-point computing capabilities and devices with certain floating-point training computing power, achieving flexible adaptation in different application scenarios and solving the optimization problems caused by resource limitations.

[0135] 4. The present invention can reduce computational overhead. Since attention clipping is only performed during the inference phase and does not require additional training or model parameter adjustment, it achieves optimization with almost no increase in inference overhead, solving the problem of increased computational burden that may be caused by traditional optimization methods.

[0136] 5. The present invention can improve data utilization efficiency; only a small-scale validation set is required to optimize the attention clipping strategy, achieving efficient optimization under limited data resources, solving the problem of large-scale data training dependence, and reducing the cost of data collection and processing.

[0137] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0138] The above descriptions are only partial embodiments of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. An attention tailoring method, characterized in that: The method comprises: Collect multimodal sample pairs as a validation set. If the current scene does not have floating-point computing capabilities, use a genetic algorithm to locate harmful attention heads. If the current scene has floating-point computing capabilities, use a back-propagation algorithm to locate harmful attention heads. The multimodal sample pairs include text information and image information corresponding to the text. If a genetic algorithm is used, all attention heads are encoded into binary vectors, and a set number of different binary vectors are randomly generated as the initial population; the set number is 15-30% of the total number of attention heads; A fitness function is defined, and crossover and mutation are iteratively performed on the initial population using a genetic algorithm and a tournament selection strategy; the fitness function is used to evaluate the performance of each binary vector on the validation set; determining an optimal binary vector according to the fitness function, and obtaining positioning information of the harmful attention head according to the optimal binary vector; If the back propagation algorithm is used, a suppression coefficient is set for each of the attention heads, and a contrastive loss function is used to train on the validation set until the set conditions are met to obtain the positioning information of the harmful attention heads; According to the positioning information of the harmful attention head, a suppression operation is performed on the value corresponding to the harmful attention head in the attention weight matrix of the cross-modal model to be optimized; the suppression operation includes multiplying by a coefficient less than 1; The suppressed attention weight matrix is ​​row-normalized so that the pruned attention head output meets the row normalization constraint of softmax, and a new attention weight matrix is ​​obtained.

2. The attention tailoring method according to claim 1, wherein: The above code encodes all attention heads into binary vectors, including: Count the total number of attention heads in the image encoder of the cross-modal model to be optimized, and assign a unique number to each of the attention heads; A binary vector is generated according to the total number of the attention heads; a value of 1 in the binary vector indicates that the corresponding attention head is a harmful attention head that needs to be pruned.

3. The attention tailoring method according to claim 1, wherein: Setting a suppression coefficient for each attention head includes: Setting a learnable parameter for each attention head in the image encoder of the cross-modal model to be optimized as a suppression coefficient of the attention head; the initial value of the suppression coefficient is set to 1, indicating no clipping; An update formula for the suppression coefficient is defined according to the requirements of the cross-modal model to be optimized; the update formula is used to control the suppression coefficient to vary between 0 indicating clipping and 1 indicating no clipping.

4. The attention tailoring method according to claim 3, wherein: Setting a suppression coefficient for each of the attention heads, and using a contrastive loss function to train on the validation set until a set condition is met to obtain positioning information of the harmful attention heads, includes: Using the same contrastive loss function as the cross-modal model to be optimized and setting a convergence condition, back-propagation training is performed on the inhibition coefficient using the validation set and the contrastive loss function; When the contrast loss function reaches a convergence condition, an optimal suppression coefficient is obtained, and positioning information of the harmful attention head is obtained according to the optimal suppression coefficient; during the training process, the parameters of the cross-modal model to be optimized are controlled to remain unchanged.

5. An attention tailoring device, characterized in that: The device is applied to the attention tailoring method according to claim 1, and the device includes: The validation set preparation module is used to collect multimodal sample pairs as the validation set and determine whether to use the genetic algorithm or the back propagation algorithm to locate harmful attention heads based on the resource conditions of the current scenario; A genetic algorithm positioning module is configured to, if a genetic algorithm is used, encode all attention heads into binary vectors, and optimize the binary vectors on the validation set using the genetic algorithm and a fitness function to obtain positioning information of harmful attention heads; A back-propagation positioning module is configured to set a suppression coefficient for each of the attention heads if a back-propagation algorithm is used, and to perform training on the validation set using a contrastive loss function until a set condition is met to obtain positioning information of the harmful attention heads; The attention head trimming module is used to adjust the values ​​in the attention weight matrix according to the positioning information of the harmful attention head to obtain an adjusted attention weight matrix.

6. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the attention clipping method according to any one of claims 1 to 4.

7. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement the attention tailoring method according to any one of claims 1 to 4.

Images