Pruning method and device of target detection model and electronic equipment

Abstract

The invention discloses a pruning method and device for a target detection model, and electronic equipment, and relates to the technical field of model pruning, and the method comprises the steps: initializing a mask of each module structure, obtaining an initialized mask, determining the importance score of each module structure, pruning the module structure with the minimum importance score, and obtaining a pruning result of the module structure with the minimum importance score; according to the attention head operand of each attention head in the residual attention head set and the filter operand of each filter in the residual filter set, determining a target pruning combination under the constraint of operand conditions, and pruning the target pruning combination to obtain a target detection model after initial pruning; and under the constraint of a delay condition, determining the attention head to be pruned and the filter in each layer of network in the target detection model after initial pruning, and performing pruning to obtain a pruned target detection model. The technical problem that the precision of the pruned model is poor due to the fact that the accuracy of determining the structure needing to be pruned in the model is low in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of model pruning technology, and more specifically, to a method, apparatus, and electronic device for pruning a target detection model. Background Technology

[0002] With the continuous development of computer vision and remote sensing technologies, the spatial and temporal resolutions of remote sensing images have been significantly improved, providing rich information support for the intelligent analysis of remote sensing images. In high-resolution remote sensing images, targets such as ships, vehicles, and buildings often have complex appearances and multi-scale features, which places higher demands on the model's representation and reasoning capabilities.

[0003] Currently, deep learning-based remote sensing image target detection methods are rapidly developing, especially VisionTransformer (a type of image recognition model, or ViT for short), which has demonstrated superior performance in the field of natural images. However, the large number of parameters and computational demands of the ViT model limit its deployment on resource-constrained platforms (such as UAVs, spaceborne devices, and edge computing terminals). Therefore, model compression and acceleration for remote sensing image target detection tasks have become an important research direction.

[0004] Lightweight technology for target detection models in remote sensing images has broad application prospects. For example, deploying lightweight and efficient detection models in scenarios such as traffic monitoring, disaster early warning, and resource exploration can achieve rapid response and real-time inference; tasks such as ship identification and target tracking also place higher demands on the inference speed and deployment flexibility of the models.

[0005] In related technologies, various model compression methods have been proposed for lightweighting object detection models, such as network pruning, knowledge distillation, quantization, and low-rank decomposition. However, current methods for lightweighting object detection models suffer from the following problems:

[0006] (1) Many methods use rough approximations when evaluating the importance of pruning candidates, such as relying only on attention scores, weight magnitudes or static indicators, ignoring the dependencies and differences between different layers, making it difficult to accurately judge the impact of pruning on overall performance, and easily destroying the critical information path, especially in the structurally complex Transformer.

[0007] (2) The resource constraint handling strategies are generally crude, mostly relying on fixed retention ratios or single greedy strategies. They lack fine control over resource consumption such as FLOPs (Floating point Operations per Second), number of parameters, or latency, resulting in unstable inference efficiency of the compressed model, which is difficult to meet actual deployment needs.

[0008] At present, no effective solution has been proposed for the above problems. SUMMARY

[0009] The pruning method and the device of the target detection model and the electronic equipment provided in the embodiments of the present application at least solve the technical problem that the accuracy of determining the structure to be pruned in the model is low in the related art, resulting in poor accuracy of the pruned model.

[0010] According to an aspect of the embodiments of the present application, a pruning method of a target detection model is provided, the target detection model comprising: a plurality of network layers, each network layer comprising: a plurality of module structures, the module structure being one of an attention head or a filter, comprising: initializing a mask of each module structure to obtain an initialized mask of each module structure; determining an importance score of each module structure based on a weight of each module structure and the initialized mask, and pruning an attention head with the smallest importance score and a filter with the smallest importance score to obtain a remaining attention head set and a remaining filter set; determining a target pruning combination under an operation amount condition constraint based on an attention head operation amount of each attention head in the remaining attention head set and a filter operation amount of each filter in the remaining filter set, wherein the target pruning combination is a pruning combination with the smallest total importance score, each pruning combination comprising: an attention head position of a preset number of attention heads and a filter position of a preset number of filters; pruning the attention head indicated by each attention head position and the filter indicated by each filter position in the target pruning combination to obtain an initial pruned target detection model, and determining an attention head and a filter to be pruned in each network layer in the initial pruned target detection model under a delay condition constraint; pruning the attention head and the filter to be pruned in each network layer in the initial pruned target detection model to obtain a target detection model with pruning completed.

[0011] Further, the step of determining the importance score of each module structure based on the weight of each module structure and the initialized mask comprises: determining an information loss of the module structure based on the weight of the module structure and the initialized mask; obtaining a partial derivative value by performing partial derivation on the information loss, and determining an information matrix of the module structure based on the partial derivative value and a structure position of the module structure, wherein the structure position is determined based on a number of network layers in the target detection model in which the module structure is located and a position number of the network layer in which the module structure is located; and performing diagonal approximation processing on the information matrix to obtain the importance score of the module structure.

[0012] Further, based on the attention head operation amount of each attention head in the remaining attention head set and the filter operation amount of each filter in the remaining filter set, under the operation amount condition constraint, the step of determining the target pruning combination comprises: determining a target operation amount, wherein the target operation amount is used to determine the operation amount condition constraint; based on the target operation amount, the attention head operation amount of each attention head in the remaining attention head set and the filter operation amount of each filter in the remaining filter set, determining a plurality of quantity pairs, wherein each quantity pair contains a preset attention head quantity of the attention head to be pruned and a preset filter quantity of the filter; for each quantity pair, determining all pruning combinations that satisfy the quantity pair, wherein all attention head positions and all preset filter positions in each pruning combination are not completely the same; determining the total importance score of each pruning combination; determining the pruning combination with the smallest total importance score as the target pruning combination.

[0013] Further, based on the target operation amount, the attention head operation amount of each attention head in the remaining attention head set and the filter operation amount of each filter in the remaining filter set, the step of determining a plurality of quantity pairs comprises: determining the attention head quantity of the attention head in the remaining attention head set and determining the filter quantity of the filter in the remaining filter set; traversing the attention head quantity and taking each traversed quantity as a reserved attention head quantity, wherein the traversal starts from 0 and ends at the attention head quantity, and each traversal increases by one quantity; for each reserved attention head quantity, based on the target operation amount, the attention head average operation amount and the filter average operation amount, determining the reserved filter quantity corresponding to the reserved attention head quantity, wherein the attention head average operation amount is calculated based on the attention head operation amount of each attention head in the remaining attention head set; the filter average operation amount is calculated based on the filter operation amount of each filter in the remaining filter set; based on the attention head quantity and each reserved attention head quantity, determining each preset attention head quantity, and based on the filter quantity and the reserved filter quantity corresponding to each reserved attention head quantity, determining the preset filter quantity corresponding to each preset attention head quantity, to obtain a plurality of quantity pairs, wherein the quantity pair contains a preset attention head quantity and a preset filter quantity corresponding to the preset attention head quantity, and each quantity pair contains different preset attention head quantities.

[0014] Further, the step of determining the attention heads and filters to be pruned in each network layer of the initial pruned target detection model under the delay condition constraint comprises: determining the number of attention heads and the number of filters to be retained in each network layer of the target detection model based on a preset fixed overhead, wherein the preset fixed overhead is used to determine the delay condition constraint; for each network layer, determining a plurality of structural position combinations to be pruned based on the number of attention heads and the number of filters to be retained in each network layer; for each structural position combination, determining the delay value of each network layer after pruning all module structures indicated by the structural position combination; determining the structural position combination indicated by the minimum delay value as the target structural position combination of each network layer; determining all attention heads indicated by the target structural position combination as the attention heads to be pruned and determining all filters indicated by the target structural position combination as the filters to be pruned.

[0015] Further, the step of pruning the attention heads and filters to be pruned in each network layer of the initial pruned target detection model to obtain the pruned target detection model comprises: determining a plurality of initial structural positions from the plurality of structural positions indicated by the attention heads and filters to be pruned in each network layer, and determining the number of attention heads and the number of filters to be pruned in each network layer; determining a set of to-be-pruned structural positions of each network layer based on the number of attention heads and the number of filters to be pruned in each network layer, wherein the set of to-be-pruned structural positions comprises all structural position combinations except the initial structural position combination, and the initial structural position combination is composed of the plurality of initial structural positions; determining an initial importance layer score of all module structures indicated by the initial structural position combination, and determining a to-be-pruned importance layer score of all module structures indicated by each structural position combination; selecting a minimum to-be-pruned importance layer score from all to-be-pruned importance layer scores, and in the case that the minimum to-be-pruned importance layer score is less than the initial importance layer score, determining all module structures indicated by the structural position combination corresponding to the minimum to-be-pruned importance layer score as the attention heads and filters to be pruned in each network layer; pruning the attention heads and filters to be pruned in each network layer to obtain the pruned target detection model.

[0016] Further, after obtaining the pruned target detection model, the method further comprises: determining a mask matrix corresponding to the pruned target detection model; determining a first activation matrix of the unpruned target detection model and a second activation matrix of the pruned target detection model based on the mask matrix; adjusting the value of the non-0 mask in the mask matrix until the matrix value of the difference matrix is less than a first preset threshold to obtain an initial mask matrix, wherein the difference matrix is determined based on the first activation matrix and the second activation matrix.

[0017] Further, after obtaining the initial mask matrix, the method further includes: determining a first output probability vector of the unpruned target detection model and a second output probability vector of the pruned target detection model based on the initial mask matrix; constructing a divergence loss function based on the first output probability vector and the second output probability vector; adjusting the value of the non-0 mask in the initial mask matrix until the loss value determined by the divergence loss function is less than a second preset threshold value, to obtain a target mask matrix.

[0018] According to another aspect of the embodiments of the present application, a pruning device of a target detection model is also provided. The target detection model includes: a plurality of network layers, each network layer including: a plurality of module structures, the module structure being one of an attention head or a filter, including: an initialization unit configured to initialize a mask of each module structure to obtain an initial mask of each module structure; a first determination unit configured to determine an importance score of each module structure based on a weight of each module structure and the initial mask, and prune an attention head with a minimum importance score and a filter with a minimum importance score to obtain a remaining attention head set and a remaining filter set; a second determination unit configured to determine a target pruning combination under an operation amount condition constraint based on an attention head operation amount of each attention head in the remaining attention head set and a filter operation amount of each filter in the remaining filter set, wherein the target pruning combination is a pruning combination with a minimum total importance score, each pruning combination including: an attention head position of a preset number of attention heads and a filter position of a preset number of filters; a first pruning unit configured to prune an attention head indicated by each attention head position and a filter indicated by each filter position in the target pruning combination to obtain an initial pruned target detection model, and determine an attention head and a filter to be pruned in each network layer in the initial pruned target detection model under a delay condition constraint; and a second pruning unit configured to prune the attention head and the filter to be pruned in each network layer in the initial pruned target detection model to obtain a pruned target detection model.

[0019] Further, the first determination unit includes: a first determination module configured to determine an information loss of the module structure based on the weight of the module structure and the initial mask; a first partial derivative module configured to perform partial derivation on the information loss to obtain a partial derivative value, and determine an information matrix of the module structure based on the partial derivative value and a structure position of the module structure, wherein the structure position is determined based on a network layer number of the target detection model in which the module structure is located and a position number of the network layer; and a first processing module configured to perform diagonal approximation processing on the information matrix to obtain the importance score of the module structure.

[0020] Further, the second determining unit comprises: a second determining module, configured to determine a target operation amount, wherein the target operation amount is used to determine an operation amount condition constraint; a third determining module, configured to determine a plurality of quantity pairs based on the target operation amount, an attention head operation amount of each attention head in the remaining attention head set, and a filter operation amount of each filter in the remaining filter set, wherein each quantity pair contains a preset attention head quantity of an attention head to be pruned and a preset filter quantity of a filter; a fourth determining module, configured to determine all pruning combinations satisfying the quantity pair for each quantity pair, wherein all attention head positions and all preset filter positions in each pruning combination are not completely the same; a fifth determining module, configured to determine a total importance score of each pruning combination; and a sixth determining module, configured to determine a pruning combination with the smallest total importance score as a target pruning combination.

[0021] Further, the third determining module comprises: a first determining submodule, configured to determine an attention head quantity of an attention head in the remaining attention head set and a filter quantity of a filter in the remaining filter set; a first traversal submodule, configured to traverse the attention head quantity and take each traversed quantity as a reserved attention head quantity, wherein the traversal starts from 0 and ends at the attention head quantity, and each traversal increases by one quantity; a second determining submodule, configured to determine a reserved filter quantity corresponding to each reserved attention head quantity based on the target operation amount, an attention head average operation amount, and a filter average operation amount, wherein the attention head average operation amount is calculated based on the attention head operation amount of each attention head in the remaining attention head set, and the filter average operation amount is calculated based on the filter operation amount of each filter in the remaining filter set; and a third determining submodule, configured to determine each preset attention head quantity based on the attention head quantity and each reserved attention head quantity, and determine a preset filter quantity corresponding to each preset attention head quantity based on the filter quantity and the reserved filter quantity corresponding to each reserved attention head quantity, to obtain a plurality of quantity pairs, wherein each quantity pair contains a preset attention head quantity and a preset filter quantity corresponding to the preset attention head quantity, and the preset attention head quantities contained in each quantity pair are different.

[0022] Further, the first pruning unit comprises: a seventh determination module, configured to determine, based on a preset fixed overhead, a number of attention heads and a number of filters reserved in each network layer of the target detection model, wherein the preset fixed overhead is used to determine a delay condition constraint; an eighth determination module, configured to determine, for each network layer, a plurality of structure position combinations to be pruned based on the number of attention heads and the number of filters reserved in each network layer; a ninth determination module, configured to determine, for each structure position combination, a delay value of each network layer after pruning all module structures indicated by the structure position combination; a tenth determination module, configured to determine a structure position combination indicated by a minimum delay value as a target structure position combination of each network layer; and an eleventh determination module, configured to determine all attention heads indicated by the target structure position combination as the attention heads to be pruned and all filters indicated by the target structure position combination as the filters to be pruned.

[0023] Further, the second pruning unit comprises: a twelfth determination module, configured to determine a plurality of initial structure positions from a plurality of structure positions indicated by the attention heads and the filters to be pruned in each network layer, and determine a number of the attention heads to be pruned and a number of the filters to be pruned in each network layer; a thirteenth determination module, configured to determine, based on the number of the attention heads to be pruned and the number of the filters to be pruned in each network layer, a set of structure positions to be pruned in each network layer, wherein the set of structure positions to be pruned comprises all structure position combinations except an initial structure position combination, and the initial structure position combination is composed of the plurality of initial structure positions; a fourteenth determination module, configured to determine an initial importance layer score of all module structures indicated by the initial structure position combination, and determine a to-be-pruned importance layer score of all module structures indicated by each structure position combination; a fifteenth determination module, configured to select a minimum to-be-pruned importance layer score from all to-be-pruned importance layer scores, and determine, in a case where the minimum to-be-pruned importance layer score is smaller than the initial importance layer score, all module structures indicated by a structure position combination corresponding to the minimum to-be-pruned importance layer score as the attention heads and the filters to be pruned in each network layer; and a first pruning module, configured to prune the attention heads and the filters to be pruned in each network layer to obtain a target detection model after pruning is completed.

[0024] Further, the pruning apparatus further comprises: a sixteenth determination module, configured to determine a mask matrix corresponding to the target detection model after pruning is completed; a seventeenth determination module, configured to determine, based on the mask matrix, a first activation matrix of the target detection model before pruning and a second activation matrix of the target detection model after pruning is completed; and a first adjustment module, configured to adjust a value of a non-0 mask in the mask matrix until a matrix value of a difference matrix is smaller than a first preset threshold value to obtain an initial mask matrix, wherein the difference matrix is determined based on the first activation matrix and the second activation matrix.

[0025] Further, the pruning device further includes: an eighteenth determination module, configured to, after obtaining the initial mask matrix, determine, based on the initial mask matrix, a first output probability vector of the unpruned target detection model and a second output probability vector of the pruned target detection model; a first construction module, configured to construct a divergence loss function based on the first output probability vector and the second output probability vector; and a second adjustment module, configured to adjust a value of a non-0 mask in the initial mask matrix until a loss value determined by the divergence loss function is less than a second preset threshold value, to obtain a target mask matrix.

[0026] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a non-volatile computer readable storage medium, the non-volatile computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the pruning method of the target detection model of any one of the above.

[0027] According to another aspect of the embodiments of the present application, an electronic device is also provided, including one or more processors and a memory, the memory being configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the pruning method of the target detection model of any one of the above.

[0028] In the present application, the mask of each module structure is initialized to obtain an initial mask of each module structure, the importance score of each module structure is determined based on the weight of each module structure and the initial mask, and the attention head with the minimum importance score and the filter with the minimum importance score are pruned to obtain a remaining attention head set and a remaining filter set, the target pruning combination is determined under the operation amount condition constraint based on the attention head operation amount of each attention head in the remaining attention head set and the filter operation amount of each filter in the remaining filter set, the attention head indicated by each attention head position in the target pruning combination and the filter indicated by each filter position are pruned to obtain an initial pruned target detection model, and the attention head and the filter to be pruned in each network in the initial pruned target detection model are determined under the delay condition constraint, the attention head and the filter to be pruned in each network in the initial pruned target detection model are pruned to obtain the pruned target detection model, thereby solving the technical problem that the accuracy of determining the structure to be pruned in the model is low in the related art, resulting in poor accuracy of the pruned model.

[0029] In the present application, the balance between model compression and performance maintenance is achieved by pruning the attention heads and filters with the minimum importance score in the target detection model through multi-stage unstructured pruning, finding the target pruning combination (i.e., the pruning combination with the minimum total importance score under the FLOPs limit) under the operation amount condition constraint, and then pruning the attention heads and filters to be pruned in each layer of the model under the delay condition constraint, so as to achieve the purpose of simplifying the model structure under the premise of ensuring the detection accuracy of the model, ensure that the pruned model can run efficiently in a resource-limited environment, and at the same time maintain a high detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings, which are included to provide a further understanding of the present application and constitute a part of this application, illustrate certain illustrative embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0031] Figure 1 is a flowchart of an optional pruning method for a target detection model according to an embodiment of the present application;

[0032] Figure 2 is a schematic diagram of an optional distillation structure according to an embodiment of the present application;

[0033] Figure 3 is a schematic diagram of an optional multi-stage unstructured pruning process according to an embodiment of the present application;

[0034] Figure 4 is a schematic diagram of an optional pruning device for a target detection model according to an embodiment of the present application;

[0035] Figure 5 is a hardware structure block diagram of an electronic device (or mobile device) for a pruning method for a target detection model according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0037] It should be noted that the terms "first", "second" and the like in the present application are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0038] To facilitate the understanding of the present application by those skilled in the art, the following explanations are made to some terms or nouns involved in the embodiments of the present application:

[0039] Structured pruning: a kind of technology widely used in deep neural network model compression, the core idea of which is to cut the model structure in units of channels, convolution kernels, attention heads or blocks. Although structured pruning has advantages in engineering deployment, it still has significant shortcomings in high-precision remote sensing image target detection tasks, such as coarse pruning granularity, small adjustable space, obvious compression performance bottleneck, and limited pruning location.

[0040] Unstructured pruning: a method of realizing model compression and acceleration by removing unimportant weight connections in neural networks, which can start from a finer granularity and perform sparsification at the weight or parameter level, and can more flexibly retain key paths and eliminate redundant paths. Therefore, it is feasible for the present embodiment to use unstructured pruning technology to lighten the target detection model.

[0041] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) collected and related by the present application are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards in relevant regions, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user selection authorization or refusal. For example, the system and related users or institutions are provided with an interface, and before obtaining the relevant information, the interface needs to send a request for obtaining the relevant information to the aforementioned user or institution, and after receiving the consent information feedback from the aforementioned user or institution, the relevant information is obtained.

[0042] With the development of model compression, network structure search, distillation learning and other technologies, there are various lightweight target detection methods, which have achieved good detection effect in remote sensing images, and have alleviated the bottleneck problem of detection model in computing resources and deployment efficiency. These methods improve the inference speed and deployment convenience of the model on the basis of ensuring a certain detection accuracy, providing a feasible basis for practical application. However, the current lightweight method still has problems in the design of compression strategy, such as strong structure rigidity, rough pruning strategy and uncontrollable precision loss, especially in the high-resolution and mixed target scene of remote sensing images, it is still difficult to achieve efficient and high-precision. Based on this, the present application proposes a lightweight method for remote sensing target detection model based on unstructured pruning and distillation fine-tuning, which can maximize the retention of key features while ensuring computational efficiency in multi-stage mask search and rearrangement optimization, ensuring the robustness and accuracy of the model.

[0043] In order to solve the problems of inaccurate pruning importance evaluation, insufficient pruning granularity, and not fine resource constraint control, the present application proposes a FLOPs constraint pruning algorithm, a delay constraint pruning algorithm, a mask rearrangement greedy exchange strategy, a linear least squares optimization, and a knowledge distillation optimization method. Among them, the FLOPs constraint pruning algorithm can traverse the possible number of retained heads, calculate the number of allowed filters based on the linear distribution strategy, and then use the total importance score minimization target to select the optimal pruning combination; the delay constraint pruning algorithm can model the delay function as a piecewise linear function, divide the fixed delay and linear growth area, retain the heads and filters of the fixed overhead part, and use the greedy strategy to select the least important element for pruning in the linear area; the mask rearrangement greedy exchange strategy compares the importance scores of the pruned and unpruned elements in the current layer, and if the pruned element score is higher than the unpruned element, the pruning state is exchanged, and the iteration is repeated until there is no better exchange. Linear least squares optimization models the mask optimization problem as a linear least squares problem, uses the LSMR (Least Squares Method with Residual minimization, an iterative algorithm for solving sparse linear system least squares problem) solver of CuPy (numerical calculation library) to solve, executes layer by layer, restores the model signal, and does not need to be retrained, with low calculation cost.

[0044] The present application will be described in detail below in conjunction with various embodiments.

[0045] Embodiment one

[0046] According to an embodiment of the present application, an embodiment of a pruning method of a target detection model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0047] Figure 1 is a flowchart of an optional pruning method of a target detection model according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0048] Step S101, initializing the mask of each module structure to obtain an initialized mask of each module structure.

[0049] In an embodiment of the present application, the target detection model comprises a multi-layer network, each layer of the network comprising a plurality of module structures, the module structure being one of a multi-head attention (MHA) or a filter (i.e. a feed-forward network filter, FFN).

[0050] In an embodiment of the present application, the mask of each module structure (such as multi-head attention MHA or feed-forward network FFN) can be initialized first to obtain an initialized mask of each module structure. Here, the mask is used to indicate which part (such as attention head or filter) in the model is retained or pruned. When initializing the mask, all mask variables are set to 1, indicating that at this time all attention heads (MHA) and feed-forward network filters (FFN) in the model are retained and have not been pruned. In this way, it can be ensured that the model starts from a complete structure, so that the initial output is consistent with the original model, providing an accurate and unified starting point for subsequent pruning operations.

[0051] Exemplarily, a mask matrix matching the module structure parameters in the model can be created, all elements of which are initialized to 1. This mask matrix will accompany the model in the subsequent pruning and optimization process, recording which parameters are pruned and which are retained.

[0052] Step S102, determining the importance score of each module structure based on the weight of each module structure and the initialized mask, and pruning the attention head with the smallest importance score and the filter with the smallest importance score to obtain a remaining attention head set and a remaining filter set.

[0053] In the embodiments of the present application, a mask search can be performed to determine the attention heads and filters that need to be pruned. The importance score of each attention head and each filter can be evaluated based on the weight of each module structure and the initialized mask, and the diagonal approximation of the Fisher information matrix (i.e., a tool for estimating the uncertainty of model parameters) is used to measure the contribution of the module structure to the model performance. Here, the Fisher information matrix is a statistical quantity used to measure the degree of contribution of model parameters to the output, and the attention head or filter with a lower score is considered to contribute less to the model performance and is thus a candidate for pruning.

[0054] In the embodiments of the present application, after calculating the importance score of each module structure, the attention head with the minimum importance score and the filter with the minimum importance score can be pruned to obtain a remaining attention head set and a remaining filter set.

[0055] For example, the diagonal elements of the Fisher information matrix can be calculated as the importance score of each parameter (attention head or filter), all scores are traversed, the attention head and filter with the lowest score are selected for pruning, and the pruned attention head and filter set respectively constitute the remaining attention head set and the remaining filter set for subsequent optimization and fine-tuning.

[0056] In step S103, based on the attention head operation amount of each attention head in the remaining attention head set and the filter operation amount of each filter in the remaining filter set, a target pruning combination is determined under the operation amount condition constraint, wherein the target pruning combination is a pruning combination with the minimum total importance score, and each pruning combination includes attention head positions of a preset number of attention heads and filter positions of a preset number of filters.

[0057] In the embodiments of the present application, a greedy strategy is used for optimization selection under different resource limitation conditions (such as FLOPs or delay). For example, under the FLOPs constraint (operation amount condition constraint), the possible remaining head number is traversed, the number of filters that can be retained is calculated for each configuration, and the combination with the lowest total importance score is selected as the target pruning combination to be pruned. That is, after the remaining attention head and filter set are determined, the optimal pruning combination is found according to the operation amount (FLOPs) condition constraint. The goal of this combination is to minimize the total importance score while satisfying the FLOPs constraint. Through this strategy, the calculation efficiency of the model can be improved without sacrificing performance.

[0058] Here, each pruning combination determined under the operation amount condition constraint includes attention head positions of a preset number of attention heads and filter positions of a preset number of filters, and the pruning combination with the minimum total importance score is determined as the target pruning combination by calculating the total importance score of each pruning combination.

[0059] In step S104, the attention heads indicated by each attention head position in the target pruning combination and the filters indicated by each filter position are pruned to obtain an initial pruned target detection model, and the attention heads and filters to be pruned in each network layer in the initial pruned target detection model are determined under the delay condition constraint.

[0060] In the embodiments of the present application, the delay is modeled as a piecewise linear function under the delay condition constraint, and the fixed delay and the linear growth part are processed respectively, and then the importance is selected by the greedy strategy to obtain the lowest heads and filters, so that the pruning result retains useful information as much as possible under the premise of meeting the delay limit. That is, after determining the target pruning combination, the attention heads and filters indicated in the target pruning combination are pruned. After obtaining the initial pruned model, in order to further adapt to resource-constrained devices, the delay is modeled as a piecewise linear function, so as to determine which attention heads and filters need to be pruned next under the delay condition constraint. In this way, the inference delay of the model is ensured to be within a controllable range, and it is particularly suitable for scenes with high real-time requirements.

[0061] For example, according to the target pruning combination, the mask is updated, that is, the mask values corresponding to the attention heads and filters that need to be pruned are set to 0. Under the delay condition constraint, the delay of each network layer in the initial pruned model is evaluated to determine the attention heads and filters to be pruned in each network layer in the initial pruned target detection model.

[0062] In step S105, the attention heads and filters to be pruned in each network layer in the initial pruned target detection model are pruned to obtain a target detection model after pruning is completed.

[0063] In the embodiments of the present application, the pruning operation is performed on the attention heads and filters to be pruned in each network layer in the initial pruned target detection model to complete the final compression of the model. In this way, it is ensured that the model is not only optimized in terms of computing resources (FLOPs), but also the inference delay in actual deployment is controlled, so as to realize the lightweight and efficient operation of the model.

[0064] In some optional embodiments, after determining the number of attention heads and the number of filters to be pruned in each network layer, a mask rearrangement optimization strategy can be performed to adjust the pruning position of each network layer to further improve the performance of the model.

[0065] In summary, the pruning of the attention head and the filter with the minimum importance score in the target detection model can be performed, and the target pruning combination (i.e., the pruning combination with the minimum total importance score under the FLOPs limit) can be found under the operation amount condition constraint to balance the model compression and performance maintenance. Then, the pruning of the attention head and the filter to be pruned in each layer of the network in the model can be performed under the delay condition constraint to achieve the purpose of simplifying the model structure under the premise of ensuring the detection accuracy of the model, so that the pruned model can be efficiently run in a resource-limited environment while maintaining a high detection accuracy, thereby solving the technical problem in the related art that the accuracy of determining the structure to be pruned in the model is low, resulting in poor accuracy of the pruned model.

[0066] To improve the accuracy of determining the importance score of each module structure, in the pruning method of the target detection model provided in Embodiment One of the present application, the information loss of the module structure is determined based on the weight of the module structure and the initialization mask; the partial derivative of the information loss is obtained to obtain a partial derivative value, and the information matrix of the module structure is determined based on the partial derivative value and the structure position of the module structure, wherein the structure position is determined based on the number of network layers in the target detection model where the module structure is located and the position number of the network layer.

[0067] In the embodiments of the present application, the information loss of the module structure can be calculated based on the weight of the module structure and the initialization mask. Specifically, the initialization mask marks all parameters as a reserved state, and the weight is a neural network parameter value obtained through a training process. In this way, the loss of the model in information transmission and learning ability if part of the parameters is pruned can be quantified.

[0068] For example, the Fisher information matrix can be used to estimate the information loss, which reflects the sensitivity of the parameter to the loss function. The information loss The calculation formula of the information loss is as follows:

[0069]

[0070] where H is the Hessian matrix of the loss function with respect to the mask m (i.e., a square matrix composed of second-order partial derivatives of a multivariate function at a certain point), is the weight matrix of the module structure.

[0071] After the information loss is determined, the partial derivative of the information loss can be taken to obtain a partial derivative value These partial derivatives provide local information about the importance of parameters, indicating the potential impact of pruning on model performance. Then, based on these partial derivatives and the network layer number and location information of the module structure in the target detection model (i.e., the position number of the network layer where it is located), an information matrix is determined to comprehensively reflect the importance of different parts of the model.

[0072] Information matrix The approximate Hessian matrix is calculated as follows:

[0073]

[0074] wherein, represents the partial derivative of the information loss, (x, y) represents the structural position of the module structure, x represents the network layer number of the module structure in the target detection model, y represents the position number of the network layer where the module structure is located, D represents a position set composed of the structural positions of all module structures in the target detection model, and |D| represents the number of structural positions.

[0075] Here, the calculation of the Fisher information matrix involves the second-order derivative of the loss function with respect to the parameters, but in practice, due to its huge computational complexity, a diagonal approximation can be used, i.e., only the square of the derivative of the parameter itself is considered, while the interaction between parameters is ignored, thereby simplifying the calculation process.

[0076] After diagonal approximation of the information matrix, the target is simplified as:

[0077]

[0078] wherein, represents the element on the diagonal of the information matrix, and Z(m) represents a set of diagonal elements of the information matrix.

[0079] In the embodiments of the present application, by performing diagonal approximation processing on the information matrix, the computational complexity is reduced, and the "importance score" of each module structure is extracted. The higher the importance score, the greater the contribution of the corresponding parameter to the model performance. In this way, those parameters that are crucial to the model can be preferentially retained during the pruning process, thereby reducing the risk of performance degradation while reducing the model complexity and computational load.

[0080] In this embodiment, by pruning the module structures with smaller pruning importance scores, the computational resource demand and inference delay of the target detection model can be reduced without significantly sacrificing detection performance, making it more suitable for deployment and real-time inference in resource-constrained environments (such as drones, satellites, etc.).

[0081] In order to improve the accuracy of determining the target pruning combination, in the pruning method of the target detection model provided in Embodiment One of the present application, a target operation amount is determined, wherein the target operation amount is used to determine the operation amount condition constraint; based on the target operation amount, the attention head operation amount of each attention head in the remaining attention head set, and the filter operation amount of each filter in the remaining filter set, a plurality of quantity pairs are determined, wherein each quantity pair contains a preset attention head quantity of the attention head to be pruned and a preset filter quantity of the filter; for each quantity pair, all pruning combinations that meet the quantity pair are determined, wherein all attention head positions and all preset filter positions in each pruning combination are not completely the same; the total importance score of each pruning combination is determined; and the pruning combination with the smallest total importance score is determined as the target pruning combination.

[0082] In the embodiments of the present application, the pruning problem under the FLOPs constraint can be modeled as follows:

[0083]

[0084] wherein F head ||m MHA ||0 represents the attention head operation amount of the attention head, F filter ||m FFN ||0 represents the filter operation amount of the filter, and C represents the target FLOPs.

[0085] In the embodiments of the present application, the target operation amount (FLOPs) is a key parameter for setting the operation amount condition constraint, which represents the number of floating point operations performed per second. In the process of model lightweight, the target operation amount is used to guide the pruning operation, to ensure that the model after pruning maintains a certain performance while reducing the operation amount to a pre-set threshold. A reasonable FLOPs constraint can be determined according to the actual application scenario and the computing power of the hardware device, to balance the computing efficiency and detection accuracy of the model. For example, by analyzing the original operation amount of the model and the computing power of the target device, a target operation amount value can be set. For example, based on the computing resource limitation of a drone or a satellite, the target operation amount can be set to 50% or less of the original model FLOPs.

[0086] In the embodiments of the present application, a plurality of quantity pairs can be determined according to the target operation amount, the attention head operation amount of each attention head in the remaining attention head set, and the filter operation amount of each filter in the remaining filter set. Each quantity pair includes a preset attention head quantity of the attention head to be pruned and a preset filter quantity of the filter. In this way, the optimal combination of different attention heads and filter remaining quantities under the FLOPs constraint condition can be found. For example, the target operation amount can be taken as a benchmark to traverse different attention head reservation quantities (n), calculate the corresponding reservable filter quantity (f) after reserving n attention heads to meet the FLOPs constraint, and obtain a plurality of quantity pairs according to each attention head reservation quantity (n) and the corresponding reservable filter quantity (f).

[0087] In the embodiments of the present application, for each quantity pair, all possible pruning combinations are determined. Here, the pruning combination refers to the set of specific parameters that are pruned in the model after a certain number of attention heads and filters are reserved. By exhaustively enumerating all possible combinations, the influence of different pruning schemes on the model performance can be more comprehensively evaluated to provide a basis for subsequent decision-making. For example, all possible pruning combinations of attention heads and filters are traversed, and considering that pruning will change the model structure, it is necessary to ensure that each combination can completely cover all layers of the model while reserving a preset number of heads and filters. All pruning combinations that meet the quantity pair condition can be generated by an iterative algorithm or an enumeration method.

[0088] For each pruning combination, its total importance score (i.e., the importance score of the pruned attention heads and filters) is calculated, which can be obtained based on the Fisher information matrix to reflect the potential influence of the pruned elements in the combination on the overall performance of the model. The lower the total score, the smaller the performance loss of the model after pruning, and it is the preferred pruning scheme. Then, from all pruning combinations, the combination with the smallest total importance score is selected as the target pruning combination. This combination can maximize the detection performance of the model while meeting the FLOPs constraint, thereby achieving the lightweight target.

[0089] In the embodiments, the model operation amount can be effectively reduced while ensuring the detection performance of the model, so that the lightweight model is more suitable for running on hardware devices with limited resources, improving the real-time performance and flexibility of remote sensing target detection. This pruning strategy based on the target operation amount and Fisher information avoids the problem of significant performance decline that may be caused by extensive pruning, and ensures that the pruned model can still complete the target detection task with high quality.

[0090] In order to improve the accuracy of determining the plurality of groups of quantity pairs, in the pruning method of the target detection model provided in Embodiment One of the present application, the number of attention heads in the remaining attention head set is determined, and the number of filters in the remaining filter set is determined; the number of attention heads is traversed, and each time the traversed number is taken as the reserved attention head number, wherein the traversal starts from 0 and ends at the number of attention heads, and each time the number is increased by one; for each reserved attention head number, based on the target operation amount, the average operation amount of the attention head and the average operation amount of the filter, the reserved filter number corresponding to the reserved attention head number is determined, wherein the average operation amount of the attention head is calculated based on the operation amount of each attention head in the remaining attention head set; the average operation amount of the filter is calculated based on the operation amount of each filter in the remaining filter set; based on the number of attention heads and each reserved attention head number, each preset attention head number is determined, and based on the number of filters and the reserved filter number corresponding to each reserved attention head number, the preset filter number corresponding to each preset attention head number is determined, to obtain a plurality of groups of quantity pairs, wherein the quantity pair contains the preset attention head number and the preset filter number corresponding to the preset attention head number, and each group of quantity pairs contains different preset attention head numbers.

[0091] In the embodiment of the present application, the number of attention heads in the remaining attention head set can be determined first, and the number of filters in the remaining filter set can be determined. Then, starting from 0, the number of attention heads is increased by one at a time until the number of attention heads is reached. In each traversal, the currently traversed number of attention heads is determined as the reserved attention head number. In this way, the relationship between the performance and resource consumption of the model under different reserved attention head numbers can be explored to find the balance point of resource efficiency and performance. For example, using a loop structure, the number of reserved attention heads is started from 0 and increased by 1 each time until the number of attention heads is reached.

[0092] For each traversed reserved attention head number, based on the target operation amount (FLOPs), the average operation amount of the attention head and the average operation amount of the filter, the number of filters to be reserved after reserving a certain number of attention heads is determined. In this way, it is ensured that the operation amount of the pruned model can meet the constraint of the target operation amount, while trying to minimize the impact on the performance of the model. The calculation formula of the reserved filter number is as follows:

[0093]

[0094] Wherein, f represents the number of filters allowed to be reserved, C is the FLOPs constraint (target operation amount), F head , F filterrespectively, n represents the current number of reserved attention heads, and the number of reserved filters is calculated by linear distribution.

[0095] The number of each preset attention head (i.e., the number of attention heads that need to be pruned) can be determined according to the number of attention heads and the number of each reserved attention head, and the number of preset filters corresponding to each preset attention head (i.e., the number of filters that need to be pruned) can be determined according to the number of filters and the number of reserved filters corresponding to each reserved attention head. By enumerating different preset attention head numbers and preset filter numbers corresponding thereto, a plurality of number pairs can be obtained, each number pair including a preset attention head number and a preset filter number corresponding to the preset attention head number, and the preset attention head numbers included in each number pair are different.

[0096] In the embodiment, the combination scheme of retaining different numbers of attention heads and filters can be determined under the premise of meeting the target operation amount constraint. This provides a plurality of possible pruning configurations for the lightweight of the model. By further evaluating and optimizing each configuration, a pruning scheme that meets the resource constraint and maximally maintains the detection performance of the model can be finally selected. Not only the controllability and flexibility of model pruning are ensured, but also the pruning strategy of the model can be effectively guided to ensure that the pruned model can efficiently run in an edge device or a resource-constrained scene while maintaining a high target detection accuracy.

[0097] In order to improve the accuracy of determining the attention heads and filters to be pruned in each layer of the network under the delay condition constraint, in the pruning method of the target detection model provided in Embodiment One of the present application, the number of attention heads and the number of filters retained in each layer of the network in the target detection model are determined based on a preset fixed overhead, wherein the preset fixed overhead is used to determine the delay condition constraint; for each layer of the network, a plurality of structural position combinations to be pruned are determined based on the number of attention heads and the number of filters retained in each layer of the network; for each structural position combination, the delay value of each layer of the network after pruning all module structures indicated by the structural position combination is determined; the structural position combination indicated by the minimum delay value is determined as the target structural position combination of each layer of the network; all attention heads indicated by the target structural position combination are determined as the attention heads to be pruned, and all filters indicated by the target structural position combination are determined as the filters to be pruned.

[0098] In the embodiment of the present application, under the delay condition constraint, the delay function can be approximated to a piecewise linear form, which is as follows:

[0099]

[0100] wherein, LAT(m l ) represents the delay of the lth layer of network, ml ||m represents the structure of the l-th layer network. l ||0 represents the number of module structures of the l-th layer network. c represents the fixed overhead, i.e., the basic delay that will be generated regardless of the change in the number of non-zero elements in the mask within a certain range. T is a linear region threshold value for dividing different delay calculation rules: when the number of non-zero elements in the mask is in the interval (0, T], the delay remains constant c; if the number of non-zero elements exceeds T, the delay will increase linearly with a slope a, reflecting the additional delay cost brought by the non-zero elements exceeding the threshold.

[0101] Under this delay condition constraint, the fixed overhead processing can be performed first, i.e., a certain number of attention heads and filters are reserved in each layer. head (i.e., the number of attention heads reserved in each layer network) attention heads and the first T filter (i.e., the number of filters reserved in each layer network) filters. In this way, the core components that constitute the fixed overhead c are ensured to be reserved. By reserving these preset number of attention heads and filters, the basic component c of the delay is maintained, which plays a key role in the stability of the overall delay of the model.

[0102] Then the linear region optimization is performed, i.e., among the remaining attention heads and filters (after completing the reservation of the fixed overhead part), a greedy strategy is used to select the pruning position. This greedy strategy aims to optimize the delay within the linear region. With the piecewise linear characteristic of the delay function, the elements (attention heads or filters) that can most effectively reduce the delay after pruning while satisfying the linear region threshold T constraint can be identified, ensuring that the model achieves the optimal balance between model performance and delay under the delay constraint.

[0103] Specifically, the delay characteristics of the target detection model are analyzed to determine a preset fixed overhead. For example, based on the benchmark delay of the model on a specific hardware, a minimum delay threshold can be set to ensure the basic running speed of the model. According to the preset fixed overhead, the number of attention heads and the number of filters that need to be reserved in each layer network are calculated to ensure that the delay of the pruned model will not be lower than the preset threshold.

[0104] For each layer network, based on the determined number of attention heads and filters, a plurality of structure position combinations to be pruned are generated. Here, the structure position combination refers to the set of possible pruning positions in the model after a certain number of attention heads and filters are reserved. The purpose of generating these combinations is to find the optimal pruning scheme by comparing the impact of different pruning positions on the model delay.

[0105] For each generated structure position combination, the delay value of each layer network after pruning is calculated to compare the influence of different pruning schemes on the running speed of the model. Among all the calculated delay values, the structure position combination corresponding to the minimum delay value is selected as the target structure position combination of each layer network. In this way, the delay of the model is minimized on the basis of meeting the preset fixed overhead, thereby improving the response speed and resource utilization efficiency of the model. Then, based on the target structure position combination of each layer network, the attention heads and filters indicated in the combination are determined as the objects to be pruned.

[0106] In the present embodiment, model structure optimization under delay condition constraints is achieved, which can identify and prune attention heads and filters in each layer network that contribute more to delay, ensuring that the pruned model not only meets the preset fixed overhead requirement, but also achieves the minimum delay, which is particularly important for real-time processing and resource-constrained environments. The implementation of the pruning strategy not only reduces the complexity of the model, but also ensures efficient operation of the model on specific hardware, providing a lightweight and fast response solution for remote sensing target detection.

[0107] In order to improve the accuracy of pruning the target detection model, in the pruning method of the target detection model provided in Embodiment One of the present application, a plurality of structure positions indicated by the attention heads and filters to be pruned in each layer network are determined as a plurality of initial structure positions, and the number of attention heads and the number of filters to be pruned in each layer network are determined; based on the number of attention heads and the number of filters to be pruned in each layer network, a set of structure positions to be pruned in each layer network is determined, wherein the set of structure positions to be pruned includes all structure position combinations except the initial structure position combination, and the initial structure position combination is composed of the plurality of initial structure positions; the initial importance layer scores of all module structures indicated by the initial structure position combination are determined, and the to-be-pruned importance layer scores of all module structures indicated by each structure position combination are determined; the minimum to-be-pruned importance layer score is selected from all to-be-pruned importance layer scores, and in the case that the minimum to-be-pruned importance layer score is less than the initial importance layer score, all module structures indicated by the structure position combination corresponding to the minimum to-be-pruned importance layer score are determined as the attention heads and filters to be pruned in each layer network; the attention heads and filters to be pruned in each layer network are pruned to obtain a target detection model after pruning is completed.

[0108] In the embodiments of the present application, after determining the initial structure position that needs to be pruned under the delay condition constraint, mask rearrangement can be performed, that is, the pruning position is adjusted to optimize the model performance while keeping the number of pruned layers unchanged. It can be assumed that there is no interaction between the mask variables of different layers, the Fisher information matrix is approximated to a block diagonal form, and the optimization problem is decomposed into independent optimization of each layer. Then a greedy exchange strategy is adopted, the result of the mask search is used as the mask at the beginning, and the importance scores of the pruned and unpruned attention heads or filters are compared. If the importance of the pruned item is higher than that of the unpruned item, the pruning state of the two is exchanged. By adjusting the pruning position, capturing the interaction within the layer and compensating for the layer correlation ignored in the mask search, the information loss can be reduced and the model performance can be improved.

[0109] In the embodiments of the present application, it is assumed that there is no interaction between the mask variables of different layers, and based on this, the Fisher information matrix can be approximated to a block diagonal form, which is expressed as:

[0110]

[0111] wherein, represents the Fisher matrix block of the lth layer, corresponding to a multi-head attention (MHA) layer or a feed-forward network (FFN) layer. This block diagonal approximation effectively decouples the cross-layer complexity, making the optimization problem easier to handle.

[0112] Under this approximation, the optimization problem is decomposed into independent optimization of each layer. For the lth layer, the optimization objective is:

[0113]

[0114] wherein, represents the importance layer score of the lth layer.

[0115] By independently optimizing each layer while keeping the number of pruned layers unchanged, the information loss within each layer can be minimized, and the specific pruning position within the layer can be adjusted. Essentially, it allows more fine-tuning of the pruning scheme at the layer granularity, ensuring the structural integrity and functional performance of each layer (such as MHA or FFN) to be preserved as much as possible. This hierarchical independent optimization not only simplifies the overall pruning optimization process, but also provides a relatively clear mechanism for balancing model sparsity and performance preservation at the fine-grained layer level.

[0116] In the embodiments of the present application, the greedy exchange strategy is used as the optimization of model pruning. In the initialization stage, the results of the mask search are used as the initial mask, which carries the achievements of the early mask search, and according to the specific criteria, the pruned attention heads or filters are distinguished, which lays the foundation for the subsequent exchange operation. In the exchange candidate selection stage, for the pruned attention heads or filters, the importance score is calculated by means of the Fisher matrix block of the corresponding layer, which quantifies the importance of the pruned elements; for the unpruned heads or filters, the importance of the retained elements is calculated. In the exchange condition judgment stage, if the pruning states of i and j are exchanged (wherein, represents the importance layer score of the layer where the module structure i is used as the pruning object, represents the importance layer score of the layer where the module structure j is used as the pruning object), the principle of giving priority to retaining important elements is followed, and the model performance is optimized by replacing the low-importance retained elements with more important pruned elements. In the traversal update stage, the above process is repeated until all candidates complete a round of evaluation, ensuring that the layer exchange combinations are fully reviewed, and the potential opportunities for improving the mask configuration are explored through iteration.

[0117] On the level of objective function optimization, the theoretical expression seems unchanged, but the total loss can be reduced by exchanging high-importance pruned elements with low-importance retained elements. This strategy focuses on retaining high-value (higher importance score) elements and removing low-importance elements, and the greedy exchange based on importance score continuously optimizes the mask configuration, seeking a better balance between model sparsity and performance. With the continuous application of the exchange condition, the effective loss is minimized, the running efficiency and actual effect of the model under the pruning framework are improved, the damage to the key functions of the model during pruning is reduced, the computational resources are used efficiently, and the model maintains the performance to the maximum extent while meeting the pruning requirements.

[0118] When the pruning states of i and j are exchanged, the total loss change is assumed to be The exchange effectiveness is determined by judging the size relationship between and 0: if then it indicates that this exchange can reduce the loss and is an effective operation. This analysis based on intra-layer interaction refines the pruning strategy, enabling the model to more reasonably handle the relationship between elements within the layer during pruning, further optimizing the balance between model performance and sparsity. Not only does it focus on the importance of a single element, but it also considers the interaction between elements, improving the scientificity and effectiveness of the pruning strategy, ensuring that the pruning operation is carried out under more comprehensive theoretical support, and helping the model to maintain good performance during the sparsification process.

[0119] Specifically, a plurality of structure positions of the attention heads and filters to be pruned in each network layer can be determined as a plurality of initial structure positions, while the number of the attention heads and the number of the filters to be pruned in each network layer are determined. Based on the number of the attention heads and the number of the filters to be pruned in each network layer, a set of structure positions to be pruned in each network layer is determined, which contains all possible combinations of structure positions in addition to the initial combination of structure positions. Then, initial importance layer scores of all module structures indicated by the initial combination of structure positions are calculated, and importance layer scores of all module structures indicated by each combination of structure positions to be pruned are calculated. For example, using the Fisher information matrix, the importance of each attention head and filter to be pruned in each network layer to the model performance is evaluated to form the importance layer score of the layer network. From all the calculated importance layer scores to be pruned, the smallest score is selected. If the smallest importance layer score to be pruned is smaller than the initial importance layer score, it indicates that a combination of structure positions with less performance degradation after pruning is found, and all module structures indicated by this combination can be determined as the attention heads and filters to be pruned in each network layer. Then, the actual pruning operation is performed on the determined attention heads and filters to be pruned in each network layer, so as to obtain the target detection model after pruning.

[0120] In the embodiment, the module structure with the least impact on the model performance in each network layer can be accurately located for pruning, and the model performance maximization reservation under resource constraints is realized. Not only the calculation efficiency of the model is improved and the resource consumption is reduced, but also in the pruning process, unnecessary performance loss is avoided through careful performance evaluation and optimization, and it is ensured that the model after pruning can still maintain high accuracy and real-time processing capability in the remote sensing target detection task.

[0121] In order to optimize the mask of the target detection model, in the pruning method of the target detection model provided in Embodiment One of the present application, after obtaining the target detection model after pruning, a mask matrix corresponding to the target detection model after pruning is determined; based on the mask matrix, a first activation matrix of the target detection model before pruning and a second activation matrix of the target detection model after pruning are determined; the value of the non-0 mask in the mask matrix is adjusted until the matrix value of the difference matrix is less than a first preset threshold, to obtain an initial mask matrix, wherein the difference matrix is determined based on the first activation matrix and the second activation matrix.

[0122] In the embodiments of the present application, after obtaining the target detection model with pruning completed, mask tuning can be performed, which can restore the model output signal by fine-tuning the value of the non-zero mask variable without changing the determined pruning structure. The mask tuning problem can be converted into a linear least squares problem, that is, the output activation difference between the pruned model and the original model is minimized. By calculating the activation matrix of the original model (first activation matrix) and the activation matrix after pruning (second activation matrix), an output difference vector is obtained. Then, the LSMR solver of CuPy is used to solve the linear equation set, while a regularization term is added and the range of mask value is limited to enhance stability. Mask tuning is performed from the first layer to the last layer in turn, and the output signal of the entire model is gradually restored, effectively reducing the accuracy decrease caused by pruning without retraining the model. The mask tuning method using linear least squares does not need to retrain the entire model, only adjusts the mask variable, and the calculation cost is extremely low (only a small amount of samples and a few minutes of time). However, although the model is tuned, the accuracy of the model is still greatly reduced compared to before pruning, so the distillation method can be used for fine-tuning after pruning, which can obtain a lightweight model with inference speed and will not greatly reduce the accuracy of the model.

[0123] Specifically, the mask matrix corresponding to the target detection model with pruning completed can be determined first. The mask matrix is a two-dimensional array or matrix used to indicate which parameters in the model are retained and which are pruned. Each element in the mask matrix corresponds to a module structure in the model, such as an attention head or a filter, and its value (0 or non-0) identifies whether the parameter participates in the calculation process of the model. After pruning, the value corresponding to each pruned module structure in the mask matrix is 0, and the value corresponding to the retained module structure is non-0. Then, the same input data set can be used to perform forward propagation through the unpruned and pruned models respectively, and the outputs of each layer are recorded as activation matrices. Then, using the least squares method in linear algebra, the values of the non-0 elements in the mask matrix are adjusted to minimize the difference between the first activation matrix and the second activation matrix, that is, to construct a difference matrix, and to make the matrix value less than a first preset threshold (which can be set according to actual conditions) through optimization, to obtain an initial mask matrix. During the adjustment process, the mask value is limited to change within a certain range to maintain the reasonableness and stability of the internal structure of the model.

[0124] In the embodiments, not only can the lightweight pruning of the target detection model be accurately implemented, but also the output difference between the pruned model and the original model can be effectively reduced through fine-tuning of the mask matrix value.

[0125] In order to further optimize the mask of the target detection model, in the pruning method of the target detection model provided in Embodiment One of the present application, after obtaining the initial mask matrix, based on the initial mask matrix, a first output probability vector of the unpruned target detection model and a second output probability vector of the pruned target detection model are determined; based on the first output probability vector and the second output probability vector, a divergence loss function is constructed; the value of the non-0 mask in the initial mask matrix is adjusted until the loss value determined by the divergence loss function is less than a second preset threshold, and a target mask matrix is obtained.

[0126] In the embodiments of the present application, the mask optimization is performed by constructing a distillation structure. Figure 2 is a schematic diagram of an optional distillation structure according to an embodiment of the present application, as shown in Figure 2 The distillation structure is composed of a teacher network (T) and a student network (S). The teacher network is a complete Transformer Encoder (i.e. the unpruned target detection model) with strong feature representation capability, which encodes the input data (i.e. multiple remote sensing images) through linear projection of flattened patches, position embedding, and Transformer Encoder module, and outputs high-credibility prediction results (logits) through the detection network (Detection Network) and the RoI head (Region of Interest head). The student network introduces an unstructured pruning technique to simplify the Transformer Encoder, reduces the parameter quantity and computational overhead while minimizing the performance loss, and forms a lightweight Pruned Transformer Encoder structure (i.e. the pruned target detection model). The input data (i.e. multiple remote sensing images) can be encoded through linear projection of flattened patches, position embedding, and Pruned Transformer Encoder structure, and the prediction results (logits) can be output through the detection network (Detection Network) and the RoI head (Region of Interest head). The two networks establish a distillation relationship through KL Divergence (Kullback-Leibler Divergence), and use distillation fine-tuning instead of linear least squares fine-tuning method, so that the student network has higher efficiency while maintaining good reasoning ability, achieving a balance between model compression and performance maintenance.

[0127] KL divergence is used as the loss function of knowledge distillation. KL divergence is used to quantify the difference between the output probability distribution of the teacher network and the student network. By minimizing the logits output by the two networks, and according to the KL divergence of the probability distribution after softmax processing, a distillation loss function is constructed

[0128]

[0129] where D KL (p||q) represents the KL divergence between the probability distribution p T (the first output probability vector) and q S (the second output probability vector) output by the two networks. The KL divergence D KL (p||q) is represented as follows:

[0130]

[0131] where p(i) represents the i-th prediction value (logit) output by the teacher network; q(i) represents the i-th prediction value (logit) output by the student network.

[0132] By pruning technology to reduce the redundancy of the student network, KL divergence loss guides knowledge transfer for fine-tuning, so that the student network can retain data understanding and processing ability while significantly reducing the number of parameters, providing an efficient solution for model deployment in resource-constrained scenarios. It has wide application potential in natural language processing, computer vision and other fields.

[0133] Specifically, the first output probability vector of the unpruned target detection model and the first output probability vector of the pruned target detection model can be determined by using the initial mask matrix. The output probability vector refers to the vector representation of the model's prediction of the possibility of the target class or bounding box for the input data. For example, for the unpruned model, the first output probability vector is obtained by forward propagating the input data using the complete weight parameters. For the pruned model, the input data is forward propagated, but since part of the weights are shielded by the 0 values in the mask matrix, the model actually uses the pruned structure, thereby obtaining the second output probability vector. Then, the divergence loss function can be constructed based on the first output probability vector and the second output probability vector. The divergence loss function is a function used to measure the difference between two probability distributions. For example, the KL divergence is used to quantify the difference between the output probability distributions of the two models, that is, by minimizing the KL divergence of the probability distributions obtained by processing the outputs of the two models through softmax (an activation function), a distillation loss function (i.e., divergence loss function) is constructed according to the KL divergence. Then, by adjusting the values of the non-0 masks in the initial mask matrix, the loss value determined by minimizing the divergence loss function, until the loss value is lower than the second preset threshold (which can be set according to the actual situation), for example, by using an optimization algorithm (such as gradient descent method), based on the divergence loss function, the non-0 mask values in the mask matrix are adjusted to obtain the target mask matrix.

[0134] In this embodiment, the decline in model output performance after pruning operation can be effectively compensated. The target mask matrix is accurately optimized, which can recover the output signal of the pruned model as much as possible without significantly affecting the calculation efficiency of the model, so that it is close to or equivalent to the output of the unpruned model. Not only improves the resource utilization rate of the model, but also reduces the performance loss caused by pruning, which is suitable for deploying target detection tasks in edge devices and resource-constrained environments.

[0135] Figure 3 is a schematic diagram of an optional multi-stage unstructured pruning process according to an embodiment of the present application, as Figure 3As shown, the initial mask can be first performed to obtain the initial mask matrix of the model, that is, all mask variables are set to 1, indicating that all attention heads (MHA) and feedforward network filters (FFN) in the model are retained and have not been pruned at this time. Then, under the set calculation resource constraint (such as FLOPs or delay), the mask search is performed to determine the attention heads and filters that need to be pruned, and the mask variables of the pruned attention heads and filters are updated to 0. After that, the mask rearrangement is performed to adjust the pruning position under the condition that the number of pruned layers remains unchanged, so as to optimize the model performance, that is, the number of mask variables of 0 in each layer remains unchanged, and the mask variable of 0 is adjusted. After that, the mask tuning is performed to restore the model output signal by fine-tuning the value of the non-zero mask variable without changing the determined pruning structure, that is, the specific value of the mask variable of 0 is adjusted.

[0136] In the embodiment of the application, in order to solve the problem that the current structured pruning (such as channel, layer, and head level) easily causes large information loss and insufficient flexibility, the unstructured pruning strategy is adopted, the pruning target can be freely selected at a finer parameter level (such as an element in an attention weight matrix), so that higher compression rate and more detailed structure control are realized, the pruning granularity is more detailed, and the compression is more flexible. In addition, in order to solve the problem that many current pruning methods cannot effectively control the FLOPs or inference delay of hardware resource constraints, the greedy pruning search strategy based on FLOPs / delay budget is designed, the pruning search under the resource constraint is supported, the pruning process can meet the deployment scene demand, and the availability of the model on the edge device is improved. In addition, in order to solve the problem that if important parameters are pruned in the current pruning method, the retraining needs to be performed at a high cost to restore the model performance, the greedy mask exchange and rearrangement optimization mechanism is proposed, the pruning error can be alleviated, the error decision in the pruning process can be repaired without increasing additional calculation, and thus the performance degradation is reduced.

[0137] The following will be described in detail in combination with another embodiment.

[0138] Embodiment two

[0139] The pruning device for the target detection model provided in the embodiment includes a plurality of implementation units, and each implementation unit corresponds to each implementation step in the above-mentioned embodiment one.

[0140] Figure 4 is a schematic diagram of an optional pruning device for a target detection model according to an embodiment of the application, as Figure 4 shown, the pruning device can include an initialization unit 40, a first determination unit 41, a second determination unit 42, a first pruning unit 43, and a second pruning unit 44.

[0141] The initialization unit 40 is configured to initialize the mask of each module structure to obtain an initialized mask of each module structure.

[0142] The first determination unit 41 is configured to determine an importance score of each module structure based on the weight of each module structure and the initialized mask, and prune the attention head with the minimum importance score and the filter with the minimum importance score to obtain a remaining attention head set and a remaining filter set.

[0143] The second determination unit 42 is configured to determine a target pruning combination under the operation amount condition constraint based on an attention head operation amount of each attention head in the remaining attention head set and a filter operation amount of each filter in the remaining filter set, wherein the target pruning combination is a pruning combination with the minimum total importance score, and each pruning combination includes an attention head position of a preset number of attention heads and a filter position of a preset number of filters.

[0144] The first pruning unit 43 is configured to prune the attention head indicated by each attention head position in the target pruning combination and the filter indicated by each filter position to obtain an initial pruned target detection model, and determine the attention head and the filter to be pruned in each layer of the network in the initial pruned target detection model under the delay condition constraint.

[0145] The second pruning unit 44 is configured to prune the attention head and the filter to be pruned in each layer of the network in the initial pruned target detection model to obtain a pruned target detection model.

[0146] The pruning device can prune the attention head and the filter with the minimum importance score in the target detection model, find the target pruning combination (i.e., the pruning combination with the minimum total importance score under the FLOPs constraint) under the operation amount condition constraint, balance the model compression and performance maintenance, determine the attention head and the filter to be pruned in each layer of the network in the model under the delay condition constraint, and achieve the purpose of simplifying the model structure under the premise of ensuring the detection accuracy of the model, so that the pruned model can run efficiently in a resource-limited environment while maintaining high detection accuracy.

[0147] Optionally, the first determining unit comprises: a first determining module, configured to determine the information loss of the module structure based on the weight of the module structure and the initialization mask; a first partial derivative module, configured to perform partial derivation on the information loss to obtain a partial derivative value, and determine the information matrix of the module structure based on the partial derivative value and a structure position of the module structure, wherein the structure position is determined based on a network layer number in the target detection model where the module structure is located and a position number of the network layer; and a first processing module, configured to perform diagonal approximation processing on the information matrix to obtain the importance score of the module structure.

[0148] Optionally, the second determining unit comprises: a second determining module, configured to determine the target operation amount, wherein the target operation amount is used to determine the operation amount condition constraint; a third determining module, configured to determine a plurality of quantity pairs based on the target operation amount, the attention head operation amount of each attention head in the remaining attention head set, and the filter operation amount of each filter in the remaining filter set, wherein each quantity pair contains a preset attention head quantity of the attention head to be pruned and a preset filter quantity of the filter; a fourth determining module, configured to determine all pruning combinations satisfying the quantity pair for each quantity pair, wherein all attention head positions and all preset filter positions in each pruning combination are not completely the same; a fifth determining module, configured to determine the total importance score of each pruning combination; and a sixth determining module, configured to determine the pruning combination with the smallest total importance score as the target pruning combination.

[0149] Optionally, the third determining module comprises: a first determining submodule, configured to determine the attention head quantity of the attention head in the remaining attention head set, and determine the filter quantity of the filter in the remaining filter set; a first traversal submodule, configured to traverse the attention head quantity, and take each traversed quantity as a reserved attention head quantity, wherein the traversal starts from 0 and ends at the attention head quantity, and each traversal increases by one quantity; a second determining submodule, configured to determine, for each reserved attention head quantity, a reserved filter quantity corresponding to the reserved attention head quantity based on the target operation amount, the attention head average operation amount, and the filter average operation amount, wherein the attention head average operation amount is calculated based on the attention head operation amount of each attention head in the remaining attention head set, and the filter average operation amount is calculated based on the filter operation amount of each filter in the remaining filter set; and a third determining submodule, configured to determine each preset attention head quantity based on the attention head quantity and each reserved attention head quantity, and determine a preset filter quantity corresponding to each preset attention head quantity based on the filter quantity and the reserved filter quantity corresponding to each reserved attention head quantity, to obtain the plurality of quantity pairs, wherein each quantity pair contains a preset attention head quantity and a preset filter quantity corresponding to the preset attention head quantity, and the preset attention head quantities contained in each quantity pair are different.

[0150] Optionally, the first pruning unit comprises: a seventh determination module configured to determine, based on a preset fixed overhead, a number of attention heads and a number of filters reserved in each network layer of the target detection model, wherein the preset fixed overhead is used to determine a delay condition constraint; an eighth determination module configured to determine, for each network layer, a plurality of structure position combinations to be pruned based on the number of attention heads and the number of filters reserved in each network layer; a ninth determination module configured to determine, for each structure position combination, a delay value of each network layer after pruning all module structures indicated by the structure position combination; a tenth determination module configured to determine a structure position combination indicated by a minimum delay value as a target structure position combination of each network layer; and an eleventh determination module configured to determine all attention heads indicated by the target structure position combination as the attention heads to be pruned and all filters indicated by the target structure position combination as the filters to be pruned.

[0151] Optionally, the second pruning unit comprises: a twelfth determination module configured to determine a plurality of initial structure positions from a plurality of structure positions indicated by the attention heads and the filters to be pruned in each network layer, and determine a number of the attention heads to be pruned and a number of the filters to be pruned in each network layer; a thirteenth determination module configured to determine, based on the number of the attention heads to be pruned and the number of the filters to be pruned in each network layer, a set of structure positions to be pruned in each network layer, wherein the set of structure positions to be pruned comprises all structure position combinations except an initial structure position combination, and the initial structure position combination is composed of the plurality of initial structure positions; a fourteenth determination module configured to determine an initial importance layer score of all module structures indicated by the initial structure position combination, and determine a to-be-pruned importance layer score of all module structures indicated by each structure position combination; a fifteenth determination module configured to select a minimum to-be-pruned importance layer score from all to-be-pruned importance layer scores, and determine, in a case where the minimum to-be-pruned importance layer score is smaller than the initial importance layer score, all module structures indicated by a structure position combination corresponding to the minimum to-be-pruned importance layer score as the attention heads and the filters to be pruned in each network layer; and a first pruning module configured to prune the attention heads and the filters to be pruned in each network layer to obtain the target detection model after pruning is completed.

[0152] Optionally, the pruning device further comprises: a sixteenth determination module configured to determine a mask matrix corresponding to the target detection model after pruning is completed, after the target detection model after pruning is completed is obtained; a seventeenth determination module configured to determine, based on the mask matrix, a first activation matrix of the target detection model before pruning and a second activation matrix of the target detection model after pruning is completed; and a first adjustment module configured to adjust a value of a non-0 mask in the mask matrix until a matrix value of a difference matrix is smaller than a first preset threshold value to obtain an initial mask matrix, wherein the difference matrix is determined based on the first activation matrix and the second activation matrix.

[0153] Optionally, the pruning device further comprises: an eighteenth determination module, configured to, after obtaining the initial mask matrix, determine a first output probability vector of the unpruned target detection model and a second output probability vector of the pruned target detection model based on the initial mask matrix; a first construction module, configured to construct a divergence loss function based on the first output probability vector and the second output probability vector; and a second adjustment module, configured to adjust the value of the non-0 mask in the initial mask matrix until the loss value determined by the divergence loss function is less than a second preset threshold value, and obtain the target mask matrix.

[0154] The pruning device described above can further comprise a processor and a memory, and the initialization unit 40, the first determination unit 41, the second determination unit 42, the first pruning unit 43, the second pruning unit 44, and the like are stored in the memory as program units, and the corresponding functions are realized by the processor executing the program units stored in the memory.

[0155] The processor described above comprises a core, and the core retrieves the corresponding program units from the memory. The core can be set to one or more, and the core parameters are adjusted to prune the attention heads and filters to be pruned in each layer of the initial pruned target detection model, and obtain the pruned target detection model.

[0156] The memory described above can include a non-permanent memory in a computer readable medium, a random access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory includes at least one memory chip.

[0157] The application further provides a computer program product adapted to execute the program of the following method steps when executed on a data processing device: initializing a mask for each module structure to obtain an initial mask for each module structure, determining an importance score for each module structure based on the weight of each module structure and the initial mask, and pruning the attention head with the smallest importance score and the filter with the smallest importance score to obtain a remaining attention head set and a remaining filter set, determining a target pruning combination under the operation amount condition constraint based on the attention head operation amount of each attention head in the remaining attention head set and the filter operation amount of each filter in the remaining filter set, pruning the attention head indicated by each attention head position and the filter indicated by each filter position in the target pruning combination to obtain an initial pruned target detection model, and determining the attention heads and filters to be pruned in each layer of the initial pruned target detection model under the delay condition constraint, pruning the attention heads and filters to be pruned in each layer of the initial pruned target detection model to obtain a pruned target detection model.

[0158] According to another aspect of the embodiments of the present application, there is also provided a computer program product comprising a non-transitory computer readable storage medium storing a computer program which, when executed by a processor, implements the pruning method of the target detection model of any of the above.

[0159] According to another aspect of the embodiments of the present application, there is also provided an electronic device comprising one or more processors and a memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the pruning method of the target detection model of the above.

[0160] Figure 5 is a hardware structure block diagram of an electronic device (or a mobile device) for a pruning method of a target detection model according to an embodiment of the present application. As shown in Figure 5 , the electronic device can include one or more processors (for example, processors 502a, 502b, …, 502n in Figure 5 , which can include but are not limited to processing devices such as microprocessors MCU or programmable logic devices FPGA, etc.), a memory 504 for storing data. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. Those skilled in the art can understand that Figure 5 , the structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the electronic device can also include more or less components than Figure 5 , or have a different configuration from Figure 5 .

[0161] The above-mentioned serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0162] The embodiments or examples of the present disclosure are not exhaustive, and are only a part of the embodiments or examples, and are not specific limitations on the protection scope of the present disclosure. Each step in a certain embodiment or example can be implemented as an independent example, and the steps can be combined arbitrarily, for example, a scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be exchanged arbitrarily, in addition, the optional ways or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the embodiments or examples can be combined arbitrarily, for example, the steps of different embodiments or examples can be combined arbitrarily, a certain embodiment or example can be combined with the optional ways or optional examples of other embodiments or examples.

[0163] In the above-described embodiments of the present disclosure, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0164] In the several embodiments of the present disclosure, it should be understood that the disclosed technology can be implemented in other ways. Of course, the embodiment described above is only a schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0165] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0166] In addition, each functional unit in each embodiment of the present disclosure can be integrated in a processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0167] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0168] The above is only the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method for pruning a target detection model, characterized in that, The target detection model comprises: a multi-layer network, each layer of the network comprising: a plurality of module structures, the module structures being one of attention heads or filters, comprising: initializing a mask of each of the module structures to obtain an initialized mask of each of the module structures; determining an importance score of each of the module structures based on a weight of each of the module structures and the initialized mask, and pruning the attention head with the minimum importance score and the filter with the minimum importance score to obtain a remaining attention head set and a remaining filter set; determining a target pruning combination under an operation amount condition constraint based on an attention head operation amount of each of the attention heads in the remaining attention head set and a filter operation amount of each of the filters in the remaining filter set, wherein the target pruning combination is a pruning combination with the minimum total importance score, and each of the pruning combinations comprises: an attention head position of a preset number of attention heads and a filter position of a preset number of filters; pruning the attention head indicated by each of the attention head positions in the target pruning combination and the filter indicated by each of the filter positions to obtain an initial pruned target detection model, and determining the attention head and the filter to be pruned in each layer of the network in the initial pruned target detection model under a delay condition constraint; pruning the attention head and the filter to be pruned in each layer of the network in the initial pruned target detection model to obtain the target detection model after pruning is completed.

2. The pruning method of claim 1, wherein, The step of determining the importance score of each of the module structures based on the weight of each of the module structures and the initialized mask comprises: determining an information loss of the module structure based on the weight of the module structure and the initialized mask; obtaining a partial derivative value by performing partial derivation on the information loss, and determining an information matrix of the module structure based on the partial derivative value and a structure position of the module structure, wherein the structure position is determined based on a number of network layers in the target detection model where the module structure is located and a position number of the network layer where the module structure is located; performing diagonal approximation processing on the information matrix to obtain the importance score of the module structure.

3. The pruning method of claim 1, wherein, The step of determining a target pruning combination under an operation amount condition constraint based on an attention head operation amount of each of the attention heads in the remaining attention head set and a filter operation amount of each of the filters in the remaining filter set comprises: determining a target operation amount, wherein the target operation amount is used to determine the operation amount condition constraint; determining a plurality of quantity pairs based on the target operation amount, the attention head operation amount of each of the attention heads in the remaining attention head set and the filter operation amount of each of the filters in the remaining filter set, wherein each quantity pair contains the preset number of attention heads of the attention heads to be pruned and the preset number of filters of the filters; For each of the quantity pairs, determine all the pruning combinations satisfying the quantity pairs, wherein all the attention head positions and all the preset filter positions in each of the pruning combinations are not completely identical; Determine the total importance score of each of the pruning combinations; Determine the pruning combination with the minimum total importance score as the target pruning combination.

4. The pruning method of claim 3, wherein, Based on the target operation amount, the attention head operation amount of each of the remaining attention heads in the remaining attention head set, and the filter operation amount of each of the filters in the remaining filter set, the step of determining a plurality of quantity pairs comprises: Determine the number of attention heads in the remaining attention head set and the number of filters in the remaining filter set; Iterate through the number of attention heads, and take each iteration as a reserved attention head number, wherein the iteration starts from 0 and ends at the number of attention heads, and each iteration increases by one number; For each of the reserved attention head numbers, determine the corresponding reserved filter number based on the target operation amount, the average attention head operation amount, and the average filter operation amount, wherein the average attention head operation amount is calculated based on the attention head operation amount of each of the remaining attention heads in the remaining attention head set; and the average filter operation amount is calculated based on the filter operation amount of each of the filters in the remaining filter set; Based on the number of attention heads and each of the reserved attention head numbers, determine each of the preset attention head numbers, and based on the number of filters and the reserved filter number corresponding to each of the reserved attention head numbers, determine the preset filter number corresponding to each of the preset attention head numbers, to obtain a plurality of quantity pairs, wherein the quantity pair contains the preset attention head number and the preset filter number corresponding to the preset attention head number, and the preset attention head numbers contained in each of the quantity pairs are different.

5. The pruning method of claim 1, wherein, Under the delay condition constraint, the step of determining the attention heads and the filters to be pruned in each layer of the initial pruned target detection model comprises: Based on a preset fixed overhead, determine the number of attention heads and the number of filters to be retained in each layer of the target detection model, wherein the preset fixed overhead is used to determine the delay condition constraint; For each layer of network, based on the number of attention heads and the number of filters to be retained in each layer of network, determine a plurality of structure position combinations to be pruned; For each of the structure position combinations, determine the delay value of each layer of network after pruning all the module structures indicated by the structure position combination; Determine the structure position combination indicated by the minimum delay value as the target structure position combination of each layer of network; Determine all the attention heads indicated by the target structure position combination as the attention heads to be pruned, and determine all the filters indicated by the target structure position combination as the filters to be pruned.

6. The pruning method of claim 1, wherein, The step of pruning the attention heads and the filters to be pruned in each network in the initial pruned target detection model to obtain the pruned target detection model comprises: determine a plurality of initial structure positions as the structure positions indicated by the attention heads and the filters to be pruned in each network, and determine the number of the attention heads and the number of the filters to be pruned in each network; determine a set of structure positions to be pruned in each network based on the number of the attention heads and the number of the filters to be pruned in each network, wherein the set of structure positions to be pruned comprises all structure position combinations except an initial structure position combination, and the initial structure position combination is composed of a plurality of initial structure positions; determine initial importance layer scores of all module structures indicated by the initial structure position combination, and determine importance layer scores to be pruned of all module structures indicated by each structure position combination; select a minimum importance layer score to be pruned from all the importance layer scores to be pruned, and determine all module structures indicated by the structure position combination corresponding to the minimum importance layer score to be pruned as the attention heads and the filters to be pruned in each network if the minimum importance layer score to be pruned is less than the initial importance layer score; prune the attention heads and the filters to be pruned in each network to obtain the pruned target detection model.

7. The pruning method of claim 1, wherein, After obtaining the pruned target detection model, further comprising: determine a mask matrix corresponding to the pruned target detection model; determine a first activation matrix of the unpruned target detection model and a second activation matrix of the pruned target detection model based on the mask matrix; adjust the value of the non-0 mask in the mask matrix until the matrix value of the difference matrix is less than a first preset threshold to obtain an initial mask matrix, wherein the difference matrix is determined based on the first activation matrix and the second activation matrix.

8. The pruning method of claim 7, wherein, After obtaining the initial mask matrix, further comprising: determine a first output probability vector of the unpruned target detection model and a second output probability vector of the pruned target detection model based on the initial mask matrix; construct a divergence loss function based on the first output probability vector and the second output probability vector; adjust the value of the non-0 mask in the initial mask matrix until the loss value determined by the divergence loss function is less than a second preset threshold to obtain a target mask matrix.

9. An apparatus for pruning a target detection model, comprising: The target detection model comprises a plurality of networks, each network comprising a plurality of module structures, the module structures being one of an attention head or a filter, comprising: an initialization unit configured to initialize a mask of each module structure to obtain an initial mask of each module structure; The first determining unit is configured to determine an importance score of each of the module structures based on a weight of each of the module structures and the initialization mask, and to prune the attention head with the minimum importance score and the filter with the minimum importance score to obtain a remaining attention head set and a remaining filter set; The second determining unit is configured to determine a target pruning combination under operation amount condition constraints based on an attention head operation amount of each of the attention heads in the remaining attention head set and a filter operation amount of each of the filters in the remaining filter set, wherein the target pruning combination is a pruning combination with the minimum total importance score, and each of the pruning combinations includes attention head positions of a preset attention head quantity and filter positions of a preset filter quantity; The first pruning unit is configured to prune the attention head indicated by each of the attention head positions and the filter indicated by each of the filter positions in the target pruning combination to obtain an initial pruned target detection model, and to determine the attention head and the filter to be pruned in each layer of network in the initial pruned target detection model under delay condition constraints. The second pruning unit is configured to prune the attention head and the filter to be pruned in each layer of network in the initial pruned target detection model to obtain a pruned target detection model.

10. An electronic device, comprising: The device comprises one or more processors and a memory, and the memory is configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the pruning method of the target detection model in any one of claims 1 to 8.

Images