Structure packet disassociation pruning method and infrared small target lightweight detection method

Abstract

The application discloses a structure grouping disassociation pruning method and an infrared small target lightweight detection method, which are applied to the field of model compression, a shielding layer is determined from a model layer, and a to-be-pruned layer set is constructed based on the model layer except the shielding layer; neurons in the to-be-pruned layer set are divided into a plurality of neuron substructures based on a neuron coupling relationship; neuron substructures with similar structures are divided into the same neuron substructure set; a to-be-pruned neuron substructure is determined in each neuron substructure set, the to-be-pruned neuron substructure is pruned, an infrared small target detection model after pruning is obtained, and the infrared small target is detected through the model after pruning. The method of the application protects key structures of the model by setting the shielding layer, identifies and forms substructures with strong correlation based on the coupling dependence relationship of the neurons, forms groups according to structural similarity, effectively maintains the integrity of the cross-layer feature fusion path, and avoids performance degradation of the model after pruning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of model compression, in particular to a structure grouping de-association pruning method and an infrared small target lightweight detection method. BACKGROUND

[0002] With the rapid iterative development of deep learning, the infrared small target detection model tends to be complex and redundant in structure, and the parameter quantity and the calculation quantity show an increasing trend. The structured pruning method has become the mainstream paradigm for realizing lightweight network due to its advantage of realizing model inference acceleration on general hardware. The existing model pruning methods are mostly designed for natural image tasks, and are not optimized for the characteristics of small target size, deep target response and background interference of the infrared small target detection task. Direct application of existing pruning methods will cause problems such as misdeletion of key structures or decline of model performance. Moreover, the existing pruning methods rely on expert-driven model structure division, and the process is complex and redundant and difficult to adapt to the parameter sharing mechanism in cross-layer connection. SUMMARY

[0003] The purpose of the present application is to provide a structure grouping de-association pruning method and an infrared small target lightweight detection method, which is applied to the field of model compression. The method protects the key structure of the model by setting a shielding layer, and identifies and forms sub-structures with strong association based on the coupling dependence relationship of neurons. According to the structural similarity, groups are formed, effectively maintaining the integrity of the cross-layer feature fusion path and avoiding the decline of the model performance after pruning.

[0004] To solve the above technical problems, the present application provides a structure grouping de-association pruning method, comprising:

[0005] determining each model layer of an infrared small target detection model, determining a shielding layer from the model layer, and constructing a set of layers to be pruned based on the model layer except the shielding layer;

[0006] dividing the neurons in the set of layers to be pruned into a plurality of neuron sub-structures based on the coupling relationship of neurons;

[0007] performing structural similarity verification between the neuron sub-structures, and dividing the neuron sub-structures with structural similarity into the same neuron sub-structure set;

[0008] determining a neuron sub-structure to be pruned in each neuron sub-structure set, pruning the neuron sub-structure to be pruned based on a preset pruning method, and obtaining a pruned infrared small target detection model.

[0009] Optionally, dividing the neurons in the set of layers to be pruned into a plurality of neuron sub-structures based on the coupling relationship of neurons comprises:

[0010] determining a target neuron from neurons in the set of layers to be pruned that are not divided into the neuron substructures;

[0011] determining an associated neuron having a coupling relationship with the target neuron from the neurons in the set of layers to be pruned that are not divided into the neuron substructures;

[0012] constructing the neuron substructure based on the target neuron and the associated neuron;

[0013] starting the step of determining a target neuron from the neurons in the set of layers to be pruned that are not divided into the neuron substructure again until all neurons in the set of layers to be pruned are included in the neuron substructure.

[0014] Optionally, structure similarity verification is performed between the neuron substructures, and neuron substructures that are structurally similar are divided into the same neuron substructure set, including:

[0015] neuron number consistency verification, parameter space consistency verification, neuron dimension consistency verification, and index range consistency verification are performed between the neuron substructures;

[0016] neuron substructures that pass the neuron number consistency verification, the parameter space consistency verification, the neuron dimension consistency verification, and the index range consistency verification are divided into the same neuron substructure set;

[0017] if there is a neuron substructure that does not pass the neuron number consistency verification, the parameter space consistency verification, the neuron dimension consistency verification, and the index range consistency verification with any other neuron substructure, the neuron substructure is divided into a neuron substructure set.

[0018] Optionally, the parameter space consistency verification includes divergence verification and cosine similarity verification.

[0019] Optionally, a neuron substructure to be pruned is determined in each neuron substructure set, including:

[0020] a neuron substructure importance value of each neuron substructure in the neuron substructure set is determined, and importance calculation and sorting of the neuron substructures in the neuron substructure set are performed based on the neuron substructure importance value;

[0021] a preset pruning ratio is determined, and the neuron substructures in the neuron substructure set that have been subjected to importance calculation and sorting are divided into a pre-retained neuron substructure and the neuron substructure to be pruned based on the preset pruning ratio.

[0022] Optionally, the neuron substructure importance value of each neuron substructure in the set of neuron substructures is determined, comprising:

[0023] The L2 norm of the neuron weight in each neuron substructure in the set of neuron substructures is determined, and the neuron importance value is determined based on the L2 norm of the neuron weight;

[0024] The neuron normalization scaling factor is determined based on the neuron importance value;

[0025] The product of the neuron importance value and the neuron normalization scaling factor of each neuron in the neuron substructure is summed to obtain the neuron substructure importance value.

[0026] Optionally, the pruning of the to-be-pruned neuron substructure is performed based on a preset pruning method to obtain a pruned infrared small target detection model, comprising:

[0027] The number of iterations of regular pruning is determined, the weight penalty term in each iteration is determined, and the bias penalty term in each iteration is determined;

[0028] The weight term and the bias term of each neuron in the to-be-pruned neuron substructure are updated in each iteration based on the weight penalty term and the bias penalty term to obtain the pruned infrared small target detection model.

[0029] Optionally, the weight penalty term in each iteration is determined, comprising:

[0030] An attenuation coefficient and a penalty coefficient are determined, and the square of the two-norm of each neuron weight in the to-be-pruned neuron substructure is determined;

[0031] The first derivative between the square of the two-norm of the neuron weight and the neuron weight is determined;

[0032] The product of the first derivative, the attenuation coefficient, the penalty coefficient, and the neuron weight is determined as the weight penalty term of each neuron in the to-be-pruned neuron substructure.

[0033] Optionally, the method further comprises:

[0034] An infrared small target detection training data set is obtained, and the pruned infrared small target detection model is parameter fine-tuned based on the training data set.

[0035] To solve the above technical problems, the present application provides an infrared small target lightweight detection method, comprising:

[0036] Obtain a to-be-detected image, input the to-be-detected image into an infrared small target detection model, and obtain an infrared small target detection result output by the model.

[0037] The infrared small target detection model is pruned according to the structure grouping deassociation pruning method.

[0038] It can be seen that the method provided in the present application determines each model layer of the infrared small target detection model, determines a shielding layer from the model layer, constructs a to-be-pruned layer set based on the model layers other than the shielding layer, divides neurons in the to-be-pruned layer set into a plurality of neuron substructures based on the coupling relationship of the neurons, performs structure similarity verification between the neuron substructures, divides neuron substructures with similar structures into the same neuron substructure set, determines a to-be-pruned neuron substructure in each neuron substructure set, prunes the to-be-pruned neuron substructure based on a preset pruning method, and obtains the infrared small target detection model after pruning is completed. The method provided in the present application protects the key structure of the model by setting the shielding layer, identifies and forms a substructure with strong correlation based on the coupling dependence relationship of the neurons, forms groups according to the structure similarity, effectively maintains the integrity of the cross-layer feature fusion path, and avoids performance degradation of the model after pruning. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0040] Figure 1 A flowchart of a structure grouping deassociation pruning method provided by the embodiment of the present application;

[0041] Figure 2 An example diagram of a shielding layer provided by the embodiment of the present application;

[0042] Figure 3 An example diagram of a neuron substructure provided by the embodiment of the present application;

[0043] Figure 4 An example diagram of a neuron substructure grouping provided by the embodiment of the present application. DETAILED DESCRIPTION

[0044] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0045] With the rapid iterative development of deep learning, infrared small target detection models tend to be complex and redundant in structure, and the parameter quantity and the amount of calculation show an increasing trend. The structured pruning method has the advantage of realizing model inference acceleration on general hardware, and has become the mainstream paradigm for realizing lightweight networks. Existing pruning methods are mostly designed for natural image tasks, and are not optimized for the characteristics of small target size, deep target response and background interference of the infrared small target detection task. Direct application of existing pruning methods will cause problems such as misdeletion of key structures or decline in model performance. The original intention of the present application is to realize intelligent infrared small target detection with lightweight and high precision performance through structured pruning.

[0046] The existing pruning method cannot effectively solve the demand of complex detection network with lightweight and high performance due to its own design characteristics, and has the following disadvantages:

[0047] (1) Sensitivity missing problem of mechanism module. Complex detection models usually introduce attention mechanism modules to enhance the ability of small target feature representation. However, the existing pruning method ignores the difference between the conventional convolution layer and the mechanism module, which will weaken the detection performance of the model after pruning.

[0048] (2) Coupling dependence processing problem of complex structure. The existing pruning method relies on expert-driven structure division strategy, and the process is complex and redundant, and it is difficult to adapt to the parameter sharing mechanism in cross-layer connection.

[0049] (3) Weight correlation and bias optimization problem. The existing pruning method focuses on network neuron removal, easily ignores the internal high collinearity processing of weights, and does not make redundant optimization to the bias term.

[0050] The existing structured pruning method has problems such as loss of small target information in the pruning process, mis-pruning of important modules, insufficient coupling and dependency processing, and lack of weight correlation and bias optimization when processing an infrared small target detection model, which limits the performance of the pruned model. The present application first protects the key modules of the network through a shielding layer to avoid mis-pruning of key structures and ensure stable extraction of small target features. Secondly, a hierarchical coupling structure grouping strategy is adopted to divide groups according to the hierarchical coupling dependency relationship, effectively maintaining the integrity of the cross-layer feature fusion path. At the same time, the de-correlated L2 norm regularization pruning technology is combined to jointly optimize the filter weight and bias term, constrain the activation strength of redundant channels, and reduce the correlation between filters. Comprehensive channel, layer and module level processing is implemented to achieve a precise balance between lightweight and high detection performance.

[0051] The following will be described in combination with Figure 1 , Figure 1 The flowchart of the structural grouping de-correlation pruning method provided by the embodiment of the present application can include:

[0052] S101: Determine each model layer of the infrared small target detection model, determine the shielding layer from the model layer, and construct a set of layers to be pruned based on the model layers other than the shielding layer.

[0053] The embodiment can first determine each model layer of the infrared small target detection model, and determine the shielding layer from the model layer. The present embodiment does not limit the specific type of the infrared small target detection model, and existing models can be pruned.

[0054] The infrared small target detection model is designed to add modules important to the performance of the model, such as small target feature extraction, feature enhancement and feature fusion. By means of the electrostatic shielding phenomenon in physics, the concept of shielding layer is introduced. According to the structural design characteristics of the model itself, important modules are added to the shielding layer, and the pruning operation is no longer performed on the internal modules of the shielding layer. Only the modules in the shielding layer need to be matched with other layers for adaptive channel matching, so as to maintain the overall performance of the pruned model to the greatest extent.

[0055] The present embodiment does not limit the specific way of determining the shielding layer from each model layer. Generally, the importance value of each model layer on the performance of the model can be determined, and the shielding layer can be selected based on the importance value to avoid weakening the detection performance of the model after pruning. The model can also add a special designed hierarchical layer to the shielding layer. Complex detection models usually introduce attention mechanism modules to enhance the small target feature representation capability, such as Figure 2 As shown in the figure, in an example, the attention mechanism layer containing channel attention and spatial attention in the infrared small target detection model can be set as the shielding layer.

[0056] Specifically, assuming that the set of all model layers of the infrared small target detection model can be M={m1, m2, …, mN}, the shielding layer can be determined according to the importance value of each model layer on the performance of the model.s}, wherein m i is a model layer, and the subscript of the model layer represents the number of the model layer, and s represents the number of the model layers.

[0057] The embodiment can determine a shielding layer from the model layers and add the shielding layer to a shielding layer set SL, and the model layers actually participating in pruning can be M-SL=N={m1, m2, …, mj}. j}, wherein N represents a layer set to be pruned, and m i is a model layer, and the number of the model layers in the layer set to be pruned is j.

[0058] By constructing the shielding layer, the key functional modules in the infrared small target detection network can be protected, and the degradation of feature extraction capability caused by false pruning can be avoided. The model layers in the shielding layer are separated from the pruning process, only the input and output channels are adjusted and matched to ensure data flow consistency, and do not participate in subsequent coupling structure grouping and regular constraint pruning.

[0059] S102: Dividing neurons in the layer set to be pruned into multiple neuron substructures based on neuron coupling relationships.

[0060] There are often many coupling dependent structures in a complex detection model. In order to effectively group, the intuitive solution is to remove channels according to expert knowledge driven, and the channel level related to more coupling dependencies can be effectively excluded. However, various complex architecture designs and mechanisms make manual grouping design face great challenges. We start from the level coupling relationship of the model, focus on identifying the dependencies between input and output, and the dependencies between layers, so that neurons with coupling form substructures, and then according to the substructure similarity idea, construct groups according to certain rules, and prune the module channels in the substructure according to the selected pruning evaluation criteria.

[0061] The embodiment can divide the neurons in the layer set to be pruned into multiple neuron substructures based on neuron coupling relationships. Specifically, a target neuron can be determined from the neurons in the layer set to be pruned which are not divided into the neuron substructures; further, an associated neuron having a coupling relationship with the target neuron can be determined from the neurons in the layer set to be pruned which are not divided into the neuron substructures; and further, a neuron substructure can be constructed based on the target neuron and the associated neuron.

[0062] The step of determining the target neuron from the neurons in the layer set to be pruned which are not divided into the neuron substructures is executed again until the neuron substructure contains all the neurons in the layer set to be pruned.

[0063] In the face of hierarchical coupling dependencies in neural networks, neuronal substructures are built based on coupling. Starting with an independent neuron, other structural nodes coupled to that independent neuron are activated, initiating a self-dependent process until the coupling relationship can no longer be continued. At this point, a complete neuronal substructure is formed. Neurons that have participated in the construction of a neuronal substructure cannot serve as nodes in other neuronal substructures.

[0064] For example, for neurons {n1,n2,…,n} within the model j Based on the coupling relationship, it can be divided into multiple neuron substructures {s1,s2,…,s}. n}, where n i Here, s represents neurons, j represents the number of neurons, and s represents the number of neurons. i Let be a neuron substructure, and n be the number of neuron substructures. Then, a certain substructure can be represented as s. i ={n1,n2,…,n i}

[0065] In this embodiment, to facilitate the determination of the coupling relationship between neurons, the input-output hierarchy within the model can be analyzed, N={m1,m2,…,m j The internal focus is on the input and output of each layer, with the input of each model layer set to m. i - The output of each model layer is m i +, then the set of layers to be pruned can be represented as {m1-,m1+,m2-,m2+,…,m j -,m j +}.

[0066] The coupling dependency rule is not only related to the forward computation process, but also to the directional relationship between the two. Removing nodes in subsequent layers will lead to the removal of related nodes in previous layers, ensuring the connectivity of the overall structure.

[0067] For handling inter-layer dependency connections, if there are direct connections between input and output, such as sequential connections between different convolutional layers and skip connections, the connection can be represented by D(m). i +,m j -)=1 indicates that there is coupling dependency between layers.

[0068] For intra-layer dependency handling, special layers have the same data flow and processing logic for their input and output. For example, a batch normalization layer's dependency connection can be represented as D(m i -,m i +)=1, and the same pruning process can be used afterwards.

[0069] In this embodiment, an example of the divided neuronal substructure can be as follows: Figure 3As shown, the neurons are divided into multiple neuron substructures based on the hierarchical coupling relationship, such as s a , s b , s c , and the like.

[0070] S103: Structure similarity verification is performed between the neuron substructures, and neuron substructures with similar structures are divided into the same neuron substructure set.

[0071] The present embodiment can perform structure similarity verification between the neuron substructures, and neuron substructures with similar structures are divided into the same neuron substructure set. For example, neuron substructures with similar structures and capable of matching are divided into multiple neuron substructure sets {g1, g2, …, g q}, where a single neuron substructure set can be represented as g i ={s1, s2, …, s w}, where q is the number of neuron substructure sets, and w is the number of neuron substructures in a single neuron substructure set. The present embodiment allows the comparability of substructures within a group, and to a certain extent, avoids removing layer blocks with relatively small importance in evaluation but large impact on performance in the pruning process, while maintaining the overall structure of the model as much as possible.

[0072] The present embodiment does not limit the specific way of performing structure similarity verification between the neuron substructures, and generally can perform neuron quantity consistency verification, parameter space consistency verification, neuron dimension consistency verification, and index range consistency verification between the neuron substructures.

[0073] Neuron substructures that pass the neuron quantity consistency verification, parameter space consistency verification, neuron dimension consistency verification, and index range consistency verification are divided into the same neuron substructure set.

[0074] If there is a single neuron substructure that does not pass the neuron quantity consistency verification, parameter space consistency verification, neuron dimension consistency verification, and index range consistency verification with any other neuron substructure, the single neuron substructure is divided into a neuron substructure set.

[0075] Specifically, for any neuron substructure s i , the parameter matrix can be represented as s i (W), the dimension can be represented as s i (d), the neuron index position can be represented as s i (k), and the number of neurons can be represented as s i (num).

[0076] Using the concept of structural equivalence in mathematics, we can describe the relationships between the attributes of objects. For example, sets P={1,2,3} and Q={4,5,6} are equivalent in algebraic structure, although their names differ; they are similar in structure and properties. The structural similarity criterion for any two substructures can be determined based on the structural similarity of their substructures, and can be expressed by the following formula:

[0077] s i (num)=s j (num); Formula for verifying consistency in the number of neurons;

[0078] s i (k),s j (k)∈R 1×n R 1×n Given a sequence of real numbers from 1 to n; the formula for verifying the consistency of the index range.

[0079] s i (d) s j (d); Neuron dimensional consistency verification formula;

[0080] s i (W) s j (W); Formula for verifying consistency of parameter space.

[0081] Furthermore, this embodiment does not limit the specific method of performing parameter space consistency verification. Generally, parameter space consistency verification may include divergence verification and cosine similarity verification.

[0082] The similarity measure of the parameter matrix of neuronal substructures can be determined by using the divergence criterion to capture the microscopic morphological features of the parameter distribution and the cosine similarity criterion to characterize the macroscopic directional structure of the parameter space.

[0083] The divergence criterion first reconstructs the parameter matrix into a discrete probability distribution:

[0084] p(θ) = softmax(W vec (θ));

[0085] In the formula, softmax is the normalized exponential function, θ is the input variable, p(θ) is the probability distribution, and W vec (θ) is the parameter matrix.

[0086] Further calculation of divergence is used to determine the statistical similarity of the parameter distribution states:

[0087] ;

[0088] In the formula, D kl (p(θ i )||p(θj )) is the KL divergence between the probability distribution p(θ i ) of the neuron substructure s i and the probability distribution p(θ j ) of the neuron substructure s j , D kl (p(θ j ||p(θ i )) is the KL divergence between the probability distribution p(θ j ) of the neuron substructure s j and the probability distribution p(θ i ) of the neuron substructure s i , θ i is the input variable of the neuron substructure s i , θ j is the input variable of the neuron substructure s j , the KL divergence can measure the difference between the true probability distribution and the contrast probability distribution, r kl is the divergence constant, which can generally be taken as [0.1, 0.3].

[0089] The formula of the above statistical similarity is a parameter distribution similarity measurement based on KL divergence, and the idea is to quantify the functional similarity of two neuron substructures. If the parameter distributions of two substructures are highly similar, they can be grouped together for comparison. In information theory, KL divergence is a method for measuring the difference between two probability distributions. It describes the amount of information lost when one distribution is approximated by another.

[0090] Because the calculation results between the original KL divergence and are different, Q and P are the original probability distributions of the original KL divergence, so the divergence in the symmetric form, that is, the sum of the KL divergences in both directions, can provide a relatively stable and symmetric similarity measurement.

[0091] The derivation idea of the above statistical similarity formula is as follows:

[0092] (1) From parameters to probability distribution: the weight itself does not constitute a probability distribution, and it needs to be converted into a discrete probability distribution by normalizing the weight matrix through softmax.

[0093] (2) Calculate the probability distribution p(θ j ) of the neuron substructure s j and the probability distribution p(θ i ) of the neuron substructure s i respectively through the original KL divergence calculation formula.

[0094] The original KL divergence calculation formula is:

[0095] ;

[0096] where, is the original KL divergence of the original probability distribution P and Q when the original input variable is x.

[0097] Then the divergence of the symmetric form is:

[0098] ;

[0099] The smaller the divergence value is, the more similar the two original probability distributions P and Q are.

[0100] (3) Set threshold judgment, set an upper threshold for the symmetric divergence, that is, the divergence constant r kl , less than the threshold, it can be considered that the parameter distribution of the two distributions is similar in form.

[0101] Expected technical effect: capture the distribution difference of weight values from the perspective of statistics, avoid misclassification of substructures; the symmetric processing makes the judgment standard and comparison order irrelevant, and has fairness and robustness.

[0102] And the cosine similarity criterion first constructs the normalized vector representation of the parameter tensor:

[0103] ;

[0104] where, vec(θ) is a one-dimensional vector of input variable θ, vec norm (θ) is the normalized unit vector of input variable θ, ||vec(θ)||2 is the two norm value of vector vec(θ).

[0105] Then calculate the geometric direction consistency in high-dimensional space:

[0106] ;

[0107] where, cos sim is the cosine similarity of two vectors vec(θ i ) and vec(θ j ), vec(θ i ) is the parameter vector of neuron substructure s i , vec(θ j ) is the parameter vector of neuron substructure s j , p is the cosine angle constant, the value range can be [0.85, 0.95].

[0108] The formula for calculating the geometric direction consistency of high-dimensional space is a parameter direction consistency measure based on cosine similarity. The idea is that the distribution pattern alone is not enough to determine functional similarity. Two structures with similar distributions may have different weight vector directions, resulting in completely different feature extraction patterns. In vector space, cosine similarity measures the consistency of the direction of two vectors by calculating the cosine of the angle between them. From the perspectives of divergence and cosine, similarity is jointly evaluated.

[0109] The derivation of the formula for calculating the geometric direction consistency of high-dimensional space is as follows:

[0110] (1) Vectorization and normalization: eliminate the influence of weight amplitude and only retain the direction information.

[0111] (2) Calculate the cosine similarity: calculate the dot product between two normalized vectors, i.e., their cosine similarity.

[0112] The specific calculation formula of cosine similarity is:

[0113] ;

[0114] In the formula, and are the two-norm values of vectors vec(θ i ) and vec(θ j ).

[0115] The value range of cosine similarity is [-1, 1], and the closer the value is to 1, the more consistent the direction of the two vectors; the closer the value is to -1, the more opposite the direction of the two vectors.

[0116] (3) Set the threshold value: set a suitable lower threshold p, i.e., the cosine angle constant, if the cosine similarity is greater than or equal to the threshold, it is considered that the two vectors corresponding to the sub-structure are consistent in the direction of the parameter space.

[0117] Expected technical effects: ensure that sub-structures grouped together are not only similar in numerical distribution, but also similar in feature extraction pattern; and complement the KL divergence to jointly build a similarity evaluation system and improve accuracy.

[0118] The grouping is aimed to identify and group substructures with structural similarity, and keep consistency on key attributes: module node number consistency, the number of modules (nodes) contained in each substructure is the same, ensuring the mapping of component granularity; parameter space consistency, the parameter matrix of the corresponding nodes meets the similarity measure, and the divergence analysis and cosine space embedding double threshold mechanism are used to verify the similarity; node dimension consistency, the input and output dimensions of the corresponding nodes are the same, ensuring the compatibility of data flow; index range consistency, the node index in the substructure should be in the same range, to ensure the symmetry and mapping effectiveness of the structure.

[0119] The substructures meeting the above structural similarity standards will be divided into the same structural group. If a certain substructure cannot be matched with other substructures, it will be isolated to form a single group. Finally, the network will be decomposed into several groups in the form of a matrix, providing a basis for subsequent pruning evaluation strategies for each group. As shown in Figure 4 For each neuron substructure, the neuron substructure set can be divided by the above structural similarity determination rules. Neuron substructure s A and neuron substructure s J may be a separate neuron substructure set, neuron substructure s B and neuron substructure s C are divided into the same neuron substructure set, neuron substructure s G and neuron substructure s H are divided into the same neuron substructure set.

[0120] S104: Determine the neuron substructure to be pruned in each neuron substructure set, prune the neuron substructure to be pruned based on the preset pruning method, and obtain the infrared small target detection model after pruning is completed.

[0121] The present embodiment can determine the neuron substructure to be pruned in each neuron substructure set, prune the neuron substructure to be pruned based on the preset pruning method, and obtain the infrared small target detection model after pruning is completed.

[0122] The present embodiment is not limited to the specific way of determining the neuron substructure to be pruned in each neuron substructure set. Generally, the neuron substructure importance value of each neuron substructure in the neuron substructure set is determined, and the importance of the neuron substructure in the neuron substructure set is calculated and sorted based on the neuron substructure importance value.

[0123] A preset pruning ratio is determined, and the neuron substructure in the neuron substructure set after importance calculation and sorting is divided into a pre-preserved neuron substructure and a neuron substructure to be pruned based on the preset pruning ratio.

[0124] The embodiment does not limit the specific way of calculating the neuron substructure importance value, and the L2 norm (Euclidean norm) of the neuron weight in each neuron substructure in the neuron substructure set can be calculated in general, and the neuron importance value is determined based on the L2 norm of the neuron weight:

[0125] ;

[0126] In the formula, w i is the neuron weight of the i th neuron in the neuron substructure, is the square of the L2 norm of the neuron weight of the i th neuron, imp i is the importance value of the i th neuron in the neuron substructure.

[0127] Further, the neuron normalization scaling factor is determined based on the neuron importance value:

[0128] ;

[0129] In the formula, y i is the normalization factor of the i th neuron in the neuron substructure, a is an exponential factor constant, imp max is the maximum importance value of the neurons in the neuron substructure, imp min is the minimum importance value of the neurons in the neuron substructure.

[0130] The product of the neuron importance value and the neuron normalization scaling factor of each neuron in the neuron substructure is summed to obtain the neuron substructure importance value:

[0131] ;

[0132] In the formula, imp s is the importance value of the neuron substructure, and m is the number of neurons in the neuron substructure.

[0133] For any neuron substructure in a neuron substructure set, the importance value of the neuron substructure can be obtained by the above method, and the neuron substructures in the neuron substructure set are divided into pre-retained neuron substructures and to-be-pruned neuron substructures by the importance value of the neuron substructure.

[0134] The pre-retained neuron substructure set Sr is constructed based on the pre-retained neuron substructure, and the to-be-pruned neuron substructure set Sl is constructed based on the to-be-pruned neuron substructure. The present embodiment can prune the to-be-pruned neuron substructure based on a pre-set pruning method to obtain a pruned infrared small target detection model.

[0135] The embodiment is not limited to a specific method of neuron pruning. In the prior art, after determining the structure to be pruned, the entire structure to be pruned is directly pruned. This method may interrupt the continuity of feature distribution and reduce the expression ability of the target feature.

[0136] Therefore, the application can perform progressive enhancement regularization constraints on the weights of the substructure to be removed, update the bias term according to the importance scaling factor, force the decorrelation between the substructure to be retained and the substructure to be removed, and form a pruned model composed of the retained substructure after T iterations.

[0137] First, the number of iterations of the regularization pruning can be determined, the weight penalty term in each iteration can be determined, and the bias penalty term in each iteration can be determined.

[0138] Based on the weight penalty term and the bias penalty term, the weight term and the bias term of each neuron in the neuron substructure to be pruned are updated in each iteration, and an infrared small target detection model after pruning is obtained.

[0139] When processing channel pruning, both the bias term and the weight term are considered for updating to maintain the trainable recovery ability of the pruned model. The gradient update rule of the layer L can be expressed as:

[0140] ;

[0141] The formula is a conventional representation of the existing gradient update, wherein, is the gradient of layer L, is the loss function of layer L, k is a constant factor, w is the weight term, b is the bias term, y is the normalization factor term, the gradient of the weight is constantly updated based on kyw in the iteration process , the gradient of the bias is updated by kyb , the parameters of the pruned model are updated, and the representation ability is enhanced.

[0142] The embodiment is not limited to a specific method of determining the weight penalty term in each iteration. Generally, the decay coefficient and the penalty coefficient can be determined, the square of the two-norm of the weight of each neuron in the neuron substructure to be pruned can be determined, the first derivative between the square of the two-norm of the weight of each neuron and the weight of each neuron can be determined, and the product of the first derivative, the decay coefficient, the penalty coefficient, and the weight of each neuron can be determined as the weight penalty term of each neuron in the neuron substructure to be pruned.

[0143] Specifically, stronger norm regularization is imposed on the neuron substructures in the set of neuron substructures to be pruned Sl, while no additional regularization is imposed on the neuron substructures in the set of pre-retained neuron substructures Sr. This embodiment can adopt a step-by-step strengthening regularization strategy, gradually adjust the penalty coefficient λ and the decay coefficient η, guide weight sparsification, and gradually reduce the weights of unimportant substructures. In a set number of iterations T, the penalty coefficient λ of the neuron substructures to be pruned is updated each time, and the update of the penalty coefficient follows a fixed step size λ = λ + Δλ, where Δλ is a step size, and the upper limit of the regularization is set to 1. , The upper limit of the regularization is set to 1.

[0144] Therefore, in this embodiment, the expression for updating the neuron weights can be:

[0145] ;

[0146] In the expression, w i-old is the updated neuron weight of the i-th neuron in the neuron substructure to be pruned in the last iteration step, w i-new is the updated neuron weight of the i-th neuron in the neuron substructure to be pruned in the current iteration step, η is the decay coefficient, λ i is the penalty coefficient of the i-th neuron in the neuron substructure to be pruned, and is the square of the 2-norm of the updated neuron weight of the i-th neuron in the neuron substructure to be pruned in the last iteration step.

[0147] The expression for updating the neuron weights is a weight update based on gradual regularization. The idea is that if a large regularization strength is applied at once, it will roughly destroy the features learned by the network, resulting in performance degradation. By using a gradual strategy, the penalty on the structure to be pruned is gradually increased with the number of iterations, guiding the removal of unimportant channels.

[0148] The derivation process of the expression for updating the neuron weights is as follows:

[0149] (1) Dynamic penalty coefficient: The penalty coefficient λ is not a constant, but a quantity that changes with the number of iterations.

[0150] (2) Substitute this dynamically increasing penalty coefficient λ into the gradient update rule. For the neurons in the set to be pruned, the regularization term will gradually increase in strength with the increase in the number of iterations, and the weights of the corresponding channels are continuously suppressed. For the neurons in the retained set, the regularization term does not work, and the performance is maintained.

[0151] Expected technical effects: Avoid performance mutations caused by excessive regularization, make the pruning process more smooth and stable, and indirectly control the pruning process by controlling the penalty coefficient. This gradual and gentle way maximizes the performance of important structures and provides protection for high-performance pruning.

[0152] In each iteration, the update of the weight term, the cross-correlation term of the Sl and Sr set tends to zero under the regularization constraint, while the autocorrelation term of the reserved subset remains a significant non-zero value.

[0153] Performance recovery compensation is the last step to obtain a lightweight and high-performance model. For the sparse model composed of the reserved neuron substructure, the original training parameters (such as learning rate, decay function, etc.) are used to fine-tune the pruned model end-to-end to obtain the final lightweight model.

[0154] Specifically, an infrared small target detection training data set is obtained, and the parameter of the pruned infrared small target detection model is fine-tuned based on the training data set.

[0155] Based on the above embodiment, the method of the application protects the key structure of the model by setting a shielding layer and identifies and forms a strongly associated substructure based on the coupling dependence of neurons. According to the structural similarity, groups are formed, the integrity of the cross-layer feature fusion path is effectively maintained, and the performance of the pruned model is avoided.

[0156] The following is an infrared small target lightweight detection method provided by an embodiment of the application. The method can include:

[0157] Obtaining a to-be-detected image, inputting the to-be-detected image into an infrared small target detection model, and obtaining an infrared small target detection result output by the model;

[0158] The infrared small target detection model is an infrared small target detection model pruned according to the structure grouping decoupling pruning method.

[0159] In this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

Claims

1. A lightweight infrared small target detection method, characterized in that, include: Acquire the image to be detected, input the image to be detected into the infrared small target detection model, and obtain the infrared small target detection result output by the model; The infrared small target detection model is the infrared small target detection model after being pruned according to the structural grouping deassociation pruning method; The structural grouping deassociation pruning method includes: Each model layer of the infrared small target detection model is determined, a shielding layer is determined from the model layers, and a set of layers to be pruned is constructed based on the model layers excluding the shielding layer. Based on the neuronal coupling relationship, the neurons in the set of layers to be pruned are divided into multiple neuronal substructures; Structural similarity is verified among the neuronal substructures, and neuronal substructures with similar structures are grouped into the same set of neuronal substructures; In each set of neuron substructures, the neuron substructure to be pruned is determined, and the neuron substructure to be pruned is pruned based on a preset pruning method to obtain the infrared small target detection model after pruning. The substructures of neurons to be pruned are pruned based on a preset pruning method to obtain a pruned infrared small target detection model, including: Determine the number of iterations for regular pruning, determine the weight penalty term in each iteration, and determine the bias penalty term in each iteration. Based on the weight penalty term and the bias penalty term, the weight term and bias term of each neuron in the neuron substructure to be pruned are updated in each of the said iterations to obtain the infrared small target detection model after pruning.

2. The lightweight infrared small target detection method according to claim 1, characterized in that, Based on the neuron coupling relationship, the neurons in the set of layers to be pruned are divided into multiple neuron substructures, including: Target neurons are identified from neurons within the set of layers to be pruned that have not been assigned to the neuronal substructures. Identify the associated neurons that are coupled with the target neuron from among the neurons in the set of layers to be pruned that have not been assigned to the neuronal substructure; The neuron substructure is constructed based on the target neuron and the associated neurons; The step of re-determining the target neuron from the neurons in the set of layers to be pruned that have not been assigned to the neuronal substructure is executed until the neuronal substructure contains all the neurons in the set of layers to be pruned.

3. The lightweight infrared small target detection method according to claim 1, characterized in that, Perform structural similarity checks between the neuron substructures, and group structurally similar neuron substructures into the same set of neuron substructures, including: Perform consistency checks on the number of neurons, parameter space, neuron dimension, and index range among the neuron substructures. Neuron substructures that pass the consistency checks of the number of neurons, parameter space, neuron dimension, and index range are divided into the same set of neuron substructures. If a single neuron substructure fails to pass the consistency checks for the number of neurons, parameter space, neuron dimension, and index range with any other neuron substructure, then that single neuron substructure is divided into a set of neuron substructures.

4. The lightweight infrared small target detection method according to claim 3, characterized in that, The parameter space consistency check includes divergence check and cosine similarity check.

5. The lightweight infrared small target detection method according to claim 1, characterized in that, Identify the neuronal substructures to be pruned from each of the aforementioned sets of neuronal substructures, including: Determine the importance value of each neuron substructure within the set of neuron substructures, and calculate and sort the importance of each neuron substructure within the set of neuron substructures based on the importance value; A preset pruning ratio is determined, and based on the preset pruning ratio, the neuron substructures within the set of neuron substructures, after importance calculation and sorting, are divided into pre-reserved neuron substructures and neuron substructures to be pruned.

6. The lightweight infrared small target detection method according to claim 5, characterized in that, Determining the importance value of each neuron substructure within the set of neuron substructures includes: Determine the L2 norm of the neuron weights in each of the neuron substructures within the set of neuron substructures, and determine the neuron importance value based on the L2 norm of the neuron weights; The neuron normalization scaling factor is determined based on the neuron importance value; The importance value of the neuron substructure is obtained by summing the product of the neuron importance value and the neuron normalization scaling factor of each neuron within the neuron substructure.

7. The lightweight infrared small target detection method according to claim 1, characterized in that, Determine the weight penalty term in each of the aforementioned iteration numbers, including: Determine the attenuation coefficient and penalty coefficient; determine the square of the L2 norm of the weights of each neuron in the substructure of the neuron to be pruned; Determine the first derivative between the square of the L2 norm of the neuron weights and the neuron weights; The product of the first derivative, the decay coefficient, the penalty coefficient, and the neuron weight is determined as the weight penalty term for each neuron in the substructure of the neuron to be pruned.

8. The lightweight infrared small target detection method according to claim 1, characterized in that, Also includes: Obtain an infrared small target detection training dataset, and fine-tune the parameters of the pruned infrared small target detection model based on the training dataset.

Images