A model compression method based on coded metasurface frequency response fast prediction

Abstract

The application discloses a model compression method based on coded metasurface frequency response fast prediction, relates to the field of coded metasurface design, and comprises the following steps: step one, a coded matrix representing the surface structure of a coded metasurface unit is imported through CST-Python joint simulation, coded metasurface frequency response is generated, a frequency response curve is obtained, and a data set is made; step two, a forward network for coded metasurface frequency response fast prediction is pre-trained, and mapping from a coded metasurface to frequency response is realized; step three, based on the pre-trained complete forward network obtained in step two, the network output is used as correlation and backward propagated through LRP, the correlation obtained by a convolution kernel is used as a measurement standard for structured pruning, and the forward network is compressed; and step four, the coded matrix representing the surface structure of the coded metasurface unit is input into the compressed forward network, the real part curve and the imaginary part curve of the metasurface frequency response are obtained, and then amplitude and phase values are obtained.

Description

TECHNICAL FIELD

[0001] The application relates to the field of coding metasurface design, and in particular to a model compression method based on fast prediction of frequency response of coding metasurfaces. BACKGROUND

[0002] The unit structure of a traditional electromagnetic metasurface is still described by continuous (uniform or non-uniform) equivalent medium parameters, so from the perspective of circuit, such a metasurface is similar to an analog circuit and can be regarded as an "analog metasurface". In order to realize a "digital version" of the metasurface, a team proposes a new type of coding metasurface based on digital representation, that is, electromagnetic waves are regulated by a digital coding sequence. This "digital metasurface" based on binary digital logic greatly simplifies the design process of the metasurface, improves the flexibility of regulating electromagnetic waves, and expands the regulation range. In recent years, intelligent algorithms covering traditional heuristic algorithms and neural network algorithms gradually increase in the proportion of metasurface design. Designing metasurfaces based on intelligent algorithms can break the limitations of traditional design methods in different substrate systems, different frequency bands and different performance indicators, and exhibit unique advantages of rapid design and architecture innovation. With the improvement of the performance of deep neural networks, the number of network model parameters and the amount of calculation are also increasing. The depth of classic neural network models such as AlexNet, VGGNet, GoogleNet and ResNet is increasing, and gradually exceeds 100 layers.

[0003] The fast prediction of frequency response of deep coding metasurfaces can be realized by using a neural network. However, the deployment of the neural network requires a very high amount of calculation and storage resources, which makes it difficult to apply to resource-limited mobile and wearable devices, and the application is greatly limited. At the same time, there are many redundant parameters in the neural network.

[0004] Therefore, it is necessary to compress the model of neural networks. There are five main methods for model compression: compact structure design, quantization, low-rank decomposition, knowledge distillation, and pruning. The application scenarios of compact structure design are mostly in the network design for new tasks, and it is limited for networks with fixed structures. Quantization of complex task networks usually leads to a significant decrease in accuracy. The low-rank decomposition method mainly uses matrix decomposition, but it has the problem of not converging after decomposition, and the time cost of decomposition is relatively high. In addition, in recent years, models use a large number of 1x1 convolutions, which further reduces the scope of matrix decomposition. Knowledge distillation can greatly reduce the network computation for deep networks, but due to the use of the softmax function, its generalization is limited for tasks other than classification, and its compression ratio is also inferior to other methods. Pruning can be divided into unstructured pruning and structured pruning according to the granularity. The weakness of unstructured pruning is obvious. Due to the internal irregular connection, it needs to rely on special runtime libraries and hardware devices, so structured pruning is more popular

[0005] Structured pruning at the level of convolution kernels not only greatly reduces the use of storage space, but also reduces the computational cost of online inference. The core of structured pruning is the selection of convolution kernel measurement standards, which requires the highest compression ratio with the smallest accuracy loss. Existing structured pruning methods at the level of convolution kernels can be divided into two categories:

[0006] First, without changing the loss function of the convolutional neural network, pruning is performed using the inherent properties of the neural network, such as the structured pruning method based on the sparsity of convolution kernel weights APOZ, the structured pruning method based on the L1 norm of convolution kernel weights, and the structured pruning method based on the first-order Taylor expansion.

[0007] Second, the loss function of the convolutional neural network is changed, and the pruning standard needs to be embedded into the network training loss. Joint optimization is used to produce adaptive pruning decisions, such as embedding the scaling factor of the BN layer into the loss function for training. After training, the convolution kernel with a smaller scaling factor can be considered relatively unimportant.

[0008] Compared with the first pruning method, the second pruning method can usually obtain better compression and acceleration results due to joint optimization. However, due to the change in the loss, the retraining step requires a large amount of machine time and manpower, and usually requires another round of hyperparameter tuning. For some methods, the modified loss even requires a special optimizer. The structured pruning method also has the problem of large computational complexity. For example, the structured pruning method based on the first-order Taylor expansion not only needs to calculate the gradient, but also needs to perform layer regularization on the obtained first-order Taylor term, and it also lacks interpretability of the convolution kernel measurement standard.

[0009] Based on the above problems, the application proposes a structured pruning method based on LRP (Layer-wise Relevance Propagation). Unlike the LRP-based pruning method applied to classification problems, the application applies the pruning method to the forward network compression of the fast prediction of the frequency response of the coded metasurface. The forward network for the fast prediction of the frequency response of the coded metasurface aims to realize the mapping of the coded metasurface and its frequency response through the training of a large data set, which is a regression problem. The structured pruning method based on LRP (Layer-wise Relevance Propagation) only needs to backward propagate the network output from the output layer to the input layer to obtain the contribution of each convolution kernel in each layer to the network output, which is called relevance. Since the pruning criterion is directly related to the network output, and the constraint of constant relevance between layers is imposed during the backward propagation process, the relevance obtained by LRP as the pruning criterion becomes interpretable. SUMMARY

[0010] The application proposes a model compression method based on the fast prediction of the frequency response of the coded metasurface, which compresses the forward network for the fast prediction of the frequency response of the coded metasurface through the structured pruning method based on LRP. The structured pruning method based on LRP greatly reduces the network parameters under the premise of basically unchanged network performance, solving the problems of excessive network storage space requirements and slow prediction speed caused by the excessive number of network model parameters and the large amount of calculation.

[0011] Technical scheme: A model compression method based on the fast prediction of the frequency response of the coded metasurface, comprising the following steps:

[0012] Step one, import the coding matrix representing the surface structure of the coded metasurface unit through CST-Python joint simulation, generate the frequency response of the coded metasurface, obtain the frequency response curve, and make a data set;

[0013] Step two, pre-train the forward network for the fast prediction of the frequency response of the coded metasurface to realize the mapping of the coded metasurface to the frequency response;

[0014] Step three, based on the pre-trained complete forward network obtained in step two, use LRP to backward propagate the network output as relevance, and use the relevance obtained by the convolution kernel as the measurement standard of structured pruning to compress the forward network, and retrain the compressed forward network;

[0015] Step four, input the coding matrix representing the surface structure of the coded metasurface unit into the forward network obtained in step three to obtain the real part curve and the imaginary part curve of the frequency response of the metasurface, and further obtain the amplitude and phase values.

[0016] Further, in the step one, the working frequency of the coded metasurface unit is 8-12 GHz; the coded metasurface unit includes three layers, the lowermost layer is a copper plate with an electrical conductivity of 5.8e+007 S / m; the middle layer is a dielectric plate, the dielectric plate is F4B material with a dielectric constant of 2.65, the length and width are both 10 mm, and the thickness is 2 mm; the uppermost layer is a metal patch layer, the uppermost layer has 16*16 units, which are symmetrically formed by 8*8 units, each unit is represented by 1 or 0 to indicate the presence or absence of the patch, and the length and width of a single patch are 0.5 mm and the thickness is 0.017 mm.

[0017] Further, in the step one, the generated metasurface frequency response is denoted as S11, the matrix representing the metasurface frequency response S11 is interval sampled to obtain the correspondence between the metasurface unit and its frequency response, and form a plurality of initial data; the initial data is divided into a training set, a validation set and a test set according to a ratio of 8:1:1; the training set is expanded to 2 times by data augmentation;

[0018] In the parameter representation of S11, the amplitude Amp and the phase The real part Re and the imaginary part Im correspond to the following relationship:

[0019]

[0020]

[0021]

[0022]

[0023] Further, in the step two, the forward network for fast prediction of the coded metasurface frequency response includes a real part sub-network Rnet and an imaginary part sub-network Inet, the two sub-networks are in parallel joint form, and the two sub-networks adopt the same structure; the input of the two sub-networks is a coding matrix representing the surface structure of the coded metasurface unit; the outputs of the two sub-networks correspond to the real part and the imaginary part of the metasurface frequency response S11 respectively, and based on the correspondence between the amplitude, the phase, the real part and the imaginary part, the amplitude and the phase of the network output are calculated;

[0024] When pre-training the forward network:

[0025] 1) The error function used by the network is the minimum mean square error MSE, which is represented as,

[0026]

[0027] wherein, The prediction result of the forward network to the coded metasurface frequency response S11, y is the actual frequency response of the coded metasurface, and N is the frequency point number obtained by equally sampling the coded metasurface frequency response S11 in the frequency domain;

[0028] 2) The network loss function during pre-training is:

[0029]

[0030] Wherein, loss_train is the loss function of the network during pre-training, Re, respectively, the real part of the real value of the S11 parameter, and the real part of the real part of the sub-network prediction, Im, respectively, the real part of the real value of the S11 parameter, and the real part of the real part of the sub-network prediction.

[0031] Further, in step three, the steps of the structured pruning method are as follows:

[0032] Step A, calculate the activation value;

[0033] Randomly select a coded metasurface unit as a reference sample in the training set, input the coding matrix representing the surface structure of the reference sample metasurface unit into the pre-trained complete forward network in step two, and independently forward propagate the real part sub-network and the imaginary part sub-network of the forward network, respectively, to obtain the output value of each layer in the real part sub-network and the imaginary part sub-network, which is the activation value, and the pre-activation value is the output value without passing through the activation function in the layer;

[0034] Step B, correlation calculation based on LRP;

[0035] Backpropagate the output of the pre-trained complete forward network in step two to the input layer, and independently backpropagate the output of the real part sub-network and the imaginary part sub-network of the forward network to the network input layer, and obtain the correlation between the convolution kernel and the network output in each layer of the real part sub-network and the imaginary part sub-network based on the LRP rule;

[0036] Step C, calculate the correlation of each convolution kernel;

[0037] Based on the LRP rule, after backpropagating the network output to the input end, the activation values of all layers in the network obtain the correlation with the network output. For the convolution layer, the correlation of the convolution kernel is the sum of the correlation of all activation values output by the convolution kernel;

[0038] Step D, according to the correlation of the convolution kernel, the structured pruning is performed in the form of overall pruning;

[0039] The real part sub-network and the imaginary part sub-network of the pre-trained complete forward network in step two are pruned respectively; a pruning ratio, i.e. the ratio of pruned convolution kernels to all convolution kernels, is determined, all convolution kernels are sorted according to the correlation, and a correlation threshold is obtained; the convolution kernels with a correlation lower than the threshold and the corresponding input-output connections are pruned to obtain a compressed forward network.

[0040] Further, the specific steps of step B are as follows:

[0041] Step 1, the correlation between the convolution kernels of the front layer and the back layer is represented by the ratio of the pre-activation values associated between the convolution kernels to the total pre-activation values of all associated front layer convolution kernels of the back layer convolution kernel, and the propagation formula of the correlation is:

[0042]

[0043] z ij = a i W ij

[0044] wherein, is the correlation of the jth pre-activation value of the L+1th layer; is the correlation of the jth pre-activation value of the L+1th layer transmitted to the ith activation value of the Lth layer; j z ii is the pre-activation value of the jth convolution kernel of the L+1th layer associated with the ith convolution kernel of the Lth layer; i is the ith activation value of the Lth layer; ij is the weight between the ith activation value of the Lth layer and the jth pre-activation value of the L+1th layer;

[0045] Step 2, considering that a small z j value may produce uncontrollable values, a preset number greater than 0, ε, is added, and the propagation formula of the original correlation is represented as:

[0046]

[0047] Step 3, since the correlation is absorbed by ε in the propagation process, the sum of the correlation obtained by the convolution layers closer to the input is smaller, so that the comparison of the correlation in different convolution layers is meaningless, and the propagation formula of the correlation is represented as:

[0048]

[0049] wherein, α+β=1 is required, and the pre-activation value can be divided into two parts greater than 0 and less than 0, wherein ‘+’ represents a value greater than 0, ‘-’ represents a value less than 0, effectively avoiding the problem of abnormal values;

[0050] Taking α = 1 and β = 0, and considering that the activation values of all layers except the output layer are not less than 0, the propagation formula of the correlation becomes:

[0051]

[0052] Step 4, during back propagation, set the network output to 1 as the starting point of correlation propagation, and only consider pre-activation values not less than 0 during propagation, so that the correlation of all neurons in the network is not less than 0;

[0053] According to the propagation rule when α = 1 and β = 0, the sum of the correlations of different layers is equal without considering the bias term of the convolution layer or the bias term of the fully connected layer, that is, it satisfies:

[0054]

[0055] where f(x) is the network output, V(L) represents the number of activation values of the Lth layer, represents the correlation of the dth activation value of the Lth layer; when considering the bias term of the convolution layer or the bias term of the fully connected layer, a small part of the correlation is absorbed by the bias term during network propagation.

[0056] Further, in step D, a physical constraint term is added to the obtained compressed forward network, and the performance of the forward network is restored through retraining:

[0057] 1) The network loss function for retraining the forward network is:

[0058]

[0059]

[0060] where loss_retrain is the loss function of the pruned network during retraining; λ is a hyperparameter; Re, respectively, the real part of the real value of the S11 parameter, and the real part of the predicted value of the subnetwork, Im, respectively, the imaginary part of the real value of the S11 parameter, and the imaginary part of the predicted value of the subnetwork, Amp, respectively, the amplitude of the real value of the S11 parameter, and the amplitude of the predicted value, respectively, the phase of the real value of the S11 parameter, and the phase of the predicted value; the amplitude of the predicted value and the phase of the predicted value can be calculated from and ; in the loss function As a physical constraint term;

[0061] 2) In addition to the network output layer, the activation function of the remaining layers in the network is Relu, and the function expression of Relu is

[0062]

[0063] Since the value range of the real part and the imaginary part of the frequency response of the coding metasurface is [-1, 1], the activation function of the two sub-network output layers is tanh:

[0064]

[0065] During training, the Adam optimizer is selected, the training number is set to 120 rounds, the initial learning rate is 1e-3, and the learning rate is reduced by half every 40 rounds of training.

[0066] Beneficial effects:

[0067] 1) The structured pruning method based on LRP is applied to the model compression of the coding metasurface forward prediction network, and compared with the pruning method based on the L1 norm of the convolution kernel weight without retraining the network after pruning:

[0068] On the real part sub-network and the imaginary part sub-network, the structured pruning method based on LRP obtains better compression effect than the pruning method based on the L1 norm of the convolution kernel weight, and is more suitable for the compression of the coding metasurface forward prediction network.

[0069] Meanwhile, the structured pruning method based on LRP has small changes in network accuracy when the pruning ratio is certain, and it is more reasonable to prune by using the contribution of each network unit to the network output, the correlation of each convolution kernel is directly related to the network output, and the constraint of the layer-to-layer correlation is unchanged, so that the correlation can be explained as a measurement standard for structured pruning, and the correlation of the convolution kernel, the BN layer, the pooling layer and the fully connected layer can be back propagated, so that LRP has strong adaptability in different networks.

[0070] 2) The structured pruning method based on LRP is used to compress the coding metasurface frequency response forward network to different degrees, and the balance of network parameter quantity, inference speed and network loss can be obtained:

[0071] Using the structured pruning method based on LRP to compress the forward network can greatly reduce the network parameter quantity, significantly speed up the network inference speed, and occupy less memory space under the premise of ensuring the prediction accuracy of the network to the frequency response S11 (real part representation, imaginary part representation, amplitude representation, phase representation), which makes the prediction network can be applied to resource-limited devices, and improves the portability of the network. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 Schematic diagram for forward design and reverse design of coding metasurface;

[0073] Figure 2 Structural diagram of coding metasurface unit;

[0074] Figure 3 Flow chart of frequency response fast prediction method for coding metasurface;

[0075] Figure 4 Schematic diagram of unit structure of forward network;

[0076] Figure 5 Schematic diagram of structured pruning of frequency response prediction network of coding metasurface based on LRP;

[0077] Figure 6 Schematic diagram of forward propagation of forward network;

[0078] Figure 7 Schematic diagram of backward propagation based on LRP;

[0079] Figure 8 Curve of change of sum of correlation of real part subnetwork and imaginary part subnetwork with backward propagation;

[0080] Figure 9 Correlation map of each pixel point of metasurface unit;

[0081] Figure 10 Comparison diagram of compression effect of structured pruning method based on LRP (sample number is 1) and structured pruning method based on L1 norm of convolution kernel weight in real part subnetwork;

[0082] Figure 11 Comparison diagram of compression effect of structured pruning method based on LRP (sample number is 1) and structured pruning method based on L1 norm of convolution kernel weight in imaginary part subnetwork;

[0083] Figure 12 Comparison diagram of compression effect of structured pruning method based on LRP (sample number is 5) and structured pruning method based on L1 norm of convolution kernel weight in real part subnetwork;

[0084] Figure 13 Comparison diagram of compression effect of structured pruning method based on LRP (sample number is 5) and structured pruning method based on L1 norm of convolution kernel weight in imaginary part subnetwork;

[0085] Figure 14 Comparison diagram of compression effect of structured pruning method based on LRP (sample number is 10) and structured pruning method based on L1 norm of convolution kernel weight in real part subnetwork;

[0086] Figure 15 This is a comparison of the compression performance of the LRP-based structured pruning method (with 10 samples) and the L1 norm-based structured pruning method based on convolutional kernel weights in the imaginary part subnetwork.

[0087] Figure 16 This is a comparison of the compression performance of the LRP-based structured pruning method (with 50 samples) and the L1 norm-based structured pruning method based on convolutional kernel weights in the real part subnetwork.

[0088] Figure 17 This is a comparison of the compression performance of the LRP-based structured pruning method (with 50 samples) and the L1 norm-based structured pruning method based on convolutional kernel weights in the imaginary part subnetwork. Detailed Implementation

[0089] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the embodiments.

[0090] This invention primarily utilizes neural networks to design coded metasurfaces. Neural networks, through learning from large datasets, can achieve a mapping from input to output. The design of coded metasurfaces, based on the relationship between the neural network's input and output, can be broadly categorized as follows: Figure 1 The invention illustrates two designs: forward design (a) and reverse design (a). This invention primarily focuses on forward design, such as... Figure 1 As shown in (a), electromagnetic responses (such as transmission spectra or differential scattering cross sections) are predicted by input structural parameters (such as the geometry of the microwave structure). These trained neural networks are used to replace the numerical simulation process of electromagnetic simulation, which saves a lot of time compared to simulating and outputting the electromagnetic response in CST, and is more adaptable to changes in the input structure.

[0091] like Figure 3 As shown in this embodiment, a model compression method based on fast prediction of the frequency response of encoded metasurfaces includes the following steps:

[0092] Step 1: Using CST-Python co-simulation, import the encoding matrix representing the surface structure of the encoded metasurface unit, generate the frequency response of the encoded metasurface, obtain the frequency response curve, and create a dataset;

[0093] In step one, the structure of the encoded metasurface unit is as follows: Figure 2As shown, the operating frequency of the metasurface is 8GHz-12GHz; the metasurface unit includes three layers, the lowermost layer is a copper plate with an electrical conductivity of 5.8e+007 S / m; the middle layer is a dielectric plate, the dielectric plate is F4B material with a dielectric constant of 2.65, and the length, width and thickness are all 10mm, 2mm respectively; the uppermost layer is a metal patch layer, the uppermost layer has 16*16 units, which are symmetrically formed by 8*8 units, each unit is represented by 1 or 0 to indicate the presence or absence of the patch, and the length, width and thickness of a single patch are 0.5mm, 0.5mm and 0.017mm respectively; by changing the distribution of 0 and 1 of the 8*8 unit (a total of 2 64 different patterns of metasurface units can be obtained, thereby realizing different frequency responses.

[0094] In step one, the generated frequency response of the coded metasurface is denoted as S11, the matrix representing the frequency response S11 of the metasurface is interval sampled to obtain the correspondence between the metasurface unit and its frequency response, and a plurality of initial data are formed; the initial data are divided into a training set, a validation set and a test set according to a ratio of 8:1:1; the training set is expanded to 2 times by data augmentation; S11 can be represented by amplitude, phase, real part and imaginary part, i.e. {amplitude Amp, phase real part Re, imaginary part Im}; the correspondence of the parameters of S11 is as follows:

[0095]

[0096]

[0097]

[0098]

[0099] In this embodiment, the matrix of S11 is a [1001, 4] matrix, 1001 is the initial frequency point number of the frequency response in the frequency domain, and the present application is interval sampled with a step of 25 to obtain 41 frequency points, i.e. a [41, 4] matrix, thereby realizing the correspondence between the metasurface unit and its frequency response and forming a data, and the initial data has 40000. The ratio of the training set, the validation set and the test set is 8:1:1, and the training set has 32000, in order to improve the network generalization ability, the training set is expanded to 64000 by data augmentation based on the characteristics of the coded metasurface unit.

[0100] Step two, pre-training the forward network for fast prediction of the frequency response of the coded metasurface, to realize the mapping from the coded metasurface to the frequency response;

[0101] ​The forward network includes a real part sub-network Rnet and an imaginary part sub-network Inet, the two sub-networks are in parallel joint form, and the two sub-networks adopt the same structure; the network unit structure is as shown in Figure 4 The parameters are updated during training; the inputs of the two sub-networks are both coding matrices representing the surface structure of the coding metasurface unit; the outputs of the two sub-networks correspond to the real part and the imaginary part of the frequency response S11 of the metasurface respectively; based on the corresponding relationship between the amplitude, the phase and the real part and the imaginary part, the amplitude and the phase of the network output are calculated;

[0102] When the forward network for fast prediction of the frequency response of the coding metasurface is pre-trained, the error function used by the network is the minimum mean square error (MSE) and the network loss function is as follows:

[0103] 1) The error function used by the network is the minimum mean square error (MSE) and is as follows:

[0104]

[0105] Wherein, is the prediction result of the forward network for the frequency response S11 of the coding metasurface, y is the actual frequency response of the coding metasurface, and N is the frequency point number obtained by equally sampling the frequency response S11 of the coding metasurface in the frequency domain;

[0106] 2) The network loss function during pre-training is as follows:

[0107]

[0108] Wherein, loss_train is the loss function during pre-training of the network, Re, are the real part true value and the real part value predicted by the real part sub-network of the S11 parameter respectively, and Im, are the imaginary part true value and the imaginary part value predicted by the imaginary part sub-network of the S11 parameter respectively.

[0109] Step three, based on the pre-trained complete forward network obtained in step two, the network output is used as the correlation for back propagation using LRP, the correlation obtained by the convolution kernel is used as the measurement standard for the structured pruning of the forward network, and the compressed forward network is retrained;

[0110] As shown in Figure 5 The structured pruning method is a structured pruning method for the coding metasurface prediction network based on LRP, and the specific steps are as follows:

[0111] Step A, calculate the activation value;

[0112] A reference sample is randomly selected from the training set, and the encoding matrix representing the surface structure of the reference sample is input into the pre-trained complete forward network in step two. The real part sub-network and the imaginary part sub-network of the forward network are independently forward propagated to obtain the output value of each layer, which is the activation value. The pre-activation value is the output value without the activation function in the layer. The forward propagation of the forward network is shown in Figure 6 ;

[0113] Step B, correlation calculation based on LRP;

[0114] The pre-trained complete forward network in step two is backward propagated to the input layer. The real part sub-network and the imaginary part sub-network of the forward network are independently backward propagated to the network input layer. The backward propagation based on LRP is shown in Figure 7 . The correlation between the convolution kernel and the network output in each layer of the real part sub-network and the imaginary part sub-network is obtained based on the LRP rule. The specific steps are as follows:

[0115] Step 1, according to the activation value between the associated convolution kernels calculated by the forward propagation, the correlation between the convolution kernels of the previous layer and the next layer is represented by the ratio of the pre-activation value associated between the convolution kernels and the total pre-activation value of all associated convolution kernels of the next layer. The propagation formula of the correlation is as follows:

[0116]

[0117] z ij =a i W ij

[0118] wherein, is the correlation of the jth pre-activation value of the L+1th layer; is the correlation of the jth pre-activation value of the L+1th layer transmitted to the ith activation value of the Lth layer; j z ij is the pre-activation value of the jth convolution kernel of the L+1th layer associated with the ith convolution kernel of the Lth layer; i is the ith activation value of the Lth layer; ij is the weight between the ith activation value of the Lth layer and the jth pre-activation value of the L+1th layer;

[0119] Step 2, considering that a small value of z j may produce uncontrollable values, a number greater than 0 is added, and the original propagation formula is represented as:

[0120]

[0121] Step 3, since the correlation is absorbed by ε in the propagation process, the sum of the correlation obtained by the convolution layer closer to the input is smaller, so that the comparison of the correlation in different convolution layers is meaningless, therefore, the propagation formula is expressed as:

[0122]

[0123] where it is required that α+β=1, the rule divides the pre-activation value into two parts greater than 0 and less than 0, where ‘+’ represents the value greater than 0, and ‘-’ represents the value less than 0, which effectively avoids the problem of abnormal value;

[0124] In this embodiment, α=1 and β=0 are taken, and considering that the activation value of all layers except the output layer is not less than 0, the propagation formula becomes:

[0125]

[0126] Step 4, in the back propagation, the present embodiment does not directly take the network output f(x) as the correlation of the output layer, but takes the network output of 1 as the starting point of the correlation propagation, so as to obtain the robustness of the structured pruning; in the back propagation, since the output layer of 1 is taken as the starting point of the correlation and only the pre-activation value not less than 0 is considered in the propagation process, the correlation of all neurons in the network is not less than 0;

[0127] According to the propagation rule of α=1 and β=0, in the case where the bias term of the convolution layer or the bias term of the fully connected layer is not considered, the sum of the correlation of different layers is equal, that is, it satisfies:

[0128]

[0129] where f(x) is the network output, V(L) represents the number of activation values of the Lth layer, represents the correlation of the dth activation value of the Lth layer; when the bias term of the convolution layer or the bias term of the fully connected layer is considered, a small part of the correlation is absorbed by the bias term in the network propagation process;

[0130] The curve of the sum of the correlation of the real part sub-network and the imaginary part sub-network with the change of the back propagation is shown in Figure 8 , both of which show the same characteristics, and it can be seen from Figure 8 that the sum of the correlation from the output layer of the network to the input layer of the network is only slightly reduced, and this reduction is derived from the absorption of the bias term in the real part sub-network, and the correlation does not change when the propagation is performed in the layer without the bias term. The correlation of the real part sub-network and the imaginary part sub-network is propagated from the output layer of each to the input layer of the network, and the contribution degree of each pixel point of the hyper surface to the output can be obtained, that is, as shown in Figure 9The correlation map of each pixel of the super surface unit shown; since the correlation of each neuron is set to 1 in the output layer and the LRP rule of alpha = 1 and beta = 0 is adopted, the correlation of all neurons in the network is greater than or equal to 0, and the greater the R is, the greater the contribution to the network output is, that is, the more important the neuron is in the network;

[0131] Step C, calculating the correlation of each convolution kernel;

[0132] After the network output is back propagated to the input end based on the LRP rule, the activation values of all layers in the network obtain the correlation with the network output, and for the convolution layer, the correlation of the convolution kernel is the sum of the activation values of all the outputs of the convolution kernel; based on the direct correlation between each convolution kernel and the network output and the constraint that the layer-to-layer correlation is unchanged, the correlation can be explained as a measurement standard for structured pruning; whether it is a convolution kernel, a BN layer, a pooling layer or a fully connected layer, the correlation can be back propagated, so the LRP has strong adaptability in different networks;

[0133] Step D: according to the correlation, structured pruning is performed in the form of overall pruning;

[0134] Due to the constraint that the layer-to-layer correlation is unchanged during the propagation of LRP, the structured pruning based on LRP does not need to consider the sensitivity of the convolution layer, that is, the layered pruning method is not needed; the real part sub-network and the imaginary part sub-network of the present application have the same structure, and the same pruning ratio is adopted for pruning;

[0135] The specific steps of structured pruning are: the real part sub-network and the imaginary part sub-network of the trained complete forward network in step two are separately pruned to determine the pruning ratio, that is, the proportion of the pruned convolution kernel to all convolution kernels, the correlation of all convolution kernels is sorted to obtain a correlation threshold; the convolution kernel with a correlation lower than the threshold and the corresponding input-output connection are pruned to obtain a compressed forward network.

[0136] The compressed forward network is obtained, a physical constraint term is added, and the performance of the forward network is restored through retraining:

[0137] 1) The network loss function for retraining the forward network is:

[0138]

[0139] Wherein, loss_retrain is the loss function of the pruned network during retraining; lambda is a hyperparameter; Re, S11 are the real part real value and the real part value predicted by the real part sub-network, respectively, Im, S11 are the imaginary part real value and the imaginary part value predicted by the imaginary part sub-network, respectively, Amp, are respectively the real value of the amplitude of the S11 parameter, the predicted value of the amplitude, are respectively the real value of the phase of the S11 parameter, the predicted value of the phase; the predicted value of the amplitude and the predicted value of the phase can be calculated by and In the loss function, the physical constraint term is In this embodiment, the size of lambda is 0.01.

[0140] 2) In addition to the network output layer, the activation function of the remaining layers in the network is Relu, and the function expression of Relu is

[0141]

[0142] Since the value range of the real part and the imaginary part of the frequency response of the coded metasurface is [-1, 1], the activation function of the output layer of the two sub-networks is tanh:

[0143]

[0144] During training, the Adam optimizer is selected, the training number is set to 120 rounds, the initial learning rate is 1e-3, and the learning rate is reduced by half every 40 rounds of training.

[0145] Step four, input the coding matrix representing the surface structure of the coded metasurface unit into the forward network obtained in step three, so as to obtain the real part curve and the imaginary part curve of the frequency response of the metasurface, and further obtain the amplitude and phase values;

[0146] The compressed forward network can replace CST to predict the frequency response of the metasurface. By inputting the 16*16 coding matrix representing the surface structure of the coded metasurface unit into the forward network, the real part curve and the imaginary part curve of the frequency response S11 parameter can be obtained, and the amplitude and phase values can be calculated. In this embodiment, the error between the frequency response S11 predicted by the forward network and the actual frequency response S11 of the single metasurface (represented by the real part and the imaginary part) is small.

[0147] The forward network used in the application has a comparable prediction ability to ResNet34. Due to the uneven structure of ResNet34, ResNet34 has a total of 16 convolutional layers that can be pruned and are not constrained. The total number of prunable convolutional kernels is 3776, accounting for about 44.7% of the total number of convolutional kernels. The total number of prunable convolutional kernels of the forward network proposed in the application accounts for 100% of the total number of convolutional kernels, which is more conducive to the execution of subsequent structured pruning.

[0148] The LRP-based structured pruning method and the convolution kernel weight L1 norm-based structured pruning method are respectively used to compress the coding metasurface frequency response fast prediction network, without retraining the compressed network, and the loss change of the network on the test set under the same pruning ratio is compared, and the reference sample in the application is a coding metasurface unit randomly selected in the training set. The convolution kernel weight L1 norm-based pruning method is static and does not require a reference sample. In actual testing, the convolution kernel weight L1 norm-based pruning method without layer regularization has poor network compression effect, and the LRP-based pruning method does not require layer regularization from the perspective of interpretability. The two are mainly compared in terms of the loss of the network on the test set under the same pruning ratio, and the effect of the LRP-based pruning method is also concerned with the increase of the reference sample. In order to reduce randomness, the following results of the LRP-based pruning method are the average of five random results. The test results are as follows:

[0149] 1) from Figures 10-11 It can be seen that under the condition of a single reference sample, the LRP-based pruning method does not obtain better compression effect than the convolution kernel weight L1 norm-based pruning method under multiple pruning ratios.

[0150] 2) The test results based on the reference sample, the LRP reference sample amount (randomly selected) is increased to 5, 10 and 50 respectively, and the optimal scheme is found among them, and the results are shown in Figures 12-13 , 14-15, 16-17.

[0151] It can be seen that with the increase of the reference sample, the LRP-based pruning method obtains better compression effect than the convolution kernel weight L1 norm-based pruning method on the real part sub-network and the imaginary part sub-network, and the loss of the compressed network on the test set is smaller. The LRP-based pruning method provides interpretable pruning (correlation with network output) without the need for normalization of the correlation of each layer of convolution kernel, reduces the pruning process and speeds up the speed.

[0152] The present application uses an LRP-based structured pruning method to test the forward network for fast prediction of the frequency response of the proposed coding metasurface at different pruning ratios to obtain a balance between network parameter quantity, inference speed and network loss. The real part sub-network and the imaginary part sub-network are pruned at the same ratio. After pruning, a physical constraint term is added, and the accuracy is restored by retraining for 20 rounds on the training set, and the learning rate of the retrained network is set to 0.00003. The stronger the prediction ability of the forward network for the amplitude and phase representation of the metasurface frequency response S11, the smaller the physical constraint term; otherwise, it is larger. The original network parameter quantity is 109M, the total loss of the test set is 0.013625, the real part loss of the test set is 0.001787, the imaginary part loss of the test set is 0.002021, and the inference time for a test set of 4000 samples is 15 seconds (GPU environment).

[0153] The present application obtains a balance between loss and network parameter quantity and inference time by increasing the pruning ratio, as shown in Table 1. The total loss in the table is the test set loss after the network is pruned and retrained for 20 rounds on the training set. The total loss can be divided into amplitude loss, phase loss, real part loss and imaginary part loss, which represent the prediction ability of the compressed and retrained forward network for the amplitude representation, phase representation, real part representation and imaginary part representation of the metasurface frequency response S11. The inference time is the inference time of the compressed network for a test set of 4000 samples, and the batch_size is 128.

[0154] Table 1, network test set loss, parameter quantity, test set inference time under different pruning ratios

[0155]

[0156]

[0157] As can be seen from Table 1, the use of the LRP-based pruning method to compress the metasurface forward prediction network greatly reduces the parameter quantity of the forward prediction network. When the pruning ratio is 0.5, the total loss of the network after model compression and retraining for 20 rounds on the test set is even lower than that of the original network, the amplitude loss is reduced by 14.7%, the phase loss is reduced by 3%, the real part loss is increased by 22.5%, and the imaginary part loss is increased by 4%, but the parameter quantity is reduced by 64%, and the inference time for the entire test set is reduced from 15 seconds to 5 seconds. The prediction result of the network after model compression and retraining for 20 rounds for the metasurface frequency response S11 (real part representation and imaginary part representation) has a small error. The decrease in amplitude loss indicates that the prediction ability of the network for the amplitude representation of the metasurface frequency response S11 is improved, and the decrease in phase loss indicates that the prediction ability of the network for the phase representation of the metasurface frequency response S11 is improved.

[0158] From the above results, it can be seen that the pruning method based on LRP can compress the metasurface forward prediction network, and can greatly reduce the parameter quantity of the metasurface forward prediction network and obviously improve the inference speed while ensuring that the performance (the prediction ability for the frequency response S11) of the metasurface forward prediction network is basically unchanged.

[0159] The present application is not limited to the details of the foregoing exemplary embodiments, but can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the protective scope of the present application should be determined by the scope of the claims.

Claims

1. A model compression method based on fast prediction of the frequency response of coded metasurfaces, characterized in that, Includes the following steps: Step 1: Using CST-Python co-simulation, import the encoding matrix representing the surface structure of the encoded metasurface unit, generate the frequency response of the encoded metasurface, obtain the frequency response curve, and create a dataset; Step 2: Pre-train the forward network for fast prediction of the frequency response of the encoded metasurface to realize the mapping from the encoded metasurface to the frequency response; Step 3: Based on the pre-trained and complete forward network obtained in Step 2, use LRP to backpropagate the network output as the correlation, use the correlation obtained by the convolution kernel as the metric for structured pruning to compress the forward network, and then retrain the compressed forward network. Step 4: Input the encoding matrix representing the surface structure of the encoded metasurface unit into the forward network obtained in Step 3 to obtain the real part curve and imaginary part curve of the metasurface frequency response parameters, and then obtain the amplitude and phase values. In step three, the steps of the structured pruning method are as follows: Step A: Calculate the activation value; In the training set, a coded metasurface unit is randomly selected as a reference sample. The encoding matrix characterizing the surface structure of the reference sample metamaterial unit is input into the pre-trained and fully trained forward network in step two. The real part subnetwork and the imaginary part subnetwork of the forward network are independently propagated forward to obtain the output value of each layer in the real part subnetwork and the imaginary part subnetwork. The output value is the activation value. The pre-activation value is the output value that has not passed through the activation function in the layer. Step B: Correlation calculation based on LRP; The output of the pre-trained forward network in step two is backpropagated to the input layer. The real and imaginary subnetworks of the forward network independently backpropagate their outputs to the network input layer. Based on the LRP rule, the correlation between the convolutional kernels and the network output in each layer of the real and imaginary subnetworks is obtained sequentially. Step C: Calculate the correlation of each convolutional kernel; Based on the LRP rule, after the network output is backpropagated to the input, the activation values ​​of all layers in the network obtain the correlation with the network output. For convolutional layers, the correlation of the convolution kernel is the sum of the correlations of all activation values ​​output by that convolution kernel. Step D: Based on the correlation of convolution kernels, perform structured pruning using an overall pruning approach; Pruning is performed separately on the real and imaginary subnetworks of the pre-trained forward network in step two. The pruning ratio is determined, i.e., the proportion of pruned convolutional kernels to all convolutional kernels. All convolutional kernels are sorted according to their correlation to obtain a correlation threshold. Convolutional kernels with correlation below the threshold and their corresponding input-output connections are pruned to obtain the compressed forward network.

2. The model compression method based on fast prediction of coded metasurface frequency response according to claim 1, characterized in that, In step one, the operating frequency of the coded metasurface unit is 8GHz-12GHz; the coded metasurface unit includes three layers: the bottom layer is a copper plate with a conductivity of 5.8e+007 S / m; the middle layer is a dielectric substrate made of F4B material with a dielectric constant of 2.65, a length and width of 10mm, and a thickness of 2mm; the top layer is a metal patch layer with 16*16 units, symmetrically formed by 8*8 units. Each unit uses 1 or 0 to indicate the presence or absence of the patch. The length and width of a single patch are 0.5mm, and the thickness is 0.017mm.

3. The model compression method based on fast prediction of coded metasurface frequency response according to claim 2, characterized in that, In step one, the generated metasurface frequency response is denoted as S11. The matrix representing the metasurface frequency response S11 is sampled at intervals to obtain the correspondence between the metasurface unit and its frequency response, forming several initial data sets. The initial data sets are divided into training set, validation set, and test set in a ratio of 8:1:

1. The training set is then expanded to twice its original size through data augmentation. In the parameter characterization of the metasurface frequency response S11, the amplitude Phase , real part virtual part The correspondence is as follows: ； ； ； 。 4. The model compression method based on fast prediction of coded metasurface frequency response according to claim 3, characterized in that, The forward network for fast prediction of the metasurface frequency response in step two includes a real part subnetwork. Virtual subnetwork The two subnetworks are in parallel and joint form, and both subnetworks adopt the same structure. The input of the two subnetworks is the encoding matrix representing the surface structure of the encoded metasurface unit. The outputs of the two subnetworks correspond to the real and imaginary parts of the metasurface frequency response S11, respectively. Based on the correspondence between the amplitude and phase and the real and imaginary parts, the amplitude and phase of the network output are calculated. When pre-training a forward network: 1) The error function used by the network is the minimum mean square error (MSE), expressed as: ； in, y represents the prediction result of the forward network for the frequency response S11 of the coded metasurface, y represents the actual frequency response of the coded metasurface, and N represents the number of frequency points obtained by sampling the frequency response S11 of the coded metasurface at equal intervals in the frequency domain. 2) The network loss function during pre-training is: ； in, The loss function during network pre-training. , These are the actual real values ​​of the S11 parameters and the real values ​​predicted by the real subnetwork, respectively. , These are the true imaginary part values ​​of the S11 parameters and the imaginary part values ​​predicted by the imaginary subnetwork, respectively.

5. The model compression method based on fast prediction of coded metasurface frequency response according to claim 4, characterized in that, The specific steps of step B are as follows: Step 1: The correlation between the convolutional kernels of the preceding and following layers is represented by the ratio of the pre-activation values ​​associated with each kernel in that group to the total pre-activation values ​​of all associated preceding kernels in the following layer. Therefore, the propagation formula for the correlation is: ； ； in, The correlation of the j-th pre-activation value in the (L+1)-th layer; The correlation between the j-th pre-activation value of layer L+1 and the ith activation value of layer L; This represents the sum of the preactivation values ​​of the j-th convolutional kernel in layer L+1 and all associated convolutional kernels in layer L; The pre-activation value of the (L+1)th convolutional kernel associated with the i-th convolutional kernel in layer L; This is the i-th activation value in the L-th layer; The weight between the i-th activation value in layer L and the j-th pre-activation value in layer L+1; Step 2, considering When the value is small, it may produce uncontrolled values. Therefore, a pre-set number ε greater than 0 is added. The original correlation propagation formula is expressed as: ； Step 3: Since correlations are absorbed by ε during propagation, the sum of correlations obtained by convolutional layers closer to the input is smaller. This makes correlation comparisons between different convolutional layers meaningless. Therefore, the correlation propagation formula is expressed as: ； Where α+β=1 is required, the pre-activation value is divided into two parts: one greater than 0 and one less than 0. , '+' indicates a value greater than 0, and '-' indicates a value less than 0, effectively avoiding the problem of outliers; Taking α=1 and β=0, and considering that the activation values ​​of all layers except the output layer are not less than 0, the correlation propagation formula becomes: ； Step 4: During backpropagation, set the network output to 1 as the starting point for correlation propagation, and only consider pre-activation values ​​that are not less than 0 during the propagation process, so that the correlation of all neurons in the network is not less than 0. From the propagation rules when α=1 and β=0, we know that without considering the bias terms of convolutional layers or fully connected layers, the sum of the correlations between different layers is equal, that is, it satisfies: ； Where f(x) is the network output, and V(L) represents the number of activation values ​​in the Lth layer. This represents the correlation of the d-th activation value in layer L; when considering the bias terms of convolutional layers or fully connected layers, a small portion of the correlation is absorbed by the bias terms during network propagation.

6. The model compression method based on fast prediction of coded metasurface frequency response according to claim 5, characterized in that, In step D, physical constraints are added to the compressed forward network, and the performance of the forward network is restored through retraining. 1) The network loss function for retraining the forward network is: ； ； in, This is the loss function for retraining the network after pruning; For hyperparameters; , These are the actual real values ​​of the S11 parameters and the real values ​​predicted by the real subnetwork, respectively. , These are the actual imaginary part values ​​of the S11 parameters and the imaginary part values ​​predicted by the imaginary part subnetwork, respectively. , These are the actual amplitude value and the predicted amplitude value of parameter S11, respectively. , These are the true phase value, predicted phase value, and predicted amplitude value of the S11 parameter, respectively. and phase prediction value can be and Calculated; In the loss function For physical constraints; 2) Except for the network output layer, the activation function of the remaining layers in the network is ReLU. The ReLU function expression is: ； Since the real and imaginary parts of the encoded metasurface frequency response both range from [-1, 1], the activation function of the output layer of the two sub-networks is tanh: ； During training, the Adam optimizer was selected, the number of training rounds was set to 120, the initial learning rate was 1e-3, and the learning rate was halved every 40 training rounds.

Images