Target detection model lightweight method for satellite-borne edge calculation

Abstract

The invention provides a satellite-borne edge calculation-oriented target detection model lightweight method. The method comprises the following steps of S1, training an image target detection model capable of identifying a feature missing target by collecting a training data set, performing self-supervised contrast training and constructing a channel attention mechanism; s2, abstracting a neural network model by using a graph structure for the target detection model, and identifying and trimming redundant channels so as to compress the neural network model to obtain a compressed model; and S3, performing model performance optimization on the compression model by using a comparative generative adversarial network and a knowledge distillation method to obtain a lightweight target detection model. According to the method, under the condition that the compression model realizes lightweight deployment of the satellite-borne edge computing environment, the detection speed and precision are ensured, and the recognition capability of a feature missing target is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a lightweight method for target detection models for satellite-borne edge computing. Background Art

[0002] Satellite image target detection uses images taken by low-orbit remote sensing satellites as data sources and uses efficient image processing technology to locate and classify valuable targets in the images. Due to the limited computing power of a single satellite, satellite-borne edge computing technology is widely used in satellite image target detection. However, with the continuous increase in the payload of current satellites, the amount of data generated by satellites is also increasing significantly, and the efficiency of satellite-borne edge computing is constantly decreasing. In order to efficiently identify targets in large amounts of satellite images and realize high-efficiency satellite-borne edge computing, a lightweight target detection model method for satellite-borne edge computing is urgently needed to simplify the current satellite image target detection process.

[0003] In recent years, with the continuous development of deep learning, models represented by convolutional neural networks have shown excellent performance in image processing, providing an opportunity for accurate and efficient onboard intelligent edge computing. At present, target detection models based on deep learning can be divided into two categories: one-stage models and two-stage models. The two-stage target detection model first uses a selective search algorithm to search for possible locations of the target, then uses a convolutional neural network to extract features from these candidate regions, and finally inputs them into a classifier to mark the target. The one-stage target detection model combines the feature extraction process with the target detection process, improves the search process of the candidate region through a region proposal network, integrates the region proposal, feature extraction, and target box regression process into a unified network framework, and realizes end-to-end target detection. In order to deploy the model in a space-based edge computing environment with limited computing resources, domestic and foreign scholars use pruning algorithms to compress the model. Model pruning algorithms are divided into structured pruning and unstructured pruning. Structured pruning technology compresses the model without changing the model structure. It calculates the importance of each neuron for each layer of the neural network, and resets the weights of unimportant neurons to 0 based on the threshold, thereby compressing the model. Unstructured pruning methods involve learning channel sparsity in deep neural networks to identify channels that need to be pruned, change the structure of the model, and then achieve model compression. To compensate for the accuracy loss during model compression, existing solutions use knowledge distillation algorithms to transfer knowledge from the complex network before pruning to the compressed model after pruning, and then fine-tune the performance of the compressed model.

[0004] Although the target detection model based on the deep neural network algorithm has greatly improved the accuracy compared with the traditional detector, there are a lot of redundant calculations in the calculation process, which often requires a lot of storage space and high-performance computing units. Considering that the satellites involved in onboard edge computing are small in size, and the computing resources, storage units and payload are limited, it is impossible to deploy high-precision target detection models on satellites. Therefore, under the premise of maintaining the detection accuracy, how to achieve the lightweight model and efficient detection, so that it can also run stably in the resource-constrained onboard edge computing platform environment, has become the key to the onboard intelligent detection of satellite images. Although the model compression scheme based on structured pruning can remove some neurons with low contribution in each layer of the neural network, it does not effectively compress the redundant structure of the model and cannot effectively reduce the computing load. At present, although the model compression scheme based on unstructured pruning analyzes the contribution of neurons in the dimension of channels, it cannot effectively identify redundant channels and neurons, and cannot effectively compress the model to improve efficiency. In order to optimize the performance of the compressed model, the current scheme uses the knowledge distillation algorithm to fine-tune the model. The knowledge distillation algorithm can enable the compressed model to effectively learn the hard label and soft label knowledge of the classification, but the current knowledge distillation scheme makes it difficult for the unstructured pruned model to learn the non-critical feature knowledge of the original model. This defect makes the unstructured pruned model unable to effectively distinguish the categories of targets with incomplete features. In the application scenarios of spaceborne computing, many targets in images are deformed and displaced, and even the features of some targets are incomplete. Existing recognition schemes based on rules and complete target feature training are difficult to efficiently identify targets in such situations. Summary of the invention

[0005] In view of this, the purpose of the present invention is to propose a lightweight target detection model for satellite-based edge computing, solve the problem that the compressed model is difficult to identify targets with missing features, and improve the accuracy of target detection. The present invention uses a graph structure to represent neurons and constructs a redundant channel pruning model to solve the problem that redundant channels in deep neural network models are difficult to remove. The present invention constructs a compression model performance enhancement model based on the comparison of generative adversarial networks and knowledge distillation algorithms, trains the compression model's ability to recognize small sample features, and improves the model's target detection performance.

[0006] The technical means adopted by the present invention are as follows: A lightweight method for target detection model for space-borne edge computing includes the following steps: S1. Train an image object detection model that can identify feature-missing objects by collecting training data sets, self-supervised contrast training, and building a channel attention mechanism; S2. Use the graph structure to abstract the target detection model into a neural network structure graph, identify and prune redundant channels to compress the neural network model, and obtain a compressed model. S3. Use contrastive generative adversarial networks and knowledge distillation methods to optimize the performance of the compression model and obtain a lightweight target detection model.

[0007] Furthermore, S1 specifically includes the following steps: S11, collecting original images taken by the satellite, screening out images with the target to be identified from the original images, cropping the images of the target to be identified, and using the cropped images as a pre-detection data set; S12, inputting the original image into the contrastive learning framework for pre-detection, and outputting the category of each image; S13. Based on the category of each image, an image target detection model is built based on the YOLOX target detection framework, wherein the structure of the image target detection model includes a backbone network, a path aggregation network PAN and a YOLOX decoupling head network; The original image is input into the backbone network. The basic structure of the backbone network includes five different stages from C1 to C5. Each stage can extract the features of the input image and output feature maps of different sizes. The feature maps output by C3 to C5 are input into the PAN network. The feature map output by C5 is first The convolution adjusts the number of channels to C, and after upsampling, it is spliced ​​with the feature map output by C4 in the channel dimension; the channel segmentation and aggregation module is used to suppress the background part of the image and highlight the foreground part; the channel segmentation and aggregation module first divides the input feature map in the channel dimension to obtain two sizes of feature map, one part of the feature map is directly concatenated with the other part after two depth-wise convolutions; a channel attention module is introduced between the two depth-wise convolutions; the feature map concatenated according to the channel dimension is again concatenated Convolution is used to fuse information between different channels. For the bottom-up part of the PAN network, M3 and M4 are downsampled by 2 times and then concatenated with M4 and M5 respectively. After passing through the CSA module again, the outputs P3, P4, and P5 are obtained. The fused feature maps of different sizes are input into the YOLOX decoupling head network to classify and locate the objects contained therein. In the channel attention module, the input feature map first aggregates the global spatial features through global average pooling, and then completes the information interaction between different channels through group convolution. The number of groups can be freely selected according to the number of input channels, and then multiplied with the input after passing the Sigmoid activation function; The P3, P4, and P5 feature maps are respectively input into the YOLOX decoupling head network. The decoupling head network extracts the target position and category information respectively, and learns them separately through different convolutional neural network branches, and finally performs feature fusion. An additional angle parameter is introduced in the regression branch to convert the anchor box coordinates from (center point, width, height) to (center point, width, height, rotation angle) to achieve the prediction of rotated targets.

[0008] Furthermore, S12 specifically includes the following steps: S121. Use data enhancement method to transform the input original image to obtain two similar images And pass through the encoder Encoder to output the feature map , the encoder architecture is consistent and shares parameters; S122, feature map The feature transformation is obtained by mapping the head ; S123, by predicting the head pair Make a prediction ,calculate and The cosine similarity of .

[0009] Furthermore, the contrastive learning framework in S12 includes a data augmentation network structure, an encoder, a projection network structure, and a prediction network structure; The data enhancement network structure Augmentation is used to expand the data set and transform the input image using methods such as random cropping and horizontal flipping to obtain two similar images. And input them into the encoder respectively; The encoder uses the ResNet50 network model; The Projection network structure consists of a fully connected layer, a BN layer, and a ReLu activation function. The correct classification of the image is achieved by fusing the output features of the encoder. The two Projections have the same architecture. The prediction network structure consists of a fully connected layer, a BN layer, and a ReLu activation function. By further transforming the output of the Projection network, its output is compared with the output of another Projection network to calculate the cosine similarity, which is used as the loss to train the encoder through back propagation.

[0010] Furthermore, S2 specifically includes the following steps: S21, abstracting the neurons of the image target detection model as nodes, abstracting the transmission relationship between neurons as edges, and constructing a neural network structure diagram with several convolutional layers, wherein the neural network structure diagram is stored in the form of a matrix; S22, based on the importance and feature map of the convolution kernel itself, pruning the convolution kernel in the convolution layer of the image object detection model; S23. The connections between neurons in different layers are channels. The similarity of the channels is calculated, and the redundant channels in the image target detection model are pruned based on the similarity of the channels.

[0011] Furthermore, S22 specifically includes the following steps: S221. From the perspective of the importance of the convolution kernel itself, the convolution kernel in the convolution layer is directly pruned; for a Layer convolutional layer network, suppose we want to Convolutional Layer To trim, Represents the total number of convolution kernel, that is output channels, and the number of input channels of each convolution kernel is , the size is ; In this branch, each convolution kernel in this layer is first calculated The L1 norm of , then sort the norms of these convolution kernels from large to small, and according to the pre-set pruning rate , after removing the soft pruning method The convolution kernels are arranged from large to small from front to back; the soft pruning method is to set the convolution kernel to be pruned to 0; S222. From the perspective of feature graphs, the network is further pruned by judging the correlation between convolution kernels; for a The network at layer In the layer, Represents the total number of feature map, Represent the height and width of the feature map respectively, that is, the size of each feature map is ; Calculate the importance of each feature map, sort them in descending order of importance, and remove the most important ones. % of the feature map is cut off, and the corresponding convolution kernel is also cut off. The calculation formula for calculating the importance of each feature map is as follows:

[0012] in, Represents the importance of each feature map, That is, convert the three-dimensional feature map into the corresponding feature matrix after two-dimensional, represents the mask matrix. The first row is 0, and the rest of the rows are 1. Represents the SVD decomposition of the matrix, and the formula is as follows:

[0013] in, represents the singular values ​​of the matrix, are the left and right singular value vectors respectively, the matrix The rank of ; Change the eigenvalue Arrange from large to small and remove the ones with the largest change. % feature map, and the corresponding filters are also cut off accordingly.

[0014] Further, S23 includes the following steps: The structural similarity of the channel is calculated based on the interaction relationship between neurons. The calculation formula for the interaction relationship between neurons is:

[0015]

[0016] in, For the node contribution, is the node forward contribution, is the backward contribution of the node, is the forward node set, is the backward node set; Based on the calculated contribution of each node, the similarity between channels is calculated. The formula for calculating the similarity of channels is:

[0017]

[0018]

[0019] in, It is a channel With channel The similarity of is the structural similarity, Contribute similarity to nodes, and Calculate the channel from the input and output and the hierarchical structure With channel The structural similarity of For the Layer The node contribution of each node; Arrange the channel overlap from high to low, and put the overlap at the end. The channels with only a single branch are identified as redundant channels and pruned from the network. For those channels that have only a single branch selected for removal, their parameters are restored in the subsequent network iteration update process to participate in the next training process. The joint dynamic channel pruning algorithm is iterative until the pruning rate of the network reaches the target pruning rate. Stop when.

[0020] Furthermore, S3 specifically includes the following steps: S31, build contrastive generative adversarial network; S32, Generate feature-enhanced datasets using contrastive generative adversarial networks; S33. Knowledge distillation: Fine-tuning compressed models using feature-augmented datasets.

[0021] Further, in S31, the contrastive generative adversarial network uses the target detection model as a discriminator, and the optimization function formula of the contrastive generative adversarial network is:

[0022] in, is the distribution of real data, is the noise distribution, and are the outputs of the discriminator and the generator respectively; the loss functions of the generator and the discriminator are:

[0023]

[0024] in, is the traditional discriminator loss function, To generate the similarity between the sample and the original sample, For expected similarity, the similarity between the generated sample and the original sample is expected to be , by adjusting The feature richness of generated samples can be changed. and are the generation loss of similar data and the generation loss of heterogeneous data, respectively. A variable parameter.

[0025] Furthermore, S33 specifically includes the following steps: The feature-enhanced image is input into the original network teacher net and the pruned lightweight network student net respectively, and the classification probability is output after softmax. , The definition is as follows:

[0026] in, Output results for the network model, For each category of output, is the total number of categories; represents the temperature hyperparameter, which is used to control the probability distribution of the model output; The larger the value, the smoother the output probability distribution. The smaller the value, the more concentrated the probability distribution of the output; The output of the original network teacher net is used as the soft target, and the cross entropy loss is calculated by combining the soft target with the output of the student net. At the same time, the real label of the satellite image is used The cross entropy loss is calculated as the output of the hard target and the student net, and the parameters of the student net are updated through the back propagation algorithm. The calculation formula of the back propagation algorithm is as follows:

[0027] in, represents the total loss function of the distillation part, , represent hard distillation loss and soft distillation loss respectively, Represents the weight parameter.

[0028] Compared with the prior art, the present invention has the following advantages: In order to solve the problem that the model has difficulty in identifying targets with missing features, the present invention proposes a method for constructing a target detection model based on multi-scale feature fusion. In order to alleviate the problem of reduced detection accuracy and recall rate after model compression, the present method effectively aggregates multi-scale image features. The scheme introduces a channel attention mechanism based on the feature aggregation network and proposes a channel information aggregation scheme to help the model better aggregate shallow and deep semantic features and avoid background interference in target recognition tasks. Considering that there are a large number of rotating targets in satellite images, the present method adds additional angle prediction parameters in the detection head to predict rotating targets.

[0029] In the process of compressing the model, in order to remove redundant channels, the present invention proposes a neural network model compression method based on channel pruning. The model first uses neurons to abstract as nodes and uses a graph to represent the model structure, which can better analyze the neural network model. Then, the method identifies the channels contained in the network model based on the correlation between the neuron and the previous and next neurons. Finally, the method calculates the similarity of the channels based on the sample pairs, thereby removing redundant channels and compressing the complex model.

[0030] In the process of optimizing model performance, this paper proposes a model performance optimization method based on data enhancement to address the problem that small sample features are difficult to identify. This method uses a contrastive generative adversarial network to generate an enhanced sample set whose similarity to the original sample is within a threshold range, and uses it to train the knowledge distillation of the pre-compression model and the post-compression model. The contrastive generative network in this scheme uses the pre-compression model as a discriminator to judge the similarity between the generated sample and the original sample. By generating a sample set to optimize model performance, this method can effectively recover the accuracy loss caused by model compression. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 The figure is a flow chart of the method of the present invention.

[0033] Figure 2 Schematic diagram of the pre-training framework based on contrastive learning of the present invention.

[0034] Figure 3 This is a schematic diagram of the Projection network structure of the present invention.

[0035] Figure 4 This is a schematic diagram of the Prediction network structure of the present invention.

[0036] Figure 5 This is a target detection framework diagram of the present invention.

[0037] Figure 6 This is a structural diagram of PAN of the present invention.

[0038] Figure 7 This is a structural diagram of the CSA of the present invention.

[0039] Figure 8 This is a structural diagram of the CAM module of the present invention.

[0040] Fig. 9 This is a schematic diagram of the rotation target regression parameters of the present invention.

[0041] Fig.10 Schematic diagram of the joint channel pruning algorithm of the present invention.

[0042] Fig.11 Schematic diagram of the redundant channel pruning method of the present invention.

[0043] Fig.12 Schematic diagram of the model performance enhancement method of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] The present invention provides a lightweight target detection model for satellite-based edge computing. The method trains an image target detection model that can identify target with missing features by collecting training data sets, self-supervised comparative training, and building a channel attention mechanism. In the neural network model compression stage, the present invention proposes a neural network model compression method based on channel pruning, such as Figure 1 As shown. This method uses a graph structure to abstract the neural network model, identify and prune redundant channels, and then generate a more compact model. In the model optimization stage, the present invention proposes a model training method based on data enhancement. This method uses a contrastive generative adversarial network to generate an image feature enhanced sample set, and uses a knowledge distillation method to enable the compact model to learn the target recognition ability of the model before compression. Specifically, it includes the following steps: S1. Train an image object detection model that can identify feature-missing objects by collecting training data sets, self-supervised contrast training, and building a channel attention mechanism; S2, using a graph structure to abstract the neural network model for the target detection model, identifying and pruning redundant channels to compress the neural network model and obtain a compressed model; S3. Use contrastive generative adversarial networks and knowledge distillation methods to optimize the performance of the compression model and obtain a lightweight target detection model.

[0047] The specific steps of S1 are as follows: Collect image datasets. Filter out images with the target to be identified from the original images, and crop the resulting images to 224×224 size. When cropping, pay attention to ensuring the integrity of the target in the image to avoid damaging it and affecting the model training effect. Finally, organize and store them uniformly and use them as a pre-training dataset for self-supervised contrastive learning.

[0048] Input the collected original image into Figure 2 The contrastive learning framework shown in the figure is used for pre-training. First, the contrastive learning framework uses the data augmentation network structure Augmentation to expand the data set. Augmentation transforms the input image using random cropping, horizontal flipping and other methods to obtain two similar images and input them into the encoder respectively. The encoder architecture of the image input is consistent and shares parameters. Then, the output feature map is The feature transformation is obtained by mapping the head Finally, by predicting the head pair Make a prediction , calculate its Cosine similarity. The encoder uses the ResNet50 network model. After the image passes through the network to extract features, the final output size is 7×7×2048.

[0049] Projection network structure: Projection consists of a fully connected layer, a BN layer, and a ReLu activation function, and its input is 7×7×2048. By fusing the output features of the encoder, the correct classification of the image can be achieved. Similar to the encoder, the two projections have the same architecture. Its network structure is as follows Figure 3 shown.

[0050] Prediction network structure: The prediction head is similar to the Projection network structure, and is also composed of a fully connected layer, a BN layer, and a ReLu activation function. By further transforming the output of the Projection network, the cosine similarity between its output and the output of another Projection network is calculated, and this is used as the loss to train the encoder through back propagation. Its network structure is as follows Figure 4 shown.

[0051] Construct a target detection model. Usually, the resolution of satellite images is relatively large, and there are a large number of small targets with low resolution and difficult to identify, and the features of some targets are incomplete. In order to accurately identify these targets, the present invention builds the entire detection model based on the YOLOX target detection framework. In order to better identify small target objects in satellite images, the present invention introduces a bottom-up pathway based on the feature aggregation network FPN (Feature pyramid network, FPN) used by YOLOX to form a path aggregation network PAN (Path aggregation network, PAN), so that shallow information and deep semantic information are better integrated. In addition, an additional channel attention mechanism is introduced in the feature aggregation process to further improve the accuracy and robustness of the detection model. Finally, the rotating target in the needle-sensing image is added to the detection head. A detection box for the rotating target is added to the detection head to better identify the rotating target in the image. The target detection model consists of a backbone network, a path aggregation network PAN and a YOLOX decoupled head network. Its overall structure is as follows Figure 5 shown.

[0052] First, the image and the category label output by the self-supervised contrastive learning framework are input into the backbone network. For each batch of input images, they are first scaled to The Backbone network structure includes a downsampling layer and 5 different stages, such as Figure 5 As shown on the left. Each stage can extract the features of the input image and output feature maps of different sizes. Considering that shallow feature maps are more conducive to identifying small targets, and deep feature maps have a larger receptive field range, they are better for identifying large targets. Therefore, the present invention inputs the feature maps output by C3 to C5 into the PAN network, and through the fusion of feature information between different layers, helps the model better identify targets of different sizes in satellite images. The basic specific structure diagram of the PAN network is shown in Figure 6 shown.

[0053] Specifically, the feature map output by C5 is first The number of channels is adjusted by convolution to C, and after upsampling, it is spliced ​​with the feature map output by C4 in the channel dimension. In addition, in view of the large number of targets in satellite images and the small targets that are difficult to identify, the present invention introduces a channel separation aggregation (CSA) module to suppress the background part of the image and highlight the foreground part. The CSA model structure diagram is shown in Figure 7As shown. This module first divides the input feature map in the channel dimension to obtain two sizes of The feature map of one part is directly spliced ​​with the other part after two depth-wise convolutions, so as to reduce the amount of calculation and improve the detection speed. At the same time, the present invention introduces a channel attention module (CAM) between the two depth-wise convolutions to make the model pay more attention to the channels containing more important information. Finally, the feature map spliced ​​by channel dimension is passed through Convolution is used to fuse the information between different channels. The information fusion process of C4 and C3 is the same as above. For the bottom-up part of the PAN network, the whole process is similar to the above. M3 and M4 are downsampled by 2 times and then spliced ​​with M4 and M5 respectively. After passing through the CSA module again, the outputs P3, P4, and P5 are obtained. Finally, these fused feature maps of different sizes are input into the subsequent detection head to classify and locate the targets contained therein.

[0054] The CAM module introduced in the present invention is as follows Figure 8 As shown in the figure. The input feature map first aggregates the global spatial features through global average pooling, and then completes the information interaction between different channels through group convolution. The number of groups can be freely selected according to the number of input channels. Compared with the use of fully connected layers, the use of group convolution can effectively reduce the amount of calculation and avoid the information loss caused by dimensionality reduction. Finally, after passing the Sigmoid activation function, it is multiplied with the input, so that the model pays more attention to the channels containing important information.

[0055] Finally, the P3, P4, and P5 feature maps are respectively input into the YOLOX decoupling head network, whose structure is as follows: Figure 5 As shown on the right. Since YOLOX is an anchor-free framework, there is no need to set the shape of the anchor box in advance. In addition, for rotating targets in satellite images, the present invention introduces additional angle parameters in the regression branch to convert the anchor box coordinates from (center point, width, height) to (center point, width, height, rotation angle) to achieve the prediction of rotating targets. The schematic diagram is shown in Fig. 9 shown.

[0056] The specific steps of S2 are as follows: In order to enable the network to be deployed in a satellite environment, the present invention proposes a neural network model compression method based on channel pruning to prune the pre-trained network model. By continuously removing those "unimportant" convolution kernels during the training process, the number of network parameters is reduced, so that the model can be deployed in an on-orbit satellite. This method is divided into three stages: constructing a neural network structure diagram, neuron pruning, and channel pruning.

[0057] In the first stage, the present invention abstracts the neurons of the neural network as nodes, abstracts the transmission relationship between neurons as edges, and constructs a neural network structure diagram. The structure diagram is stored in the form of a matrix, represented as ,in is the depth of the neural network, is the maximum number of neurons in each layer.

[0058] In the second stage, the present invention prunes the convolution kernels in the convolution layer from the perspective of the importance of the convolution kernels themselves and the perspective of the feature map. The schematic diagram of the method in this stage is shown in Fig.10 shown.

[0059] First, the present invention directly prunes the convolution kernels in the convolution layer from the perspective of the importance of the convolution kernels themselves. Layer convolutional layer network, suppose we want to Convolutional Layer To trim, Represents the total number of convolution kernel, that is output channels. And the number of input channels of each convolution kernel is , the size is In this branch, each convolution kernel in this layer is first calculated The L1 norm of After that, the norms of these convolution kernels are sorted from large to small, and the pruning rate is set according to the preset Here, the present invention adopts the idea of ​​soft pruning, by setting the convolution kernel to be pruned to 0 instead of actually removing the filter from the network, so as to keep the entire network model structure unchanged, thereby minimizing the impact of different inputs on the pruning results.

[0060] Then, the present invention further prunes the network from the perspective of feature graphs by judging the correlation between convolution kernels. The network at layer In the layer, Represents the total number of A feature map. Represent the height and width of the feature map respectively, that is, the size of each feature map is By calculating the importance of each feature map, the relatively unimportant feature maps are cut off, that is, the corresponding convolution kernels are cut off. The calculation formula is as follows:

[0061] in Represents the importance of each feature map, That is, the feature matrix corresponding to the three-dimensional feature map is converted into two-dimensional. represents the mask matrix. row is 0, and the rest are 1. Represents the SVD decomposition of the matrix, and the formula is as follows:

[0062] in represents the singular values ​​of the matrix, are the left and right singular value vectors respectively. It can be seen that the matrix The rank of According to the formula, when the mask matrix Will After the row is set to 0, re- Perform SVD decomposition and get The rank of By observing the change in rank after removing the feature map, we can get Considering that the smaller the rank, the smaller the amount of information contained in the feature map, The larger the value, the more important the feature map is. , the present invention removes those The feature graph with a small value change has its corresponding filter cut off accordingly. Here, the present invention also sets the corresponding filter to 0 instead of actually removing it from the network to ensure the recoverability of the network.

[0063] Finally, the present invention simultaneously selects the channels to be removed and prunes them from the network, and for those channels that only have a single branch selected for removal, their parameters are restored in the subsequent network iteration update process to participate in the next training process. It should be noted that the joint dynamic channel pruning algorithm is iterative until the pruning rate of the network reaches the target pruning rate. Therefore, the network parameter model can be compressed to the target size.

[0064] In the third stage, the present invention further prunes redundant channels by calculating the similarity of the channels. The schematic diagram of the method in this stage is shown in FIG. Fig.11 To compare the similarity of channels, the present invention calculates the structural similarity of channels based on the interaction relationship between neurons. The calculation formula for the interaction relationship of neurons is:

[0065]

[0066] in, For the node contribution, is the node forward contribution, is the backward contribution of the node, is the forward node set, is the backward node set.

[0067] Based on the calculated contribution of each node, the similarity between channels can be calculated. The formula for calculating the similarity of channels is:

[0068]

[0069]

[0070] in, It is a channel With channel The similarity of is the structural similarity, Contribute similarity to the node. and Calculate the channel from the input and output and the hierarchical structure With channel 's structural similarity. For the Layer The node contribution of each node.

[0071] Finally, the present invention can effectively identify the overlap degree of each channel based on the calculated similarity of each channel, and remove redundant channels with a high overlap degree.

[0072] The specific steps of S3 are as follows: In S2, the heavy target detection model is effectively compressed into a compact model. However, some neurons and channels are pruned, the target detection accuracy of the model decreases, and the model loses the ability to recognize some small sample features. In order to improve the detection accuracy, the present invention proposes a model performance enhancement method based on generative adversarial networks and knowledge distillation. The method schematic diagram is shown in Fig.12 This method includes three stages: building a contrastive generative adversarial network, generating feature-enhanced datasets, and knowledge distillation.

[0073] The present invention proposes a comparative generative adversarial network model. In order to enable the lightweight model to efficiently learn the ability to recognize small sample features, this model focuses on generating image data that is similar to the original satellite image but contains richer features. The model uses the target detection model trained in step 1 as the discriminator of the generative adversarial network, which can effectively judge the similarity between the generated image and the original image, and can also effectively accelerate model training. The optimization function formula of the generative adversarial network is:

[0074] in, is the distribution of real data, is the noise distribution, and are the outputs of the discriminator and the generator respectively. In this network, the loss functions of the generator and the discriminator are:

[0075]

[0076] in, is the traditional discriminator loss function, To generate the similarity between the sample and the original sample, is the expected similarity. The similarity between the generated sample and the original sample is expected to be , by adjusting The feature richness of generated samples can be varied. and They are the generation loss of similar data and the generation loss of heterogeneous data, respectively. A variable parameter.

[0077] In the knowledge distillation stage, the present invention uses the feature enhancement dataset to fine-tune the pruned network. First, the feature-enhanced image is input into the original network teacher net and the pruned lightweight network student net respectively, and the classification probability is output after softmax. , which is defined as follows.

[0078]

[0079] in Output results for the network model, For each category of output, is the total number of categories, Represents the temperature hyperparameter, which is used to control the probability distribution of the model output. The larger the value, the smoother the output probability distribution. The smaller the value, the more concentrated the output probability distribution is.

[0080] The present invention uses the output of the original network teacher net as a soft target and calculates the cross entropy loss with the output of the student net. At the same time, the real label of the satellite image is used , use it as the hard target and the output of the student net to calculate the cross entropy loss, and update the parameters of the student net through the back propagation algorithm. The calculation formula is as follows:

[0081] in, represents the total loss function of the distillation part, , represent hard distillation loss and soft distillation loss respectively, Represents the weight parameter.

[0082] Since the current satellite image target detection model has many parameters and is difficult to deploy on resource-constrained satellites for edge computing, this paper proposes a lightweight target detection model method for satellite-borne edge computing. This method ensures detection speed and accuracy and enhances the ability to identify targets with missing features under the condition that the compression model realizes lightweight deployment in the satellite-borne edge computing environment.

[0083] In the process of model construction, the present invention proposes a method for constructing a target detection model based on multi-scale feature fusion. This method can detect targets with deformation, displacement and incomplete features in satellite images, and effectively improve the accuracy and recall rate of target recognition and detection after model compression.

[0084] In the process of model compression, the present invention proposes a neural network model compression method based on channel pruning. The method identifies and removes redundant channels in the neural network model, reduces the number of model parameters, and improves the compression rate of the model. By focusing on pruning redundant model channels, the method can achieve a balance between detection accuracy and model parameter quantity.

[0085] In the process of model performance optimization, the present invention proposes a model performance optimization method based on data enhancement. This method uses an uncompressed model as a discriminator of a generative adversarial network to generate a feature-enhanced sample set, thereby improving the accuracy of the model in detecting feature-missing targets. This method applies a knowledge distillation algorithm to enable the compressed model to effectively learn feature recognition experience, effectively improving the accuracy of lightweight model target detection and making its performance close to that of the model before compression.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight target detection model method for space-borne edge computing, characterized in that: The steps include: S1. Train an image object detection model that can identify feature-missing objects by collecting training data sets, self-supervised contrast training, and building a channel attention mechanism; S2. Use the graph structure to abstract the target detection model into a neural network structure graph, identify and prune redundant channels to compress the neural network model, and obtain a compressed model. S3. Use contrastive generative adversarial networks and knowledge distillation methods to optimize the performance of the compression model and obtain a lightweight target detection model.

2. The method for lightweight target detection model for satellite-based edge computing according to claim 1, characterized in that: S1 specifically includes the following steps: S11, collecting original images taken by the satellite, screening out images with the target to be identified from the original images, cropping the images of the target to be identified, and using the cropped images as a pre-detection data set; S12, inputting the original image into the contrastive learning framework for pre-detection, and outputting the category of each image; S13. Based on the category of each image, an image target detection model is built based on the YOLOX target detection framework, wherein the structure of the image target detection model includes a backbone network, a path aggregation network PAN and a YOLOX decoupling head network; The original image is input into the backbone network. The basic structure of the backbone network includes five different stages from C1 to C5. Each stage can extract the features of the input image and output feature maps of different sizes. The feature maps output by C3 to C5 are input into the PAN network. The feature map output by C5 is first The convolution adjusts the number of channels to C, and after upsampling, it is spliced ​​with the feature map output by C4 in the channel dimension; the channel segmentation and aggregation module is used to suppress the background part of the image and highlight the foreground part; the channel segmentation and aggregation module first divides the input feature map in the channel dimension to obtain two sizes of feature map, one part of the feature map is directly concatenated with the other part after two depth-wise convolutions; a channel attention module is introduced between the two depth-wise convolutions; the feature map concatenated according to the channel dimension is again concatenated Convolution is used to fuse information between different channels. For the bottom-up part of the PAN network, M3 and M4 are downsampled by 2 times and then concatenated with M4 and M5 respectively. After passing through the CSA module again, the outputs P3, P4, and P5 are obtained. The fused feature maps of different sizes are input into the YOLOX decoupling head network to classify and locate the objects contained therein. In the channel attention module, the input feature map first aggregates the global spatial features through global average pooling, and then completes the information interaction between different channels through group convolution. The number of groups can be freely selected according to the number of input channels, and then multiplied with the input after passing the Sigmoid activation function; The P3, P4, and P5 feature maps are respectively input into the YOLOX decoupling head network. The decoupling head network extracts the target position and category information respectively, and learns them separately through different convolutional neural network branches, and finally performs feature fusion. An additional angle parameter is introduced in the regression branch to convert the anchor box coordinates from (center point, width, height) to (center point, width, height, rotation angle) to achieve the prediction of rotated targets.

3. The method for lightweight target detection model for satellite-based edge computing according to claim 2, characterized in that: S12 specifically includes the following steps: S121. Use data enhancement method to transform the input original image to obtain two similar images And pass through the encoder Encoder to output the feature map , the encoder architecture is consistent and shares parameters; S122, feature map The feature transformation is obtained by mapping the head ; S123, by predicting the head pair Make a prediction ,calculate and The cosine similarity of .

4. The method for lightweight target detection model for satellite-based edge computing according to claim 2, characterized in that: The contrastive learning framework in S12 includes data augmentation network structure Augmentation, encoder Encoder, Projection network structure and Prediction network structure; The data enhancement network structure Augmentation is used to expand the data set and transform the input image using methods such as random cropping and horizontal flipping to obtain two similar images. And input them into the encoder respectively; The encoder uses the ResNet50 network model; The Projection network structure consists of a fully connected layer, a BN layer, and a ReLu activation function. The correct classification of the image is achieved by fusing the output features of the encoder. The two Projections have the same architecture. The prediction network structure consists of a fully connected layer, a BN layer, and a ReLu activation function. By further transforming the output of the Projection network, its output is compared with the output of another Projection network to calculate the cosine similarity, which is used as the loss to train the encoder through back propagation.

5. The method for lightweight target detection model for satellite-based edge computing according to claim 1, characterized in that: S2 specifically includes the following steps: S21, abstracting the neurons of the image target detection model as nodes, abstracting the transmission relationship between neurons as edges, and constructing a neural network structure diagram with several convolutional layers, wherein the neural network structure diagram is stored in the form of a matrix; S22, based on the importance and feature map of the convolution kernel itself, pruning the convolution kernel in the convolution layer of the image object detection model; S23. The connections between neurons in different layers are channels. The similarity of the channels is calculated, and the redundant channels in the image target detection model are pruned based on the similarity of the channels.

6. The method for lightweight target detection model for satellite-based edge computing according to claim 5, characterized in that: S22 specifically includes the following steps: S221. From the perspective of the importance of the convolution kernel itself, the convolution kernel in the convolution layer is directly pruned; for a Layer convolutional layer network, suppose we want to Convolutional Layer To trim, Represents the total number of convolution kernel, that is output channels, and the number of input channels of each convolution kernel is , the size is ; In this branch, each convolution kernel in this layer is first calculated The L1 norm of , then sort the norms of these convolution kernels from large to small, and according to the pre-set pruning rate , after removing the soft pruning method The convolution kernels are arranged from large to small from front to back; the soft pruning method is to set the convolution kernel to be pruned to 0; S222. From the perspective of feature graphs, the network is further pruned by judging the correlation between convolution kernels; for a The network at layer In the layer, Represents the total number of feature map, Represent the height and width of the feature map respectively, that is, the size of each feature map is ; Calculate the importance of each feature map, sort them in descending order of importance, and remove the most important ones. % of the feature map is cut off, and the corresponding convolution kernel is also cut off. The calculation formula for calculating the importance of each feature map is as follows: in, Represents the importance of each feature map, That is, convert the three-dimensional feature map into the corresponding feature matrix after two-dimensional, represents the mask matrix. The first row is 0, and the rest of the rows are 1. Represents the SVD decomposition of the matrix, and the formula is as follows: in, represents the singular values ​​of the matrix, are the left and right singular value vectors respectively, the matrix The rank of ; Change the eigenvalue Arrange from large to small and remove the ones with the largest change. % feature map, and the corresponding filters are also cut off accordingly.

7. The method for lightweight target detection model for satellite-based edge computing according to claim 5, characterized in that: S23 includes the following steps: The structural similarity of the channel is calculated based on the interaction relationship between neurons. The calculation formula for the interaction relationship between neurons is: in, For the node contribution, is the node forward contribution, is the backward contribution of the node, is the forward node set, is the backward node set; Based on the calculated contribution of each node, the similarity between channels is calculated. The formula for calculating the similarity of channels is: in, It is a channel With channel The similarity of is the structural similarity, Contribute similarity to nodes, and Calculate the channel from the input and output and the hierarchical structure With channel The structural similarity of For the Layer The node contribution of each node; Arrange the channel overlap from high to low, and put the overlap at the end. The channels with only a single branch are identified as redundant channels and pruned from the network. For those channels that have only a single branch selected for removal, their parameters are restored in the subsequent network iteration update process to participate in the next training process. The joint dynamic channel pruning algorithm is iterative until the pruning rate of the network reaches the target pruning rate. Stop when.

8. The method for lightweight target detection model for satellite-based edge computing according to claim 1, characterized in that: S3 specifically includes the following steps: S31, build contrastive generative adversarial network; S32, Generate feature-enhanced datasets using contrastive generative adversarial networks; S33. Knowledge distillation: Fine-tuning compressed models using feature-augmented datasets.

9. The method for lightweight target detection model for satellite-based edge computing according to claim 8, characterized in that: In S31, the contrastive generative adversarial network uses the target detection model as a discriminator, and the optimization function formula of the contrastive generative adversarial network is: in, is the distribution of real data, is the noise distribution, is the output of the discriminator, is the output of the generator; the loss functions of the generator and the discriminator are: in, is the traditional discriminator loss function, To generate the similarity between the sample and the original sample, For expected similarity, the similarity between the generated sample and the original sample is expected to be , by adjusting The feature richness of generated samples can be changed. is the generation loss of similar data, is the generation loss of heterogeneous data, A variable parameter.

10. The method for lightweight target detection model for satellite-based edge computing according to claim 8, characterized in that: S33 specifically includes the following steps: The feature-enhanced image is input into the original network teacher net and the pruned lightweight network studentnet respectively, and the classification probability is output after softmax. , The definition is as follows: in, Output results for the network model, For each category of output, is the total number of categories; represents the temperature hyperparameter, which is used to control the probability distribution of the model output; The larger the value, the smoother the output probability distribution. The smaller the value, the more concentrated the probability distribution of the output; The output of the original network teacher net is used as the soft target, and the cross entropy loss is calculated by combining the soft target with the output of the student net. At the same time, the real label of the satellite image is used The cross entropy loss is calculated as the output of the hard target and the student net, and the parameters of the student net are updated through the back propagation algorithm. The calculation formula of the back propagation algorithm is as follows: in, represents the total loss function of the distillation part, represents the hard distillation loss, represents the soft distillation loss, Represents the weight parameter.

Images