An embedded lightweight remote sensing target detection system

Abstract

The present invention discloses an embedded lightweight remote sensing target detection system, including: an embedded device and a target detection model deployed on the embedded device; the target detection model adopts a deep learning model, and is deployed to the embedded device after training on a GPU platform to obtain parameters; the target detection model includes: a structural reparameterization module, which is used to absorb redundant residuals and convolutional structures in the deep learning model into a backbone network; an improved residual structure module, which is used to achieve the same dimension of input and output through an improved residual connection method. The embedded lightweight remote sensing target detection system, based on structural reparameterization, increases the use of residual connections, and improves the residual connection method when the input and output dimensions are different, reduces the FLOPs of the model, improves the model detection accuracy and maintains the inference speed of the model, so as to achieve the real-time performance of remote sensing target detection under limited computing power of the embedded device, and is convenient for improving work efficiency.

Description

Technical Field

[0001] The present invention relates to the fields of remote sensing image intelligent interpretation, deep learning and embedded systems, and in particular to an embedded lightweight remote sensing target detection system. Background Art

[0002] At present, with the continuous development of high-resolution remote sensing image technology, remote sensing image data has been widely used in various fields, such as security monitoring, resource exploration, disaster prevention and relief, military reconnaissance, etc. The acquisition of remote sensing data has the advantages of breaking through time and geographical limitations, large data volume, and cost saving. However, at the same time, only a small number of specific targets are often the objects of interest in large-format remote sensing images. Therefore, how to detect the target of interest from large-format remote sensing images is a hot issue in the field of digital image processing. A large number of algorithms based on deep neural networks have enabled remote sensing image target detection accuracy to reach a high level. However, the operation of high-precision algorithms requires a large amount of memory space and has high time complexity, so it is highly dependent on high-performance graphics processors. In the actual engineering application of remote sensing target detection, the current data processing is concentrated on ground measurement and control stations. On the one hand, such a processing platform consumes space and resources for the large number of graphics processors, so embedded systems are more general platforms; on the other hand, under the limited computing power and memory space of embedded systems, the task also requires the algorithm to have high detection accuracy and running speed. Although traditional target detection methods have low computing consumption and fast running speed, the detection accuracy is far from meeting the requirements of the task. Therefore, how to transplant remote sensing target detection algorithms based on deep learning into embedded systems is a research field with application value.

[0003] In the field of target detection in images, the traditional method is to use artificially designed operators such as SIFT features and HOG features to extract local area features of the image, and then use classifiers such as support vector machines to classify the features. With the development of deep learning technology, methods based on convolutional neural networks continue to emerge, surpassing traditional methods in detection accuracy and running speed, and gradually becoming the mainstream and forefront of research. However, remote sensing images also have their own particularities compared to more widely studied natural images. In remote sensing images, people often pay attention to sensitive targets such as aircraft, oil tanks, and ships, as well as sensitive objects such as airports and ports. Therefore, in the target detection of remote sensing images, problems such as large differences in target size and unbalanced target category labeling are particularly prominent, which is also an urgent problem to be solved in the field of remote sensing image target detection. On the other hand, remote sensing images have the characteristics of real-time, and image processing needs to be performed in real time on terminal devices such as satellites and aircraft. In practical applications, the storage resources and computing power of terminal devices are limited, and overly complex and large models cannot be deployed, and it is difficult to meet the real-time requirements. Summary of the invention

[0004] In view of the above problems, the present invention proposes an embedded lightweight remote sensing target detection system, which can solve the problems of limited computing power, inability to meet real-time requirements and low working efficiency in current embedded systems for target detection.

[0005] An embodiment of the present invention provides an embedded lightweight remote sensing target detection system, comprising: an embedded device and a target detection model deployed on the embedded device;

[0006] The target detection model adopts a deep learning model and is deployed to the embedded device after training on a GPU platform to obtain parameters; the target detection model includes:

[0007] The structure reparameterization module is used to absorb the redundant residual and convolutional structures in the deep learning model into the backbone network;

[0008] The improved residual structure module is used to achieve the same dimension of input and output through an improved residual connection method.

[0009] Furthermore, the structure of the target detection model is recompiled using the C++-based TensorRT library, and the trained parameters are converted and deployed on an embedded device for operation.

[0010] Furthermore, the structure reparameterization module is specifically used to convert the residual connection of 1×1 convolution and identity mapping in the deep learning model into a 3×3 convolution form.

[0011] Furthermore, the residual connection of 1×1 convolution and identity mapping in the deep learning model is converted into 3×3 convolution form, including:

[0012] Convert the 1×1 convolution and identity mapping into 3×3 convolution, and integrate the batch normalization operation into the convolution parameters through linear transformation. The batch normalization operation is shown in the following formula:

[0013]

[0014] Where bn(·) is the batch normalization operation; M is the feature map extracted from the remote sensing image; μ, σ, γ, β are batch normalization parameters;

[0015] For the convolution with weight matrix W, let but:

[0016] bn(M*W,μ,σ,γ,β)=M*W'+b'; absorb batch normalization into the convolution kernel.

[0017] Furthermore, the improved residual structure module is specifically used to directly intercept a feature map with the same dimension as the output and add it to the convolution result when the output dimension is smaller than the input dimension.

[0018] Furthermore, the improved residual structure module is also specifically used to complete the residual connection by copying and superimposing the input feature map along the dimension when the output dimension is larger than the input dimension, obtaining a feature map of the same dimension as the convolution output and then adding it to the convolution output.

[0019] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0020] An embedded lightweight remote sensing target detection system provided by an embodiment of the present invention includes: an embedded device and a target detection model deployed on the embedded device; the target detection model adopts a deep learning model and is deployed to the embedded device after training on a GPU platform to obtain parameters; the target detection model includes: a structure reparameterization module for absorbing redundant residuals and convolutional structures in the deep learning model into a backbone network; an improved residual structure module for achieving the same dimension of input and output through an improved residual connection method. The embedded lightweight remote sensing target detection system performs lightweight design on the deep learning model and uses an embedded system to perform target detection on remote sensing images. This paper explores the network design concept based on deep separable convolution and structural reparameterization, considers the impact of factors such as floating-point calculation number FLOPs and memory access on the model reasoning speed, and based on structural reparameterization, increases the use of residual connections and improves the residual connection method when the input and output dimensions are different, reduces the FLOPs of the model, improves the model detection accuracy and maintains the model reasoning speed, and obtains a lightweight target detection model with excellent comprehensive performance, so that embedded devices can meet the real-time and accuracy of remote sensing target detection under limited computing power, which helps to improve work efficiency.

[0021] Furthermore, in order to meet the real-time requirements of the onboard intelligent processing system, the lightweight target detection model can be deployed on the terminal embedded system using TensorRT to achieve real-time embedded system target detection.

[0022] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0023] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0025] Figure 1 A schematic diagram of oil tank and aircraft target detection by an embedded lightweight remote sensing target detection system provided by an embodiment of the present invention;

[0026] Figure 2 The overall architecture diagram of the target detection model provided by the embodiment of the present invention;

[0027] Figure 3a A schematic diagram of a residual structure in the training phase provided by an embodiment of the present invention;

[0028] Figure 3b A schematic diagram of a residual structure in a test phase provided by an embodiment of the present invention;

[0029] Figure 4a Schematic diagram of residual connection method with the same input and output dimensions;

[0030] Figure 4b A schematic diagram of a residual connection method when the input dimension is greater than the output dimension provided in an embodiment of the present invention;

[0031] Figure 4c A schematic diagram of a residual connection method when the input dimension is smaller than the output dimension provided in an embodiment of the present invention;

[0032] Figure 5 A schematic diagram of the deployment process of TensorRT provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0034] Reference Figure 1 As shown, an embodiment of the present invention provides an embedded lightweight remote sensing target detection system, the purpose of which is to lightweight the deep learning model and use the embedded system to detect targets in remote sensing images, such as the identification of storage equipment oil tanks, aircraft, ships, and dams. By exploring the network design concept based on deep separable convolution and structural reparameterization, and considering the impact of factors such as floating-point calculation number FLOPs and memory access on the model reasoning speed, a lightweight target detection model that can perform real-time detection in a terminal embedded system is designed.

[0035] The embedded lightweight remote sensing target detection system includes: an embedded device and a target detection model deployed on the embedded device; the target detection model adopts a deep learning model and is deployed to the embedded device after training on a GPU platform to obtain parameters; the target detection model includes:

[0036] The structure reparameterization module is used to absorb the redundant residual and convolutional structures in the deep learning model into the backbone network;

[0037] The improved residual structure module is used to achieve the same dimension of input and output through an improved residual connection method.

[0038] like Figure 2 As shown in the figure, it is the overall architecture diagram of the complete target detection model. In the training phase, the target detection model uses a complex multi-convolution kernel and multi-residual mode to fully model the features. In the test phase, the redundant residuals and convolution structures are absorbed into the backbone network through structural reparameterization. In the test phase, single-channel reasoning is performed to improve the reasoning speed of the network.

[0039] The specific implementation method and structural process are as follows. In the training stage, the network module adopts a multi-branch structure, and uses 3×3 convolution, 1×1 convolution and identity transformation to extract features, and uses residual connection to improve the model's feature expression ability. In the test stage, the structure of the training model is adjusted by structural reparameterization. Under the condition of keeping the mathematical operation unchanged, the 1×1 convolution and identity transformation operations are absorbed into the 3×3 convolution, so that the test model presents a single-channel information flow. Such a structure can maintain a faster reasoning speed. At the same time, the residual connection method is modified when the input and output dimensions are different, and the 1×1 convolution is omitted and replaced by the stacked feature layer to improve the model reasoning speed. For embedded systems, the model structure is recompiled using the C++-based TensorRT library, and the trained parameters are converted to further improve the running speed on the embedded system and realize real-time detection.

[0040] In this embodiment, the target detection model is based on structural reparameterization, increases the use of residual connections, and improves the residual connection method when the input and output dimensions are different, reduces the FLOPs of the model, improves the accuracy of the model in detecting remote sensing targets, and maintains the reasoning speed of the model. In specific implementation, for example, in order to meet the real-time requirements of the on-board intelligent processing system, the target detection model can be deployed on the terminal embedded system using TensorRT to achieve real-time embedded system target detection.

[0041] The technical solution of the present invention is specifically described below through three aspects:

[0042] 1. Lightweight design of the model based on structural reparameterization

[0043] In practical applications, the inference speed of the model is also affected by factors such as memory access and hardware optimization. In the most popular residual structure of convolutional neural networks, the model's instantaneous memory access is greater than that of the channel model because the intermediate results need to be temporarily retained in the memory when calculating the residual. In addition, in the current mainstream neural network frameworks, such as PyTorch, the optimization of convolution leads to the broken convolution structure in the depth-separable convolution, which is not conducive to the parallel execution of convolution calculations, resulting in a decrease in running speed. Since there is a one-to-one correspondence between network structure and parameters, if the parameters of multiple layers can be merged, the network structure can respond to changes. The macro process of structural reparameterization is to convert the residual connection of 1×1 convolution and identity mapping into the form of 3×3 convolution, and then merge the three branches into one to realize a single-branch model; at the micro level, 1×1 convolution and identity mapping can be easily regarded as 3×3 convolution, and then the batch normalization operation is integrated into the convolution parameters through linear transformation to enhance the generalization of the model. The batch normalization operation is shown in the following formula:

[0044]

[0045] Where bn(·) is the batch normalization operation; M is the feature map extracted from the remote sensing image; μ, σ, γ, β are batch normalization parameters;

[0046] For the convolution with weight matrix W, let but:

[0047] bn(M*W,μ,σ,γ,β)=M*W'+b'; After absorbing batch normalization into the convolution kernel, the residual connection can be fused into one branch by simple addition.

[0048] 2. Improve the residual structure to increase inference speed

[0049] Since ResNet first used residual connections in deep neural networks, this structure has been widely used in various network structures. The residual structure can retain shallow gradients and prevent gradient vanishing in deep network training. At the same time, it can also introduce more nonlinear transformations through the ReLU activation function to improve the expressiveness of features. However, the residual structure used in structural reparameterization is a "pseudo-residual" structure that does not contain an implicit feature layer. If the network depth is too large, gradient vanishing may still occur during the gradient back propagation process.

[0050] Reference Figure 3a As shown in the figure, the input passes through two parallel structures and then performs a residual connection with the input, and then passes through the ReLU activation function to obtain the output. Figure 3bIn the inference stage model structure shown in the figure, the 1×1 convolution and the identity residual connection of the parallel structure are absorbed into the 3×3 convolution by the structural reparameterization, thus becoming a single-branch structure. However, the residual connection of the outermost layer is retained and added to the result after convolution before the activation function. This structure retains the residual connection during the inference process, and retains the network's ability to resist gradient disappearance through the ReLU nonlinear change after the residual connection.

[0051] In addition, the improved residual connection method can also meet the requirements of residual connections with different input and output dimensions. Figure 4a-4c As shown in Figure 1, for different input and output dimensions, the residual connection method is also different. Figure 4a This shows the case where the input and output dimensions are the same. In this case, the residuals are connected in the same way as in most networks, using the identity mapping and adding it to the convolutional feature map. Figure 4b In the case where the output dimension is smaller than the input dimension (common in the downsampling stage), the feature dimension of the residual branch is not changed by 1×1 convolution. Instead, the feature map with the same output dimension is directly intercepted and added to the convolution result. Figure 4c In the case where the output dimension is larger than the input dimension (common in the upsampling stage), no 1×1 convolution is performed. Instead, the input feature map is copied and superimposed along the dimension to obtain a feature map of the same dimension as the convolution output, and then the residual connection is completed by adding it to the convolution output. For the case where the input and output resolutions are different after downsampling, the maximum pooling is first performed in the residual branch, and then Figure 4a-4c Through this residual connection method, the residual branch does not consume any FLOPs, and the running speed is improved by reducing parallel convolution calculations.

[0052] 3. Deployment of lightweight models in embedded systems

[0053] Most current neural network models rely on deep learning frameworks for high-performance servers, such as TensorFlow and PyTorch. On the one hand, these frameworks are based on the Python language. Although they are simpler and more friendly in development and design, their operation efficiency is low; on the other hand, these frameworks are developed based on CUDA and optimized for GPU platforms. They are not suitable for the computing methods of embedded systems, so their running speed on embedded systems is not good. To this end, the embodiment of the present invention also uses the TensorRT library designed specifically for embedded systems to rewrite the model structure so that it can be deployed on embedded platforms and further improve the reasoning speed of the model.

[0054] Since the computing power of embedded platforms is poor and training is not possible, it is necessary to first train on the GPU to obtain parameters and then deploy them to the embedded platform for inference. Using the TensorRT library, the PyTorch model for GPU can be converted to a model for embedded platforms. The deployment process is as follows: Figure 5 As shown. First, according to the designed model structure, the PyTorch model is built on the server side using the Python-based PyTorch framework, and the training image is input for iterative training to finally obtain the model parameters. Then, the model is rebuilt using the C++-based TensorRT library, and its structure needs to be consistent with the PyTorch model to ensure the correctness of parameter loading. After the TensorRT model is compiled and the model parameters are loaded on the embedded terminal, the model engine for inference can be obtained. As shown Figure 5 As shown, by inputting the test image into the inference engine, reasoning can be performed in the embedded system, and then the accurate detection of the target in the test image can be output.

[0055] For example, the large-scale computing capability and intelligence level of the current satellite remote sensing processing system are relatively poor. The embedded lightweight remote sensing target detection system provided by the embodiment of the present invention can be used to design a lightweight target detection model. The embodiment of the present invention performs a lightweight design on the remote sensing target detection model. The structural reparameterization module designs a lightweight target detection network based on the structural reparameterization method, and analyzes the influence of the residual branch on the reasoning speed by improving the residual structure module, and improves the residual connection method when the input and output dimensions are different, thereby reducing the FLOPs of the model, improving the model detection accuracy and maintaining the reasoning speed of the model, so that the embedded device can meet the real-time and accuracy of remote sensing target detection under limited computing power, which helps to improve work efficiency.

[0056] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An embedded lightweight remote sensing target detection system, characterized in that: include: An embedded device and an object detection model deployed on the embedded device; The embedded device is an on-board remote sensing processing system; 1) The target detection model adopts a deep learning model and is deployed to the embedded device after training on a GPU platform to obtain parameters; The object detection model includes: The structure reparameterization module is used to absorb the redundant residual and convolutional structures in the deep learning model into the backbone network; Improved residual structure module, used to achieve the same dimension of input and output through improved residual connection method; The improved residual structure module is specifically used for directly intercepting a feature map with the same dimension as the output and adding it to the convolution result when the output dimension is smaller than the input dimension; and is also specifically used for completing the residual connection by copying and superimposing the input feature map along the dimension to obtain a feature map with the same dimension as the convolution output and then adding it to the convolution output when the output dimension is larger than the input dimension; 2) Recompile the structure of the target detection model using the C++-based TensorRT library, convert the trained parameters, and deploy them on an embedded device; In the training phase of the target detection model, the network module adopts a multi-branch structure, and uses 3×3 convolution, 1×1 convolution and identity transformation to extract features, and uses residual connection to improve the feature expression ability of the model. The input passes through two parallel structures and is residually connected with the input, and then the output is obtained through the ReLU activation function. In the test phase, the structure of the training model is adjusted by structural reparameterization, and the 1×1 convolution and identity transformation operations are absorbed into the 3×3 convolution while keeping the mathematical operation unchanged, so that the test model presents a single-channel information flow; the outermost residual connection is retained and added to the result after convolution before the activation function; The batch normalization operation is integrated into the convolution parameters through linear transformation. The batch normalization operation is shown as follows: Where bn(·) is the batch normalization operation; M is the feature map extracted from the remote sensing image; μ, σ, γ, β are batch normalization parameters; For the convolution with weight matrix W, let but: bn(M*W,μ,σ,γ,β)=M*W'+b'; absorb batch normalization into the convolution kernel.

Images