Ultra-light remote sensing image in-orbit real-time target detection method and device
Through data recombination and grouping convolution technology, the resource waste problem of deep learning object detection network is solved during on-orbit deployment, efficient real-time object detection is achieved, and the detection accuracy and resource utilization balance is ensured.
Patent Information
- Application Number
- CN202411229837.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-12-20
AI Technical Summary
When existing deep learning object detection networks are deployed on-orbit, it is difficult to meet the hardware resource requirements due to the large number of parameters and high computing volume. In addition, the channel needs to be expanded in the channel dimension parallel processing mode, resulting in wasted computing resources and affecting the inference efficiency.
Using data recombination and packet convolution technology, reverse recombination is performed to reduce feature maps by recombining high-resolution single-channel images into low-resolution feature maps and performing multi-stage packet convolution and downsampling processing in feature extraction networks. Size, avoid redundant calculations caused by channel expansion, and design ultra-lightweight backbone networks to reduce the amount of parameters and calculations.
It effectively improves the inference speed of on-site real-time object detection, makes full use of computing resources, ensures that the target detection accuracy is within an acceptable range, and reduces resource overhead.
Smart Images
Figure CN119169269A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of target detection, and in particular to an ultra-lightweight remote sensing image on-orbit real-time target detection method and device. Background Art
[0002] With the successful application of deep learning technology in the field of remote sensing, remote sensing image target detection based on deep learning has played an increasingly important role in the military and civilian fields, and is widely used in military guidance, intelligent transportation, and maritime rescue tasks. Deep learning methods have greatly improved the accuracy of target detection, but due to the large number of detection network parameters and the extremely low computing resources of the on-orbit platform, it is difficult to deploy the existing detection network on the on-orbit platform, so a high-performance on-orbit real-time target detection method is urgently needed.
[0003] At present, deep learning-based target detection algorithms are mainly divided into three categories. The first category is the two-stage target detection algorithm represented by Faster R-CNN, which generates candidate regions on the bounding box containing the target and then performs target detection. The detection accuracy is high but the efficiency is low. The second category is the single-stage target detection algorithm, mainly represented by the SSD (Single Shot MultiBox Detector) and YOLO (You Only Look Once) series. This type of algorithm does not generate candidate regions, but directly performs target detection by regression. The detection efficiency has been greatly improved compared with the first type of method. The third category is the end-to-end target detection method represented by DETR (DEtection TRansformer). This type of algorithm uses Transformer as the detection head for sparse prediction, and directly obtains the detection result without post-processing methods such as NMS (non-maximum suppression), which improves the efficiency of reasoning, but the large amount of calculation of the Transformer structure brings certain difficulties to deployment.
[0004] Most current real-time target detection methods use a single-stage solution. Although the single-stage target detection method performs well on various scene data sets and is highly efficient, there are still some challenges in its actual deployment on orbit. The single-stage target detection network faces two main problems in on-orbit real-time processing. First, the network parameters and computational complexity are large, and the hardware resources on orbit are insufficient, making it difficult to meet the needs of network deployment. Second, for the channel dimension parallel processing mode, due to the small number of image input channels, the channels need to be expanded, that is, each channel needs to be filled with zeros, which wastes computing resources. Therefore, it is necessary to lightweight the network structure and eliminate the impact of the input expansion channel on the inference speed, while ensuring that the accuracy of target detection is reduced within an acceptable range, so as to complete high-performance on-orbit real-time target detection. Summary of the invention
[0005] In view of this, the present invention provides an ultra-lightweight on-orbit real-time target detection method for remote sensing images, which can address the problems existing in the on-orbit application of the current detection network, and adopts network lightweight structure design and data reorganization to improve the reasoning speed and reduce resource overhead while ensuring network performance, so as to complete on-orbit efficient real-time target detection and solve the problem in the prior art that the image input expansion channel reduces the hardware reasoning efficiency.
[0006] In order to solve the above technical problems, the present invention is implemented as follows.
[0007] An ultra-lightweight on-orbit real-time target detection method for remote sensing images, wherein the deep learning network used for target detection includes a feature extraction network and a detection network; the method comprises the following steps:
[0008] Step 1: Reorganize the high-resolution single-channel image according to the input channel dimension N1 of the feature extraction network to obtain a low-resolution feature map of N1 channels;
[0009] The reorganization method is: according to the set step size r, the high-resolution single-channel image is divided into r-sized 2 The block, r 2 =N1; r in the same block 2 pixels are assigned to the same position in the low-resolution feature map of N1 channels;
[0010] Step 2: Input the low-resolution feature maps of the N1 channels into the feature extraction network, the feature extraction network performs multiple stages of group convolution and downsampling processing, and then performs reverse reorganization in a manner opposite to data reorganization to obtain a downsampled feature map;
[0011] Step 3: Input the feature map obtained in step 2 into the detection network to obtain the target detection result.
[0012] Preferably, in step 1, the r in the same block 2 The pixels are assigned to the same position in the low-resolution feature map of N1 channels:
[0013] Suppose the point in the high-resolution single-channel image is the initial point (h1, w1), and the position assigned to the low-resolution feature map is the corresponding point (c2, h2, w2). The correspondence between the initial point (h1, w1) and the corresponding point (c2, h2, w2) is calculated as follows:
[0014] Perform a modulo operation on the position of the initial point and the number of output feature channels N1 to obtain the channel position c2 of the corresponding point:
[0015] c2=(w1+h1×W1)%N1
[0016] According to the pixel rearrangement method, the position of the corresponding point in the c2th channel of the output feature is calculated (c2, h2, w2):
[0017]
[0018] h2=h1 / r.
[0019] Preferably, in step 2, the feature extraction network includes 4 downsampling stages and one reverse data reorganization stage;
[0020] In the 4 downsampling stages:
[0021] The input of the first stage is a low-resolution feature map P0 (W, H, 16) with 16 channels, where W and H represent the two-dimensional size of the feature map, and the number 16 represents 16 channels; the low-resolution feature map P0 passes through the first grouped convolution layer, outputs data of 16 channels, and then passes through the first pooling layer to output a feature map P1 (W / 2, H / 2, 16) with 16 channels; the first grouped convolution layer has a 3×3 convolution kernel and the channel grouping is 2; the pooling window of the first pooling layer is 2×2;
[0022] The second stage input is a 16-channel feature map P1, which is output through the second convolution layer to produce 32-channel data, and then through the second pooling layer to produce a 32-channel feature map P2 (W / 4, H / 4, 32); the second convolution layer has a 3×3 convolution kernel and the channel grouping is 2; the pooling window of the second pooling layer is 2×2;
[0023] The three-stage input is a feature map P2 of 32 channels, and the third grouped convolution layer outputs data of 32 channels, and then the fourth grouped convolution layer outputs data of 32 channels, and then the third pooling layer outputs a feature map P3 (W / 8, H / 8, 32) of 32 channels; the third grouped convolution layer has a 3×3 convolution kernel, and the channel grouping is 2; the fourth grouped convolution layer has a 3×3 convolution kernel, and the channel grouping is 4; the pooling window of the third pooling layer is 2×2;
[0024] The fourth stage input is a feature map P3 of 32 channels, and the fifth grouped convolution layer outputs data of 32 channels, and then the sixth grouped convolution layer outputs data of 32 channels, and then the fourth pooling layer outputs a feature map P4 (W / 16, H / 16, 32) of 32 channels; the fifth grouped convolution layer has a 3×3 convolution kernel, and the channel grouping is 2; the sixth grouped convolution layer has a 3×3 convolution kernel, and the channel grouping is 4; the pooling window of the fourth pooling layer is 2×2;
[0025] In the reverse data reorganization stage, the input is a 32-channel feature map P4, which is output through the first convolution layer with 64 channels of data, and then through the seventh group convolution layer to output a 512-channel feature P5 (W / 16, H / 16, 512). The feature P5 is reversely reorganized to restore it to 32-channel data P6 (W / 16, H / 16, 32), and then input into the second convolution layer to output a 64-channel feature P out (W / 16, H / 16, 64); wherein the seventh grouped convolutional layer has a 3×3 convolutional kernel and the channel grouping is 8; the first convolutional layer and the second convolutional layer have a 3×3 convolutional kernel.
[0026] Preferably, the detection network includes a regression branch and a classification branch; the regression branch and the classification branch are both composed of a 3×3 convolution with a channel grouping of 8 and an output channel of 64.
[0027] Preferably, in the target detection process of step 3, the box with a confidence score greater than 0.2 in the detection network output result is used as the target detection result.
[0028] Preferably, when performing group convolution in step 2, the p convolution kernels in each group convolution layer are divided into n groups according to the hardware processing capability, and p / n convolution kernels are processed in parallel each time.
[0029] The present invention also provides an ultra-lightweight on-orbit real-time target detection device for remote sensing images, the device comprising a data reorganization module, a feature extraction network and a detection network; the feature extraction network and the detection network constitute a deep learning network;
[0030] The data reorganization module is used to reorganize the high-resolution single-channel image according to the input channel dimension N1 of the feature extraction network to obtain the low-resolution feature map of N1 channels; the reorganization method is: according to the set step size r, the high-resolution single-channel image is divided into r-sized 2 The block, r 2 =N1; r in the same block 2 pixels are assigned to the same position in the low-resolution feature map of N1 channels;
[0031] A feature extraction network, used for performing multiple stages of grouped convolution and downsampling processing on the low-resolution feature maps of the N1 channels, and then performing reverse reorganization in a manner opposite to the data reorganization to obtain a downsampled feature map;
[0032] The detection network is used to perform target detection based on the downsampled feature map from the feature extraction network to obtain a target detection result.
[0033] Preferably, the feature extraction network comprises a first downsampling unit, a second downsampling unit, a third downsampling unit, a fourth downsampling unit and a reverse data reorganization unit;
[0034] The first downsampling unit is responsible for one-stage downsampling, including the first grouped convolution layer and the first pooling layer; the input of the first grouped convolution layer is a low-resolution feature map P0 (W, H, 16) of 16 channels, W and H represent the two-dimensional size of the feature map, and the number 16 represents 16 channels; the low-resolution feature map P0 passes through the first grouped convolution layer, outputs 16-channel data, and then passes through the first pooling layer to output a 16-channel feature map P1 (W / 2, H / 2, 16); the first grouped convolution layer has a 3×3 convolution kernel, and the channel grouping is 2; the pooling window of the first pooling layer is 2×2;
[0035] The second downsampling unit is responsible for two-stage downsampling, including the second grouped convolution layer and the second pooling layer; the input of the second grouped convolution layer is the feature map P1 of 16 channels, and the second convolution layer outputs the data of 32 channels, and then passes through the second pooling layer to output the feature map P2 (W / 4, H / 4, 32) of 32 channels; the second convolution layer has a 3×3 convolution kernel and the channel grouping is 2; the pooling window of the second pooling layer is 2×2;
[0036] The third downsampling unit is responsible for three-stage downsampling, including a third group convolution layer, a fourth group convolution layer and a third pooling layer; the input of the third group convolution layer is a feature map P2 of 32 channels, and the third group convolution layer outputs data of 32 channels, and then the fourth group convolution layer outputs data of 32 channels, and then the third pooling layer outputs a feature map P3 (W / 8, H / 8, 32) of 32 channels; the third group convolution layer has a 3×3 convolution kernel, and the channel grouping is 2; the fourth group convolution layer has a 3×3 convolution kernel, and the channel grouping is 4; the pooling window of the third pooling layer is 2×2;
[0037] The fourth downsampling unit is responsible for four-stage downsampling, including the fifth group convolution layer, the sixth group convolution layer and the fourth pooling layer; the input of the fifth group convolution layer is the feature map P3 of 32 channels, and the fifth group convolution layer outputs the data of 32 channels, and then the sixth group convolution layer outputs the data of 32 channels, and then the fourth pooling layer outputs the feature map P4 (W / 16, H / 16, 32) of 32 channels; the fifth group convolution layer has a 3×3 convolution kernel, and the channel grouping is 2; the sixth group convolution layer has a 3×3 convolution kernel, and the channel grouping is 4; the pooling window of the fourth pooling layer is 2×2;
[0038] The reverse data reorganization unit includes the first convolution layer, the seventh group convolution layer, the reverse reorganization unit and the second convolution layer; the input of the first convolution layer is the feature map P4 of 32 channels, which outputs 64 channels of data through the first convolution layer, and then outputs 512 channels of feature P5 (W / 16, H / 16, 512) through the seventh group convolution layer; the reverse reorganization unit performs reverse data reorganization on the feature P5 to restore it to 32 channels of data P6 (W / 16, H / 16, 32), and then inputs it into the second convolution layer; the second convolution layer outputs 64 channels of feature Px ut (W / 16, H / 16, 64); wherein the seventh grouped convolutional layer has a 3×3 convolutional kernel and the channel grouping is 8; the first convolutional layer and the second convolutional layer have a 3×3 convolutional kernel.
[0039] Beneficial effects:
[0040] (1) When constructing a multi-channel feature map, the present invention does not adopt the method of copying a high-resolution single-channel image and performing convolution, but adopts a data reorganization method to directly reorganize the high-resolution single-channel image into a multi-channel low-resolution feature map that adapts to the feature extraction network, and then reversely reorganizes it after downsampling processing in the feature extraction network. In the channel dimension parallel processing mode of the on-orbit platform, the convolution calculation adopts group calculation, that is, when calculating the single-channel input data, the number of channels needs to be padded to the minimum number of grouped channels, which will lead to a large waste of computing resources. The present invention reduces the size of the feature map without the need for convolution processing through this pixel rearrangement method, avoids the redundancy caused by channel expansion, and thus improves the timeliness of on-orbit applications. At the same time, this method has extremely small data information loss, so that the extracted features can effectively characterize the targets in the remote sensing image. This method effectively reduces the size of the input data, distributes the data to the channel dimension, makes full use of computing resources, and fully and effectively utilizes data information.
[0041] (2) Aiming at the purpose of on-orbit real-time target detection, the present invention provides an ultra-lightweight feature extraction backbone network based on group convolution, designs the number of stages, whether to use group convolution in each stage, and the number of groups, thereby greatly reducing the number of parameters and computational complexity of the detection network. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 This is an example of the data reorganization process of the present invention.
[0043] Figure 2 This is a structural diagram of the ultra-lightweight backbone network of the present invention.
[0044] Figure 3 Schematic diagram of the group convolution calculation process.
[0045] Figure 4Schematic diagram of the parallel computing process of group convolution on hardware.
[0046] Figure 5 The present invention is a schematic diagram of the composition of the ultra-lightweight on-orbit real-time target detection device for remote sensing images. DETAILED DESCRIPTION
[0047] The present invention is described in detail below with reference to the accompanying drawings and embodiments.
[0048] The present invention provides an ultra-lightweight on-orbit real-time target detection scheme for remote sensing images. The scheme improves the target detection scheme of deep learning from two perspectives: data reorganization and group convolution.
[0049] Regarding data reorganization: When constructing the feature map of the input feature extraction network, the method of copying the high-resolution single-channel image and performing multi-channel convolution calculations is not adopted. Instead, the data reorganization method is adopted. Through pixel position allocation, the high-resolution single-channel image is directly reorganized into a multi-channel low-resolution feature map that adapts to the feature extraction network, and then reversely reorganized after downsampling processing in the feature extraction network. The construction of the multi-channel low-resolution feature map does not require the use of large-scale convolutional layers to process high-resolution single-channel images of multiple channels. Only pixel position configuration is required to obtain the multi-channel low-resolution feature map, avoiding redundant calculations caused by the expansion of convolution calculation channels, thereby improving the timeliness of on-orbit applications. This method has extremely small data information loss, so that the extracted features can effectively characterize the targets in the remote sensing image.
[0050] For group convolution: In the feature extraction network, the number of stages, whether to use group convolution in each stage, and the number of groups are designed, which greatly reduces the number of parameters and calculations of the detection network, and obtains an ultra-lightweight backbone network. While ensuring network performance, the inference speed is improved and resource overhead is reduced, completing efficient real-time target detection on orbit.
[0051] To achieve the above object, the technical solution of the present invention comprises the following steps:
[0052] Step 1: Reorganize the input high-resolution single-channel image according to the output channel dimension N1 to obtain a low-resolution feature map of N1 channels.
[0053] The reorganization method of the present invention is: according to the set step length r, the high-resolution single-channel image is divided into r-sized 2 The block, r 2 =N1; r in the same block 2Pixels are assigned to the same position in the low-resolution feature map of N1 channels. In one embodiment of the present invention, it is necessary to obtain low-resolution feature maps of 16 channels and input them into the feature extraction network for feature extraction. Therefore, N1=16, then r is 4. Because the data of 16 channels is too dense in the attached figure, for ease of understanding, Figure 1 The figure shows how a high-resolution single-channel image of size 1×rh×rw is reorganized into a 4-channel low-resolution feature map with r=2. The 4 pixels in the same block are read in a fixed order, such as {1, 2, 7, 8}, and then assigned to the same position of the low-resolution feature map of the 4 channels, such as (1, 1).
[0054] The above pixel allocation process can be expressed as follows. Let the point in the high-resolution image (size H1×W1) be (c1, h1), and the position assigned to the low-resolution feature map be the corresponding point (c2, h2, w2). The correspondence between the initial point (h1, w1) and the corresponding point (c2, h2, w2) is calculated as follows:
[0055] Perform a modulo operation on the position of the initial point and the number of output feature channels N1 to obtain the channel position c2 of the corresponding point:
[0056] c2=(w1+h1×W1)%N1
[0057] According to the pixel rearrangement method, the position of the corresponding point in the c2th channel of the output feature is calculated (c2, h2, w2):
[0058] w2=w1 / r
[0059] h2=h1 / r
[0060] In the channel dimension parallel processing mode of the on-orbit platform, convolution calculation uses group calculation, that is, when calculating single-channel input data, the number of channels needs to be padded to the minimum number of grouped channels, which will lead to a large waste of computing resources. By rearranging pixels in this way, the size of the feature map can be reduced, and the loss of data information is minimal. This method effectively reduces the size of the input data and distributes the data to the channel dimension, so that computing resources are fully utilized, while fully and effectively utilizing data information.
[0061] Step 2: Input the reorganized data features of N1 channels in step 1 into the feature extraction network. The feature extraction network adopts a redesigned lightweight backbone network, which performs multiple stages of group convolution and downsampling processing, and then performs reverse reorganization in the opposite way to the data reorganization to obtain the downsampled feature map.
[0062] The core structure of the lightweight backbone network is group convolution, which can effectively reduce the amount of calculation. If the input feature size is (W1, H1, C1), the output feature map size is (W2, H2, C2), and the number of groups is g, then the feature size of each input group is The convolution kernels are also divided into g groups, and the convolution kernel size of each group is Total convolution kernels, group convolution outputs a total of g features, each feature size is From the calculation process of group convolution, its parameter quantity is The amount of calculation is The number of parameters of conventional convolution is k×k×C1×C2, and the amount of calculation is k×k×C1×C2×W1×H1. The amount of calculation and parameter of grouped convolution are both
[0063] The execution process of each group of operations in the above group convolution is parallel on the hardware, such as Figure 4 As shown, the grouped convolution contains convolution kernels, each of which has a size of During the calculation, a k×k window slides along the width of the input feature map to perform row processing. The features in the window need to be consistent with the corresponding convolution kernels are used to perform operations. In order to process in parallel, The convolution kernels are grouped, and n convolution kernels are involved in the operation each time, and the operation is repeated times until you get The above intermediate calculation results are stored in the accumulator to be added to the result of the next row processing. The capacity of the accumulator is Each row processing shares the same set of convolution kernels, and the size of the intermediate result is The window will continue to traverse The convolution operation is repeated for each channel and row processing is called channel processing. After that, the window will move row by row and repeat the channel processing for H1+2×pad-k+1 times, where pad is the number of zero paddings at the edge of the input image. During the whole processing, the convolution kernel weights need to be read H1+2×pad-k+1 times.
[0064] In this embodiment, a lightweight backbone network suitable for 16-fold downsampling is designed, and a 16-fold downsampling feature map is obtained based on the reorganized data features. The number of stages, whether to use group convolution in each stage, and the number of groups are designed, which greatly reduces the number of parameters and calculations of the detection network. Figure 2 The overall structure of the lightweight backbone network of the present invention is shown, which is divided into four downsampling stages.
[0065] The low-resolution feature map P0 (W, H, 16) of 16 channels after the first-stage input data is reorganized. W and H in (W, H, 16) represent the two-dimensional size of the feature map, and the number 16 represents 16 channels. Similar expressions have the same meaning below, so they are not repeated here. The first-stage input P0 is processed by convolution layer 1 and pooling layer 1, and the output feature map P1 (W / 2, H / 2, 16) is obtained. Convolution layer 1 has a 3×3 convolution kernel and a channel grouping of 2; pooling layer 1 has a 2×2 pooling window.
[0066] The second stage input feature map P1 is processed by convolution layer 2 and pooling layer 2, and the output feature map P2 (W / 4, H / 4, 32) is obtained. Among them, convolution layer 2 has a 3×3 convolution kernel, outputs 32 channels, and the channel grouping is 2; pooling layer 2 has a 2×2 pooling window.
[0067] The three-stage input feature map P2 is processed by convolution layer 3_1, convolution layer 3_2 and pooling layer 3, and the output feature map P3 (W / 8, H / 8, 32). Among them, convolution layer 3_1 has a 3×3 convolution kernel, outputs 32 channels, and the channel grouping is 2; convolution layer 3_2 has a 3×3 convolution kernel, outputs 32 channels, and the channel grouping is 4; pooling layer 3 has a 2×2 pooling window.
[0068] The four-stage input feature map P3 is processed by convolution layer 4_1, convolution layer 4_2 and pooling layer 4 to output feature map P4 (W / 16, H / 16, 32). Among them, convolution layer 4_1 has a 3×3 convolution kernel, outputs 32 channels, and the channel grouping is 2; convolution layer 4_2 has a 3×3 convolution kernel, outputs 32 channels, and the channel grouping is 4; pooling layer 4 has a 2×2 pooling window.
[0069] The output feature map P4 enters the reverse data reorganization stage. In the reverse data reorganization stage, the feature map is processed by convolution layer 5_1 and convolution layer 5_2 to obtain feature P5 (W / 16, H / 16, 512); then P5 is reversely reorganized to restore it to (W / 16, H / 16, 32), and then input into convolution layer 5_3 to obtain output feature P out (W / 16, H / 16, 64). Among them, convolution layer 5_2 is a grouped convolution. Convolution layer 5_1 and convolution layer 5_3 are not grouped. Convolution layer 5_1 has a 3×3 convolution kernel and outputs 64 channels; convolution layer 5_2 has a 3×3 convolution kernel and outputs 512 channels with 8 channel groups; convolution layer 5_3 has a 3×3 convolution kernel and outputs 64 channels.
[0070] Step 3: Set feature P out Input the detection network to obtain the target detection results, including target classification prediction results and position regression results.
[0071] The detection head consists of two decoupled parts: the regression branch and the classification branch. Both the regression branch and the classification branch consist of a 3×3 convolution with a channel grouping of 8 and an output channel of 64, respectively obtaining the regression features P reg and the classification feature P cls The target classification prediction result is given by P cls After 3×3 convolution, the output channel is the preset number of anchors × the number of categories; the target position prediction result is obtained by P reg It is obtained by 3×3 convolution with channel grouping of 2, and the output channel is the preset number of anchors × 4.
[0072] Step 4: For the obtained category and position prediction results, a one-to-one matching method is used to select training positive samples, and the total training loss is calculated. The network training is performed by iterating steps 1-4.
[0073] The minimum cost binary matching method is used to select positive training samples. The cost calculation method of the positioning branch is
[0074] C loc =λ iou *C iou +λ L1 *C L1
[0075] Among them C iou and C L1 are the IOU loss and L1 loss of the predicted sample and the true value, respectively, iou and λ L1 is the corresponding coefficient.
[0076] The cost calculation method of the classification branch is
[0077] λ cls *C cls
[0078] Among them C cls is the loss of the predicted sample and the true value category, λ cls is the corresponding coefficient.
[0079] The total cost is
[0080] C=λ cls *C cls +C loc
[0081] The matching scheme that minimizes the total cost can obtain positive samples that match the true value one-to-one. The obtained positive samples and the true value are used to calculate the L1 loss to obtain the loss of the regression branch, and the loss of the classification branch is calculated using the Focal loss.
[0082] Step 5: Use the model obtained by network training for inference. No post-processing such as non-maximum suppression is required. The boxes with confidence scores greater than 0.2 in the model output are used as target detection results.
[0083] The present invention also provides an ultra-lightweight on-orbit real-time target detection device for remote sensing images, such as Figure 5 As shown, it includes a data reorganization module, a feature extraction network and a detection network; the feature extraction network and the detection network constitute a deep learning network;
[0084] The data reorganization module is used to reorganize the data of the high-resolution single-channel image according to the input channel dimension N1 of the feature extraction network to obtain the low-resolution feature map of N1 channels; the reorganization method is as described above.
[0085] The feature extraction network is used to perform multiple stages of grouped convolution and downsampling processing on the low-resolution feature maps of N1 channels, and then perform reverse reorganization in the opposite manner to the data reorganization to obtain the downsampled feature maps. This feature extraction network includes 4 downsampling units and one reverse data reorganization unit. The 4 downsampling units correspond to the 4 downsampling stages mentioned above, and the reverse data reorganization unit corresponds to the reverse data reorganization stage mentioned above.
[0086] The detection network is used to perform target detection according to the downsampled feature map from the feature extraction network to obtain a target detection result.
[0087] The above specific embodiments only describe the design principle of the present invention. The shapes and names of the components in the description may be different and are not limited. Therefore, those skilled in the art in the field of the present invention may modify or replace the technical solutions recorded in the above embodiments; and these modifications and replacements do not deviate from the creative purpose and technical solutions of the present invention and should all fall within the protection scope of the present invention.
Claims
1. An ultra-lightweight on-orbit real-time target detection method for remote sensing images, characterized in that: The deep learning network used in target detection includes a feature extraction network and a detection network; the method includes the following steps: Step 1: Reorganize the high-resolution single-channel image according to the input channel dimension N1 of the feature extraction network to obtain a low-resolution feature map of N1 channels; The reorganization method is: according to the set step size r, the high-resolution single-channel image is divided into r-sized 2 The block, r 2 =N1; r in the same block 2 pixels are assigned to the same position in the low-resolution feature map of N1 channels; Step 2: Input the low-resolution feature maps of the N1 channels into the feature extraction network, the feature extraction network performs multiple stages of group convolution and downsampling processing, and then performs reverse reorganization in a manner opposite to data reorganization to obtain a downsampled feature map; Step 3: Input the feature map obtained in step 2 into the detection network to obtain the target detection result.
2. The method according to claim 1, characterized in that In step 1, the r in the same block 2 The pixels are assigned to the same position in the low-resolution feature map of N1 channels: Suppose the point in the high-resolution single-channel image is the initial point (h1, w1), and the position assigned to the low-resolution feature map is the corresponding point (c2, h2, w2). The correspondence between the initial point (h1, w1) and the corresponding point (c2, h2, w2) is calculated as follows: Perform a modulo operation on the position of the initial point and the number of output feature channels N1 to obtain the channel position c2 of the corresponding point: c2=(w1+h1×W1)%N1 According to the pixel rearrangement method, the position of the corresponding point in the c2th channel of the output feature is calculated (c2, h2, w2): h2=h1 / r.
3. The method according to claim 1, characterized in that In step 2, the feature extraction network includes 4 downsampling stages and one reverse data reorganization stage; In the 4 downsampling stages: The input of the first stage is a low-resolution feature map P0 (W, H, 16) with 16 channels, where W and H represent the two-dimensional size of the feature map, and the number 16 represents 16 channels; the low-resolution feature map P0 passes through the first grouped convolution layer, outputs data of 16 channels, and then passes through the first pooling layer to output a feature map P1 (W / 2, H / 2, 16) with 16 channels; the first grouped convolution layer has a 3×3 convolution kernel and the channel grouping is 2; the pooling window of the first pooling layer is 2×2; The second stage input is a 16-channel feature map P1, which is output through the second convolution layer to produce 32-channel data, and then through the second pooling layer to produce a 32-channel feature map P2 (W / 4, H / 4, 32); the second convolution layer has a 3×3 convolution kernel and the channel grouping is 2; the pooling window of the second pooling layer is 2×2; The three-stage input is a feature map P2 of 32 channels, and the third grouped convolution layer outputs data of 32 channels, and then the fourth grouped convolution layer outputs data of 32 channels, and then the third pooling layer outputs a feature map P3 (W / 8, H / 8, 32) of 32 channels; the third grouped convolution layer has a 3×3 convolution kernel, and the channel grouping is 2; the fourth grouped convolution layer has a 3×3 convolution kernel, and the channel grouping is 4; the pooling window of the third pooling layer is 2×2; The fourth stage input is a feature map P3 of 32 channels, and the fifth grouped convolution layer outputs data of 32 channels, and then the sixth grouped convolution layer outputs data of 32 channels, and then the fourth pooling layer outputs a feature map P4 (W / 16, H / 16, 32) of 32 channels; the fifth grouped convolution layer has a 3×3 convolution kernel, and the channel grouping is 2; the sixth grouped convolution layer has a 3×3 convolution kernel, and the channel grouping is 4; the pooling window of the fourth pooling layer is 2×2; In the reverse data reorganization stage, the input is a 32-channel feature map P4, which is output through the first convolution layer with 64 channels of data, and then through the seventh group convolution layer to output a 512-channel feature P5 (W / 16, H / 16, 512). The feature P5 is reversely reorganized to restore it to 32-channel data P6 (W / 16, H / 16, 32), and then input into the second convolution layer to output a 64-channel feature P out (W / 16, H / 16, 64); wherein the seventh grouped convolutional layer has a 3×3 convolutional kernel and the channel grouping is 8; the first convolutional layer and the second convolutional layer have a 3×3 convolutional kernel.
4. The method according to claim 3, characterized in that The detection network includes a regression branch and a classification branch; both the regression branch and the classification branch are composed of a 3×3 convolution with a channel grouping of 8 and an output channel of 64.
5. The method according to claim 1, characterized in that In the target detection process of step 3, the boxes with confidence scores greater than 0.2 in the detection network output results are taken as target detection results.
6. The method according to claim 1, characterized in that When the step 2 performs group convolution, the p convolution kernels in each group convolution layer are divided into n groups according to the hardware processing capability, and p / n convolution kernels are processed in parallel each time.
7. An ultra-lightweight on-orbit real-time target detection device for remote sensing images, characterized in that: The device includes a data reorganization module, a feature extraction network and a detection network; the feature extraction network and the detection network constitute a deep learning network; The data reorganization module is used to reorganize the high-resolution single-channel image according to the input channel dimension N1 of the feature extraction network to obtain the low-resolution feature map of N1 channels; the reorganization method is: according to the set step size r, the high-resolution single-channel image is divided into r-sized 2 The block, r 2 =N1; r in the same block 2 pixels are assigned to the same position in the low-resolution feature map of N1 channels; A feature extraction network, used for performing multiple stages of grouped convolution and downsampling processing on the low-resolution feature maps of the N1 channels, and then performing reverse reorganization in a manner opposite to the data reorganization to obtain a downsampled feature map; The detection network is used to perform target detection based on the downsampled feature map from the feature extraction network to obtain a target detection result.
8. The device according to claim 7, characterized in that The feature extraction network includes a first downsampling unit, a second downsampling unit, a third downsampling unit, a fourth downsampling unit and a reverse data reorganization unit; The first downsampling unit is responsible for one-stage downsampling, including the first grouped convolution layer and the first pooling layer; the input of the first grouped convolution layer is a low-resolution feature map P0 (W, H, 16) of 16 channels, W and H represent the two-dimensional size of the feature map, and the number 16 represents 16 channels; the low-resolution feature map P0 passes through the first grouped convolution layer, outputs 16-channel data, and then passes through the first pooling layer to output a 16-channel feature map P1 (W / 2, H / 2, 16); the first grouped convolution layer has a 3×3 convolution kernel, and the channel grouping is 2; the pooling window of the first pooling layer is 2×2; The second downsampling unit is responsible for two-stage downsampling, including the second grouped convolution layer and the second pooling layer; the input of the second grouped convolution layer is the feature map P1 of 16 channels, and the second convolution layer outputs the data of 32 channels, and then passes through the second pooling layer to output the feature map P2 (W / 4, H / 4, 32) of 32 channels; the second convolution layer has a 3×3 convolution kernel and the channel grouping is 2; the pooling window of the second pooling layer is 2×2; The third downsampling unit is responsible for three-stage downsampling, including a third group convolution layer, a fourth group convolution layer and a third pooling layer; the input of the third group convolution layer is a feature map P2 of 32 channels, and the third group convolution layer outputs data of 32 channels, and then the fourth group convolution layer outputs data of 32 channels, and then the third pooling layer outputs a feature map P3 (W / 8, H / 8, 32) of 32 channels; the third group convolution layer has a 3×3 convolution kernel, and the channel grouping is 2; the fourth group convolution layer has a 3×3 convolution kernel, and the channel grouping is 4; the pooling window of the third pooling layer is 2×2; The fourth downsampling unit is responsible for four-stage downsampling, including the fifth group convolution layer, the sixth group convolution layer and the fourth pooling layer; the input of the fifth group convolution layer is the feature map P3 of 32 channels, and the fifth group convolution layer outputs the data of 32 channels, and then the sixth group convolution layer outputs the data of 32 channels, and then the fourth pooling layer outputs the feature map P4 (W / 16, H / 16, 32) of 32 channels; the fifth group convolution layer has a 3×3 convolution kernel, and the channel grouping is 2; the sixth group convolution layer has a 3×3 convolution kernel, and the channel grouping is 4; the pooling window of the fourth pooling layer is 2×2; The reverse data reorganization unit includes a first convolution layer, a seventh group convolution layer, a reverse reorganization unit and a second convolution layer; the input of the first convolution layer is a feature map P4 of 32 channels, which outputs 64 channels of data after passing through the first convolution layer, and then outputs 512 channels of features P5 (W / 16, H / 16, 512) after passing through the seventh group convolution layer; The reverse reorganization unit performs reverse data reorganization on feature P5 to restore it to 32-channel data P6 (W / 16, H / 16, 32), which is then input into the second convolutional layer; the second convolutional layer outputs 64-channel feature P out (W / 16, H / 16, 64); wherein the seventh grouped convolutional layer has a 3×3 convolutional kernel and the channel grouping is 8; the first convolutional layer and the second convolutional layer have a 3×3 convolutional kernel.