High-efficiency fine-grained target identification method in high-resolution remote sensing image
A two-stage coarse-to-fine recognition strategy with knowledge distillation and filter transplantation addresses the challenges of orientation and computational intensity in high-resolution remote sensing images, enhancing recognition efficiency and accuracy.
Patent Information
- Application Number
- CN202411167029.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-10
AI Technical Summary
Fine-grained object recognition in high-resolution remote sensing images is challenging due to objects appearing in arbitrary orientations, dense arrangements, and small pixel coverage, and existing deep learning models are computationally intensive, making them unsuitable for practical applications.
A two-stage coarse-to-fine object recognition strategy combined with knowledge distillation and filter transplantation is employed, using an oriented R-CNN detector for initial localization and CF-ORNet for fine-grained feature extraction, and a lightweight model is developed through channel-wise weighted distillation and filter grafting to enhance recognition efficiency and accuracy.
The method achieves accurate and efficient fine-grained target recognition in high-resolution remote sensing images by improving feature representation and reducing computational demands, bridging the performance gap between teacher and student networks.
Smart Images
Figure CN119107569A_ABST
Abstract
Description
Technical Field
[0001] The present application pertains to efficient fine-grained target recognition methods in high-resolution remote sensing images. Background Art
[0002] In recent years, with the rapid development of high-resolution remote sensing satellites, imaging radars, and unmanned aerial vehicle technologies, the quantity and quality of remote sensing images have been significantly improved. Fine-grained recognition of geospatial objects, i.e., automatically distinguishing object categories or subcategories in high-resolution remote sensing images (HR-RSIs), has become a challenging task with important practical significance and has been widely used in many fields, including national defense security, disaster relief, natural environment protection, and urban planning and construction.
[0003] Deep learning-based techniques have dominated the state-of-the-art accuracy standards for a variety of computer vision tasks, especially for fine-grained object recognition (FGOR) in natural images. However, in high-resolution remote sensing images (HR-RSIs), objects are usually observed from a bird's-eye view and thus exhibit scale diversity, arbitrary orientations, dense arrangements, and small pixel coverage. These characteristics pose significant challenges for fine-grained object recognition in HR-RSIs. First, geospatial objects in HR-RSIs can appear in arbitrary orientations and are highly similar between different fine-grained categories. Directly applying existing fine-grained object recognition methods is not applicable due to the limited number of pixels available for learning object features. Therefore, a rotation detector including an oriented bounding box (OBB) is required to accommodate arbitrary orientations of objects.
[0004] In addition, the network structure of the deep model for fine-grained recognition is relatively complex, and the large number of parameters and the demand for computing resources are not conducive to deployment in practical applications. This means that it is necessary to introduce model lightweight technology to reduce the computational cost of fine-grained object recognition. Knowledge distillation has become a popular method due to its excellent model compression capability. However, limited by the feature representation capability, it is difficult for the student network to break through the performance bottleneck. Therefore, how to enhance the learning effect of the student network in the knowledge distillation process and further narrow the performance gap with the teacher network has become an important challenge. To this end, the present invention proposes an efficient fine-grained target recognition method from knowledge distillation to filter transplantation. Specifically, we first use the oriented R-CNN as a two-stage object detector to achieve more accurate positioning and preliminary recognition of targets in any direction. On this basis, a coarse-to-fine recognition strategy is designed to improve the fine-grained representation capability of target features. In addition, we also propose a lightweight model that combines knowledge distillation with filter transplantation to accelerate fine-grained target recognition in HR-RSIs. Summary of the invention
[0005] In view of the above challenges, the present invention focuses on developing an efficient fine-grained target recognition method for high-resolution remote sensing images. The overall structure of the present invention consists of two parts: a fine-grained object recognition module and a lightweight model module, such as Figure 1 As shown in Figure 2. In Part A, we design a two-stage fine-grained object recognition network, including rotation object detection and coarse-to-fine object recognition network (CF-ORNet). In Part B, we combine knowledge distillation and filter transplantation into a lightweight fine-grained object recognition model.
[0006] Specifically, in the two-stage fine-grained object recognition process in Part A:
[0007] The first stage is to obtain the accurate location and preliminary classification of objects in any orientation by adopting the Oriented R-CNN detector.
[0008] The second stage is to build CF-ORNet, which achieves fine-grained object recognition by deeply mining the features of each object and correcting the wrong categories in the initial recognition.
[0009] In part B, the model is lightweighted by following the steps below:
[0010] The two teacher networks in Part A are utilized to guide an efficient student network via knowledge distillation.
[0011] Filter transplantation is performed during the process of knowledge distillation to promote mutual learning between student networks and further improve recognition performance and efficiency.
[0012] The technical solution of the present invention is: an efficient fine-grained target recognition method in high-resolution remote sensing images, which is special in that it includes the following steps:
[0013] In order to solve the problem of poor performance in fine-grained recognition of dense objects due to not considering the location of geospatial objects during target detection, and lack of reliability and stability in multi-object recognition. Fine-grained object recognition methods need to mine more fine-grained content for each object. A feasible approach is to combine object localization with fine-grained classification by implementing a coarse-to-fine or repeated classification strategy, thereby improving the recognition ability and representation granularity of object features. Therefore, the present invention proposes a coarse-to-fine recognition strategy, and designs a fine-grained target recognition network in HR-RSIs to fully extract the fine-grained features of each target and correct the misclassification in the initial recognition.
[0014] In order to solve the problem that deep models are difficult to directly apply to practical engineering due to their complex structure, large number of parameters and dependence on computing power, some lightweight technologies such as model compression and mutual learning have been proposed to reduce the number of parameters of deep models while maintaining an appropriate performance level. Considering that filter transplantation is a model interaction training method, if used in combination with it, it can further narrow the performance gap between the student network and the teacher network. Therefore, the present invention proposes an efficient fine-grained object recognition model from knowledge edge distillation to filter transplantation, aiming to ensure accuracy while speeding up the reasoning process.
[0015] The dataset is established as follows: The HRSC2016 dataset contains 1061 images with sizes ranging from 300×300 to 1500×900, with a total of 2976 ship targets. This study identified 22 categories of fine-grained ship targets. According to the HRSC2016 official dataset division scheme, 436 images for training, 181 images for verification, and 438 images for testing were obtained. The VEDAI dataset contains 1210 remote sensing images with a size of 1024×1024, with a total of 3640 vehicle objects with orientation box annotations. This invention identifies six types of fine-grained vehicle targets. The VEDAI dataset is divided into training and test sets in a ratio of 4:1, that is, 968 images for training and 242 images for testing. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 The fine-grained object recognition and lightweight model provided by the present invention;
[0017] Figure 2 The reasoning process of CF-ORNet provided by the present invention;
[0018] Figure 3 A lightweight model from knowledge extraction to filter grafting provided by the present invention;
[0019] Figure 4 Details of filter grafting between student networks provided by the present invention. DETAILED DESCRIPTION
[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present application.
[0022] The present invention designs a two-stage coarse-to-fine object recognition strategy for fine-grained object recognition. In the first stage, an oriented R-CNN detector is used for object localization and preliminary classification. The main role of oriented R-CNN is to act as a baseline rotation detector to achieve accurate localization and as a teacher network to guide the compaction process of subsequent networks. In the second stage, a coarse-to-fine strategy is implemented to construct CF-ORNet (Coarse-to-Fine Object Recognition Network) for fine-grained feature representation and category correction.
[0023] The working process of the first stage for the R-CNN teacher network is as follows:
[0024] Step 1: Extract multi-scale features P{P2,P3,P4,P5,P6} from Block 1 to Block 4 through the ResNet-101 backbone network-FPN, and form the feature M∈R H×W×256 .
[0025] Step 2: Transform M∈R through a 1×1 convolution H×W×256 Compressed to R H×W×64 Finally, the oriented RPN generates a series of high-quality oriented proposals (x, y, w, h, Δα, Δβ) in an almost cost-free manner.
[0026] Step 3: Using the multi-scale features P{P2,P3,P4,P5,P6} and the oriented proposals (FC1 and FC2), the oriented R-CNN head is designed to refine the candidate regions (ROIs) and identify them through rotating RoIAlign and two fully connected layers (see Figure 1 ).
[0027] Step 4: Through regression and classification, the detection results mainly include two parts: 1) Using the precise location of the geospatial target (t x ,t y ,t w ,t h ,t θ ) to extract targets from remote sensing images (RSIs); 2) initial categories and confidences (c0, c1, c2, …, c N ) will provide experience reference for the classification of subsequent modules.
[0028] In the second stage, based on the localization and preliminary classification results of the rotation detector, CF-ORNet achieves fine-grained object recognition by implementing a coarse-to-fine recognition strategy. This strategy is divided into three modules: object extraction, coarse object recognition, and fine-grained category correction (e.g. Figure 2 ). Object extraction: CF-ORNet’s object extraction includes steps such as OBB (oriented bounding box) conversion, cropping, rotation, and padding, where the position and initial category information is provided by the rotation detector. Specifically:
[0029] Step 1: By analyzing the position and initial category information G i The position result is converted into an oriented bounding box (OBB). x and y are the horizontal and vertical coordinates of the center point of the OBB (oriented bounding box), w is the length of the long side, h is the length of the short side, and θ is the angle between the side length and the horizontal positive semi-axis, and its value range is [-π / 4,-3π / 4].
[0030] OBB=(x,y,w,h,θ)
[0031]
[0032] Step 2: Crop the located objects into an object patch O0, and record the category and confidence of each object. The confidence here reflects the degree of certainty that the object belongs to a certain category and the quality of object positioning.
[0033] Step 3: Considering that OBB in any direction (x, y, w, h, θ) is difficult to classify in fine-grained manner, rotate it to the horizontal direction (a x ,a y ,b x ,b y ,c x ,c y ,d x ,d y ), which can effectively reduce the impact and reduce the learning difficulty of the classification network. By calculating the rotation matrix R(θ) of each OBB, the object O1 in the horizontal direction can be obtained.
[0034] Step 4: Since the input size of the classification network is usually set to 224px×224px, it is necessary to scale and pad the long and short sides of each OBB to 224px by non-distortion padding to generate an image block O2 of standardized size.
[0035]
[0036] The performance of deep networks usually benefits from a large number of parameters, but their complex structure and dependence on computational cost lead to a "balance" problem between accuracy and speed in large networks. This paper aims to improve the speed and efficiency of fine-grained object recognition while ensuring appropriate accuracy, thus achieving a balance between accuracy and speed. Figure 3 As shown,
[0037] Considering that both localization accuracy and classification accuracy require a lot of computing resources, the knowledge distillation methods of compressed weight difference (CWD) and weakly supervised localized difference (WSLD) are used to combine the knowledge of object detection and fine-grained classification, so as to train a model with fewer parameters and faster recognition speed. However, this method inevitably degrades the performance of lightweight models. In order to bridge this performance gap, a filter grafting method is proposed during the training process of the student network. Specifically, the following steps are taken for channel-wise knowledge extraction: CWD is designed for R-CNN type object detectors and is implemented as follows:
[0038] Step 1: Input the randomly selected image (RSI) into the teacher and student detectors at the same time to generate the corresponding feature maps M respectively. T and M S .
[0039] Step 2: Referring to the design concept of the softmax function, use the following formula to represent the probability distribution of feature maps on different channels, and calculate the KL divergence based on it. In order to ensure the effective transfer of knowledge, softmax normalization is used to eliminate the impact of different scales between large networks and compact networks. Since softmax normalization is conducive to the knowledge distillation process, we define the probability distribution Φ(M) as follows:
[0040]
[0041] Among them, M c,i represents the value of the output feature map at the i-th spatial position in the c-th channel; c∈[1,C] is the total number of channels in the feature map i∈[1,W·H]; W and H represent the width and height of the feature map, respectively; and τ is an adjustable hyperparameter (usually called "temperature"). By converting the original feature map M into a probability distribution Φ(M), we can more conveniently measure M T and M S KL divergence distance between two distributions The goal of distillation is to minimize That is, the gap between the student network and the teacher network. Therefore, when the teacher network produces a probability distribution The probability distribution of the student network at the same location Should be close To enable teachers to provide guidance to students on the Internet.
[0042]
[0043] Step 3: Distance between teachers and students Can be used as the loss function L cwd The knowledge distillation penalty term in the student network is related to the loss function L self (G,M S ) together constitute the overall loss function. The overall loss function is defined as follows:
[0044] Where α is a hyperparameter used to balance the knowledge distillation loss and the loss of the student network itself.
[0045] In short, the lightweight detector gains the ability to accelerate inference through Channel-wise Weighted Distillation (CWD) and improves its object recognition performance under the guidance of a teacher network.
[0046] Weighted Soft Label Distillation (WSLD) is specially designed for CF-ORNet. The specific steps are as follows:
[0047] Step 1: For the same object block O2, the teacher and student networks will obtain two classification vectors y respectively T and s , used to distinguish image categories.
[0048] Step 2: The weighted soft label (WSL) loss is the product of two multiplicative terms, where the first term is a scaling factor WSL that is dynamically adjusted based on the teacher and student predictions; while the second term is the general knowledge distillation loss L kd , which uses the classification results of the teacher network to guide the training of the student network. The WSL loss is defined as:
[0049]
[0050] where y k Т and y k S represents the kth element of the classification vector output by the teacher and the student, τ is a hyperparameter, LSCE and LTCE are the cross entropy loss functions of the student and the teacher, respectively. Step 3: Combine the knowledge distillation loss and the student's self-learning loss and perform iterative optimization during the training process. The final loss function is
[0051]
[0052] In the process of knowledge distillation, a hyperparameter α is introduced to balance the loss of knowledge distillation and the loss of the student network itself. The knowledge of the teacher network is transferred to ResNet-18 or other effective student networks through weighted soft label knowledge distillation (WSLD), so that the student network can better mine the discriminative feature information of the object.
[0053] In order to further improve the performance of the student network and reduce the parameters and computational complexity, filter grafting is used. This is a multi-network parallel training strategy for compressed deep neural networks, which can reduce the number of parameters and computational cost without sacrificing model accuracy.
[0054] However, the student (compressed) network using knowledge distillation usually has limitations in feature representation compared to the teacher (large) network. To bridge the gap between the teacher and student networks, we introduce filter grafting between the two student networks of the detector and classifier to keep the original accuracy as much as possible. This process is shown in Figure 4 In this way, not only the performance of the student network is improved, but also the compactness and efficiency of the model are guaranteed.
[0055] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0056] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.
Claims
1. An efficient fine-grained target recognition method in high-resolution remote sensing images, characterized in that: include: Step 1: Establishment of the dataset. The HRSC2016 dataset contains 1061 images with sizes ranging from 300×300 to 1500×900, including a total of 2976 ship targets. 22 types of fine-grained ship targets are identified. The HRSC2016 official dataset division scheme is adopted to obtain 436 images for training, 181 images for verification, and 438 images for testing. The VEDAI dataset contains 1210 RSIs with a size of 1024×1024 and 3640 vehicle objects with orientation box annotations. Six types of fine-grained vehicle targets are identified. The VEDAI dataset is divided into a training set and a test set in a ratio of 4:1, that is, 968 images for training and 242 images for testing. The remote sensing images of the training set are preprocessed to obtain a remote sensing image dataset after data enhancement. Step 2: Build an efficient fine-grained target recognition network in high-resolution remote sensing images. The efficient fine-grained target recognition network in high-resolution remote sensing images includes designing a two-stage fine-grained target recognition network in Part A, including rotation target detection and coarse-to-fine target recognition network (CF-ORNet); in Part B, knowledge distillation and filter grafting are combined into a lightweight fine-grained target recognition model.
2. The efficient fine-grained target recognition method in high-resolution remote sensing images according to claim 1, characterized in that: Pre-train the network structure and optimize the parameters of the efficient fine-grained target recognition network in the high-resolution remote sensing image built in step 2.
3. The efficient fine-grained target recognition method in high-resolution remote sensing images as claimed in claim 2, characterized in that: The preprocessed test set data set is input into the efficient fine-grained target recognition network in the high-resolution remote sensing image that has been trained in step 3, and the accurate classification results of the remote sensing image are output.
4. The efficient fine-grained target recognition method in high-resolution remote sensing images according to claim 3, characterized in that: Part A is a two-stage fine-grained object recognition, where the first stage is to obtain the accurate location and preliminary classification results of any oriented object by using the oriented R-CNN detector; The second stage is to build CF-ORNet, which fully mines the characteristics of each target and corrects the wrong categories in the initial recognition to achieve fine-grained target recognition.
5. The efficient fine-grained target recognition method in high-resolution remote sensing images as claimed in claim 4, characterized in that: In the first stage of target detection, the oriented R-CNN detector is used to obtain the accurate location and preliminary classification results of arbitrarily oriented objects. Object positioning and fine-grained classification are combined by enforcing a coarse-to-fine or repeated classification strategy. Based on the coarse-to-fine recognition strategy, a fine-grained target recognition network is designed in HR-RSIs to extract the fine-grained features of each target and correct the misclassification in the preliminary recognition.
6. The efficient fine-grained target recognition method in high-resolution remote sensing images as claimed in claim 5, characterized in that: By fully mining the features of each target and correcting the wrong categories in the initial recognition, fine-grained target recognition is achieved.