Lightweight method and system for rotating target detection based on knowledge distillation

Abstract

The present invention relates to a lightweight method and system for rotating target detection based on knowledge distillation, the method comprising respectively decoding the coordinate encoding of the detection target output by the student model and the trained teacher model into the form of rotating coordinates; converting the rotating coordinates into a two-dimensional Gaussian distribution; calculating the KL divergence of the Gaussian distribution representation of the detection target output by the student model and the teacher model, and calculating the KL divergence between the Gaussian distribution representation of the detection target output by the student model and the Gaussian distribution representation of the detection target obtained by the label true frame; normalizing the KL divergence; calculating the overall loss function of the teacher model in the process of knowledge distillation of the student model according to the normalized KL divergence; and using the distilled student model for prediction. The present invention provides more accurate positioning information for the training of lightweight networks and improves the detection performance of lightweight networks on optical remote sensing images.

Description

Technical Field

[0001] The present invention relates to the technical field of high-resolution remote sensing image information processing, and in particular to a lightweight method and system for detecting rotating targets based on knowledge distillation. Background Art

[0002] Object detection in remote sensing images aims to locate objects of interest (such as vehicles, aircraft, and ships) on the surface of the earth and predict their categories. Remote sensing images are mostly viewed from a bird's-eye view, and the spatial scenes they contain are larger and more complex. The distribution of objects of interest is uneven, and there are sparse and dense distributions. Remote sensing target detection has the difficulties of small objects, dense distribution, and arbitrary directions. When the length-to-width ratio of the object of interest is large, there is an inclination, and it is closely arranged, using only a horizontal detection frame, the size and length-to-width ratio cannot truly reflect the target object, the target and background pixels cannot be effectively separated, and dense objects are difficult to separate. Each detection frame will contain adjacent objects, and the intersection-union ratio between horizontal detection frames is high, which is easily suppressed in the non-maximum suppression process, resulting in low final detection accuracy. Since the target orientation, that is, the rotation angle, has periodic characteristics, the design of a suitable loss function is crucial for model training optimization.

[0003] Traditional target detection methods often require complex manual feature extraction and fine parameter adjustment, and the model has poor generalization and cannot adapt to the ever-changing environment. In recent years, with the improvement of hardware computing power, the development of remote sensing imaging technology and the convenience of remote sensing image resources, deep learning technology has been widely developed, especially the convolutional neural network model, which has strong generalization performance due to its powerful robust feature extraction ability, function fitting ability and end-to-end network model design. Based on the convolutional neural network model, most of the most advanced target detection methods focus on designing advanced network structures or loss functions to improve detection performance, but this method has expensive computational costs. Therefore, some subsequent methods are committed to improving detection performance by using lightweight methods to distill the knowledge in the (teacher) large model. For example, the knowledge distillation method based on the intermediate layer features has proved the importance of the intermediate layer features in the training and prediction process of the detection network, but if only the intermediate layer features are distilled, the information of the coordinate space and the category space will be lost, which will affect the final detection results; there is also knowledge distillation based on generalized distribution information, which uses classification loss to solve regression problems, including rotating the target position and rotating the target angle, and solving the positioning uncertainty problem, but it has quantization errors. Converting the rotated coordinates into Gaussian distribution representation has been widely used in the field of rotation detection in recent years, but there is little research on the lightweight method of detectors. Therefore, it is urgent to propose a lightweight method for rotating target detection based on knowledge distillation to overcome the above technical defects of the existing technology. Summary of the invention

[0004] To this end, the technical problem to be solved by the present invention is to overcome the technical defects existing in the prior art, and propose a lightweight method and system for rotation target detection based on knowledge distillation, which uses the coordinate encoding output of the teacher and student models for knowledge distillation, calculates the KL divergence between the Gaussian distribution representations of the rotation coordinates as the added distillation loss to train the student model parameters, provides more accurate positioning information for the training of the lightweight network, and improves the detection performance of the lightweight network on optical remote sensing images.

[0005] In order to solve the above technical problems, the present invention provides a lightweight method for rotating target detection based on knowledge distillation, comprising the following steps:

[0006] S1: Decoding the coordinate encodings of the detection target output by the student model and the trained teacher model into rotation coordinate forms respectively to obtain the rotation coordinate representation of the detection target;

[0007] S2: converting the rotation coordinate representation into a two-dimensional Gaussian distribution to obtain a Gaussian distribution representation of the detection target;

[0008] S3: Calculate the KL divergence of the Gaussian distribution representation of the detection target output by the student model and the teacher model to obtain a first KL divergence, and calculate the KL divergence between the Gaussian distribution representation of the detection target output by the student model and the Gaussian distribution representation of the detection target obtained by the label true frame to obtain a second KL divergence;

[0009] S4: normalizing the first KL divergence and the second KL divergence;

[0010] S5: According to the normalized first KL divergence and the second KL divergence, the overall loss function of the teacher model in the process of knowledge distillation of the student model is calculated;

[0011] S6: Prediction using the distilled student model.

[0012] In one embodiment of the present invention, in step S2, the method of respectively decoding the coordinate encodings of the detection target output by the student model and the trained teacher model into the form of rotation coordinates includes:

[0013] Decode the coordinate encoding Y of the detected target output by the student model and the trained teacher model: (dx, dy, dw, dh, dθ) into the form of rotated coordinates (x, y, w, h, θ), where:

[0014] x=x a +dx*w a

[0015] y=ya +dy*h a

[0016] w=wa*e dw

[0017] h=h a *e dh

[0018] θ=θ a +dθ

[0019] In the formula, A:(x a ,y a , w a ,h a ,θ a ) is the preset anchor box of the model.

[0020] In one embodiment of the present invention, in step S3, the method of converting the rotated coordinate representation into a two-dimensional Gaussian distribution includes:

[0021] Represent the rotated coordinates (x, y, w, h, θ) is converted into a two-dimensional Gaussian distribution The calculation formula of (μ, ∑) is:

[0022] μ=(x,y) T

[0023]

[0024] Where (x, y, w, h, θ) is the rotation coordinate representation of the detection target, which respectively represents the horizontal coordinate, vertical coordinate, width, height and rotation angle of the detection target center point, and (μ, ∑) represents the mean and covariance matrix of the Gaussian distribution.

[0025] In one embodiment of the present invention, in step S4, the method of calculating the KL divergence of the Gaussian distribution representation of the detection target output by the student model and the teacher model includes:

[0026] The calculation formula of the KL divergence represented by the Gaussian distribution of the detection target output by the student model and the teacher model is:

[0027]

[0028] In the formula, (x S ,y S , w S ,h S ,θ S ), (μ S ,∑ S )and (x T ,y T , w T ,h T ,θ T ), (μ T ,∑ T ) represent the rotation coordinate representation and Gaussian distribution representation of the detection target output by the student model and the teacher model respectively, Δx = x S -x T , Δy=y S -y T , Δθ=θ S -θ T They respectively represent the difference in horizontal and vertical coordinates and the difference in rotation angle between the center point of the detected target output by the student model and the label real box.

[0029] In one embodiment of the present invention, in step S4, a method for calculating the KL divergence between the Gaussian distribution representation of the detection target output by the student model and the Gaussian distribution representation of the detection target obtained by the label true frame includes:

[0030] The calculation formula for the KL divergence between the Gaussian distribution representation of the detection target output by the student model and the Gaussian distribution representation of the detection target obtained by the label real box is:

[0031]

[0032] In the formula (x S ,y S , w S ,h S ,θ S ), (μ S ,∑ S )and (x G ,y G , w G ,h G ,θ G ), (μ G ,∑ G ) represent the detection target output by the student model and the rotation coordinate representation and Gaussian distribution representation of the detection target obtained by the label real box, respectively, Δx = x S -x G , Δy=y S -y G , Δθ=θ S -θ GThey respectively represent the difference in horizontal and vertical coordinates and the difference in rotation angle between the center point of the detection target output by the student model and the detection target obtained by the label real frame.

[0033] In one embodiment of the present invention, in step S7, after prediction is performed using the distilled student model, the prediction box output by the student model is post-processed.

[0034] In one embodiment of the present invention, in step S7, the method for post-processing the prediction box output by the student model includes: the specific steps of post-processing are as follows:

[0035] Arrange all prediction boxes in each category in ascending or descending order by score;

[0036] Select the prediction box with the highest score as the benchmark, and calculate the intersection and union ratio of other prediction boxes with it;

[0037] The prediction boxes in each category are screened according to the intersection-and-union ratio, and the prediction boxes that do not meet the screening conditions are deleted, where the screening condition is that the intersection-and-union ratio of the prediction box to be screened and the reference prediction box is less than or equal to a preset threshold.

[0038] Among the remaining prediction boxes that meet the screening conditions, select the prediction box with the highest score as the benchmark box, and repeat the above screening steps until all prediction boxes that meet the screening conditions are used as benchmarks for screening.

[0039] In addition, the present invention also provides a lightweight system for rotating target detection based on knowledge distillation, comprising the following steps:

[0040] A coordinate decoding module, which is used to decode the coordinate encoding of the detection target output by the student model and the trained teacher model into a rotation coordinate form, so as to obtain a rotation coordinate representation of the detection target;

[0041] A coordinate conversion module, which is used to convert the rotation coordinate representation into a two-dimensional Gaussian distribution to obtain a Gaussian distribution representation of the detection target;

[0042] A KL divergence calculation module is used to calculate the KL divergence of the Gaussian distribution representation of the detection target output by the student model and the teacher model to obtain a first KL divergence, and to calculate the KL divergence between the Gaussian distribution representation of the detection target output by the student model and the Gaussian distribution representation of the detection target obtained by the label true frame to obtain a second KL divergence;

[0043] A normalization processing module, used for performing normalization processing on the first KL divergence and the second KL divergence;

[0044] A loss function calculation module, which is used to calculate the overall loss function in the process of knowledge distillation of the teacher model to the student model according to the first KL divergence and the second KL divergence after normalization;

[0045] The prediction module is used to make predictions using the distilled student model.

[0046] In one embodiment of the present invention, in a coordinate conversion module, a method for converting the rotation coordinate representation into a two-dimensional Gaussian distribution includes:

[0047] Represent the rotated coordinates (x, y, w, h, θ) is converted into a two-dimensional Gaussian distribution The calculation formula of (μ, ∑) is:

[0048] μ=(x,y) T

[0049]

[0050] Where (x, y, w, h, θ) is the rotation coordinate representation of the detection target, which respectively represents the horizontal coordinate, vertical coordinate, width, height and rotation angle of the detection target center point, and (μ, ∑) represents the mean and covariance matrix of the Gaussian distribution.

[0051] In one embodiment of the present invention, in the KL divergence calculation module, a method for calculating the KL divergence of the Gaussian distribution representation of the detection target output by the student model and the teacher model includes:

[0052] The calculation formula of the KL divergence represented by the Gaussian distribution of the detection target output by the student model and the teacher model is:

[0053]

[0054] In the formula (x S ,y S , w S ,h S ,θ S ), (μ S ,∑ S )and (x T ,y T , w T ,h T ,θ T ), (μ T ,∑ T ) represent the rotation coordinate representation and Gaussian distribution representation of the detection target output by the student model and the teacher model respectively, Δx = xS -x T , Δy=y S -y T , Δθ=θ S -θ T They respectively represent the difference in horizontal and vertical coordinates and the difference in rotation angle between the center point of the detected target output by the student model and the label real box.

[0055] The above technical solution of the present invention has the following advantages compared with the prior art:

[0056] The present invention discloses a lightweight method and system for rotating target detection based on knowledge distillation, which utilizes the coordinate encoding outputs of teacher and student models for knowledge distillation, calculates the KL divergence between the Gaussian distribution representations of the rotating coordinates as the added distillation loss to train the student model parameters, provides more accurate positioning information for the training of the lightweight network, and improves the detection performance of the lightweight network on optical remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein

[0058] Figure 1 This is a diagram of the training process of the lightweight method for rotating target detection based on knowledge distillation proposed in this invention.

[0059] Figure 2 This is the prediction flow chart of the lightweight method for rotating target detection based on knowledge distillation proposed in the present invention.

[0060] Figure 3 The detection accuracy of the distillation method proposed in the present invention and other distillation methods based on ResNet18-FPN-RetinaNet on the HRSC2016 dataset changes with the number of training rounds after 20 rounds, where w / od is the original network without distillation; FitNets is a distillation method based on all intermediate layer features; DeFeat is a distillation method based on decoupled features; LD is a distillation method based on generalized distribution information between coordinates; KLDD is a distillation method based on KL divergence between Gaussian distributions of coordinates proposed in the present invention.

[0061] Figure 4The detection accuracy of the distillation method proposed in the present invention and other distillation methods based on MobileNetv2-FPN-RetinaNet on the HRSC2016 dataset changes with the number of training rounds after 20 rounds, where w / od is the original network without distillation; FitNets is a distillation method based on all intermediate layer features; DeFeat is a distillation method based on decoupled features; LD is a distillation method based on generalized distribution information between coordinates; KLDD is a distillation method based on KL divergence between Gaussian distributions of coordinates proposed in the present invention.

[0062] Figure 5 The following is a comparison of the detection results obtained by different distillation methods on the HRSC2016 dataset, where Figure (5a) is the original annotation of the dataset, Figure (5b) is the detection result of the original ResNet18-FPN-RetinaNet detector without using any distillation method, Figure (5c) is the detection result of the FitNets distillation method based on all intermediate layer features, Figure (5d) is the detection result of the DeFeat distillation method based on the decoupled features of the target area, Figure (5e) is the detection result of the LD distillation method based on the generalized distribution information between the target position coordinates, and Figure (5f) is the detection result of the proposed KLDD distillation method based on the KL divergence between the coordinate Gaussian distributions. The cyan box in the figure indicates the detected true target (TP), the yellow box indicates the detected false target (FP), and the red box indicates the missed detection target (FN). DETAILED DESCRIPTION

[0063] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0064] The embodiment of the present invention provides a lightweight method for rotating target detection based on knowledge distillation, comprising the following steps:

[0065] S1: Decoding the coordinate codes of the detection target output by the student model and the teacher model into rotation coordinate forms respectively to obtain the rotation coordinate representation of the detection target;

[0066] S2: converting the rotation coordinate representation into a two-dimensional Gaussian distribution to obtain a Gaussian distribution representation of the detection target;

[0067] S3: Calculate the KL divergence of the Gaussian distribution representation of the detection target output by the student model and the teacher model to obtain a first KL divergence, and calculate the KL divergence between the Gaussian distribution representation of the detection target output by the student model and the Gaussian distribution representation of the detection target obtained by the label true frame to obtain a second KL divergence;

[0068] S4: normalizing the first KL divergence and the second KL divergence;

[0069] S5: According to the normalized first KL divergence and the second KL divergence, the overall loss function of the teacher model in the process of knowledge distillation of the student model is calculated;

[0070] S6: Prediction using the distilled student model.

[0071] The present invention discloses a lightweight method for rotating target detection based on knowledge distillation, which utilizes the coordinate encoding outputs of teacher and student models for knowledge distillation, calculates the KL divergence between the Gaussian distribution representations of the rotating coordinates as the added distillation loss to train the student model parameters, provides more accurate positioning information for the training of the lightweight network, and improves the detection performance of the lightweight network on optical remote sensing images.

[0072] Please refer to Figure 1 and Figure 2 As shown in the figure, the above-mentioned lightweight method for rotating target detection based on knowledge distillation mainly includes the following three parts:

[0073] (1) Train a deep convolutional neural network with strong detection accuracy as the teacher model:

[0074] This part aims to generate a teacher model that provides additional supervision information for student model training. This experiment chooses ResNet50-FPN as the feature extraction backbone network and RetinaNet as the teacher model of the detection network.

[0075] (2) KLDD distillation method based on KL divergence between coordinate Gaussian distributions:

[0076] This part aims to generate a two-dimensional Gaussian distribution representation of the rotated coordinates and calculate the KL divergence between Gaussian distributions. The specific steps include: coordinate decoding, Gaussian distribution conversion, and loss calculation of the KL divergence between Gaussian distributions.

[0077] Among them, the specific coordinate decoding calculation formula is:

[0078] x=x a +dx*w a

[0079] y=y a +dy*h a

[0080] w=w a *e dw

[0081] h=h a *e dh

[0082] θ=θ a +dθ

[0083] Get the decoded rotation coordinate representation (x, y, w, h, θ), where Y: (dx, dy, dw, dh, dθ) is the coordinate encoding of the model output, A: (x a ,y a , w a ,h a ,θ a ) is the preset anchor box of the model.

[0084] The rotation coordinates are expressed as (x, y, w, h, θ) is converted into a two-dimensional Gaussian distribution The specific calculation formula of (μ, ∑) is as follows:

[0085] μ=(x,y) T

[0086]

[0087] Where (x, y, w, h, θ) is the rotation coordinate representation of the detection target, which respectively represents the horizontal coordinate, vertical coordinate, width, height and rotation angle of the detection target center point, and (μ, ∑) represents the mean and covariance matrix of the Gaussian distribution.

[0088] Among them, the calculation formula of the KL divergence represented by the Gaussian distribution of the detection target output by the student model S and the teacher model T is:

[0089]

[0090] In the formula, (x S ,y S , w S ,h S ,θ S ), (μ S ,∑ S )and (x T ,y T , w T ,h T ,θ T ), (μ T ,∑ T ) represent the rotation coordinate representation and Gaussian distribution representation of the detection target output by the student model and the teacher model respectively, Δx = x S -x T , Δy=y S -yT , Δθ=θ S -θ T They respectively represent the difference in horizontal and vertical coordinates and the difference in rotation angle between the center point of the detected target output by the student model and the label real box.

[0091] Among them, the calculation formula of the KL divergence between the Gaussian distribution representation of the detection target output by the student model S and the Gaussian distribution representation of the detection target of the label real box G is:

[0092]

[0093] In the formula (x S ,y S , w S ,h S ,θ S ), (μ S ,∑ S )and (x G ,y G , w G ,h G ,θ G ), (μ G ,∑ G ) represent the detection target output by the student model and the rotation coordinate representation and Gaussian distribution representation of the detection target obtained by the label real box, respectively, Δx = x S -x G , Δy=y S -y G , Δθ=θ S -θ G They respectively represent the difference in horizontal and vertical coordinates and the difference in rotation angle between the center point of the detection target output by the student model and the detection target obtained by the label real frame.

[0094] Normalize the KL divergence between Gaussian distributions F(KL):

[0095]

[0096] Get the final KL divergence loss calculation formula between the coordinate parameter Gaussian distribution:

[0097]

[0098]

[0099] During training, the overall loss function of the model is for:

[0100]

[0101] Among them, λ0, λ1, and λ2 are weight parameters, which are set to 1, 15, and 5.5 respectively; is the positive sample region mask, which is determined by the intersection-over-union ratio of the anchor box and the true target box; C S , C T , C G The category confidence and label true category unique encoding output by the student and teacher models respectively; The rotated coordinates and true label coordinates output by the student and teacher models respectively; is the cross entropy loss; is the KL divergence loss between category distributions; and is the normalized KL divergence between the student-truth and student-teacher Gaussian distributions.

[0102] (3) In the prediction stage, only the distilled student model is used for prediction.

[0103] This part aims to post-process the prediction box output by the network, that is, non-maximum suppression, to avoid repeated detection of the same target. The specific steps of post-processing are as follows:

[0104] Arrange all prediction boxes in each category in ascending or descending order by score;

[0105] Select the prediction box with the highest score as the benchmark, and calculate the intersection and union ratio of other prediction boxes with it;

[0106] The prediction boxes in each category are screened according to the intersection-and-union ratio, and the prediction boxes that do not meet the screening conditions are deleted, where the screening condition is that the intersection-and-union ratio of the prediction box to be screened and the reference prediction box is less than or equal to a preset threshold.

[0107] Among the remaining prediction boxes that meet the screening conditions, select the prediction box with the highest score as the benchmark box, and repeat the above screening steps until all prediction boxes that meet the screening conditions are used as benchmarks for screening.

[0108] The beneficial effects of a lightweight method for rotating target detection based on knowledge distillation provided by the present invention are explained below in an experimental verification manner.

[0109] Example:

[0110] 1. Experimental data

[0111] In order to verify the effectiveness of the lightweight detection model method proposed in this paper, this experiment used two optical remote sensing rotating target detection datasets: DOTA and HRSC2016.

[0112] 1) DOTA dataset:

[0113] DOTA is one of the largest datasets for rotating object detection in aerial images. It contains 2806 aerial images with image sizes ranging from 800×800 to 4000×4000 pixels. It contains more than 188,000 objects of various scales, orientations, and shapes, marked with arbitrary quadrilaterals, and contains 15 object categories: airplanes, baseball fields, bridges, ground tracks, small vehicles, large vehicles, ships, tennis courts, basketball courts, storage tanks, football fields, roundabouts, ports, swimming pools, and helicopters. The training set, validation set, and test set are divided into 1 / 2, 1 / 6, and 1 / 3 ratios. Due to the large size of the image, it is cropped into 1024×1024 pixel sub-images with an overlap of 200 pixels between sub-images.

[0114] 2) HRSC2016 dataset:

[0115] HRSC2016 is a single-category dataset for ship detection. Its images come from 6 well-known ports and include two scenes: ships at sea and near the coast. The spatial resolution of the images is 0.4-2m. The dataset has a total of 1070 images and 2976 ship targets. The training set, validation set, and test set include 436, 181, and 453 images, respectively. Due to the different sizes of the images, they are scaled to 800×512 pixels. Figure (5a) shows some of the images in the dataset and the target box annotations.

[0116] 2. Experimental results

[0117] This experiment is built based on the MMRotate environment and conducted on a single NVIDIA RTX3090 GPU. This experiment selects the RetinaNet network with ResNet50-FPN as the backbone as the teacher model, and the student models select the RetinaNet network with ResNet18-FPN and MobileNetV2-FPN as the backbone. In the training phase, random horizontal, vertical, and diagonal flipping are used as data augmentation methods, and the standard momentum optimizer (Momentum) is selected. The weight decay and momentum are 0.0001 and 0.9 respectively. The initial learning rate is set to 0.0025. A total of 24 epochs are trained for the DOTA dataset, and a total of 72 epochs are trained for the HRSC2016 dataset. The batch size is 2. In the test phase, mAP is used 50 (The average precision when the intersection-over-union ratio threshold for determining the true target is 0.5) is used as an indicator to evaluate the accuracy of the detection model.

[0118] 1) DOTA dataset:

[0119] Tables 1 and 2 show the experimental results obtained using different comparative distillation methods, including the benchmark teacher model ResNet50-FPN, the benchmark student model ResNet18-FPN, and the benchmark student model MobileNetv2-FPN. From the results in the table, it can be seen that the overall accuracy of the proposed method (KLDD) is improved from 65.2% and 56.8% to 68.1% and 61.0% respectively compared with the benchmark student models (ResNet18-FPN and MobileNetv2-FPN), and it is better than the detection results of the current mainstream distillation methods (FitNets, DeFeat, GIImitation, and LD, etc.), with the overall accuracy increased by 4.45% and 7.39% respectively.

[0120] Table 3 shows the accuracy (mAP) of the two lightweight models ResNet18-FPN-RetinaNet and MobileNetv2-FPN-RetinaNet and the teacher model ResNet50-FPN-RetinaNet selected in this experiment. 50 ), floating point operations (FLOPS) and model parameters (Params), and the image size of the input network is set to 3x1024x1024.

[0121] Table 1 Experimental results of different distillation methods on the DOTA dataset

[0122]

[0123] Table 2 Experimental results of different distillation methods on the DOTA dataset

[0124]

[0125] Table 3 Experimental results before and after using the KLDD distillation method on the DOTA dataset

[0126]

[0127] 2) HRSC2016 dataset:

[0128] Tables 4 and 5 show the experimental results obtained using different comparative distillation methods, including the benchmark teacher model ResNet50-FPN, the benchmark student model ResNet18-FPN, and the benchmark student model MobileNetv2-FPN. From the results in the table, it can be seen that the overall accuracy of the proposed method (KLDD) is improved from 86.0% and 79.2% to 89.2% and 80.7% respectively compared with the benchmark student models (ResNet18-FPN and MobileNetv2-FPN), and it is better than the detection results of the current mainstream distillation methods (FitNets, DeFeat, GIImitation, and LD, etc.), with the overall accuracy increased by 3.72% and 1.89% respectively.

[0129] Table 6 shows the accuracy (mAP) of the two lightweight models ResNet18-FPN-RetinaNet and MobileNetv2-FPN-RetinaNet and the teacher model ResNet50-FPN-RetinaNet selected in this experiment. 50 ), floating point operations (FLOPS) and model parameters (Params), and the image size of the input network is set to 3x1024x1024.

[0130] Table 4 Experimental results of different distillation methods on the HRSC2016 dataset

[0131]

[0132] Table 5 Experimental results of different distillation methods on the HRSC2016 dataset

[0133]

[0134] Table 6 Experimental results before and after using the KLDD distillation method on the HRSC2016 dataset

[0135]

[0136] from Figure 5 It can be seen intuitively that the proposed method is superior to other distillation methods. For the first image, although the proposed method also has false alarms, it is still better than other distillation methods overall; for the second image, the proposed method correctly detects all ship targets, while the DeFeat distillation method based on decoupled features and the LD distillation method based on generalized distribution information between coordinates have false alarms and missed detections; for the third image, the proposed method correctly detects all ship targets, while other distillation methods generally have false alarms and missed detections.

[0137] A lightweight system for rotating target detection based on knowledge distillation disclosed in an embodiment of the present invention is introduced below. The lightweight system for rotating target detection based on knowledge distillation described below and the lightweight method for rotating target detection based on knowledge distillation described above can refer to each other.

[0138] The present invention also provides a lightweight system for rotating target detection based on knowledge distillation, comprising the following steps:

[0139] A coordinate decoding module, which is used to decode the coordinate encoding of the detection target output by the student model and the trained teacher model into a rotation coordinate form, so as to obtain a rotation coordinate representation of the detection target;

[0140] A coordinate conversion module, which is used to convert the rotation coordinate representation into a two-dimensional Gaussian distribution to obtain a Gaussian distribution representation of the detection target;

[0141] A KL divergence calculation module is used to calculate the KL divergence of the Gaussian distribution representation of the detection target output by the student model and the teacher model to obtain a first KL divergence, and to calculate the KL divergence between the Gaussian distribution representation of the detection target output by the student model and the Gaussian distribution representation of the detection target obtained by the label true frame to obtain a second KL divergence;

[0142] A normalization processing module, used for performing normalization processing on the first KL divergence and the second KL divergence;

[0143] A loss function calculation module, which is used to calculate the overall loss function in the process of knowledge distillation of the teacher model to the student model according to the first KL divergence and the second KL divergence after normalization;

[0144] The prediction module is used to make predictions using the distilled student model.

[0145] In one embodiment of the present invention, in a coordinate conversion module, a method for converting the rotation coordinate representation into a two-dimensional Gaussian distribution includes:

[0146] Represent the rotated coordinates (x, y, w, h, θ) is converted into a two-dimensional Gaussian distribution The calculation formula of (μ, ∑) is:

[0147] μ=(x,y) T

[0148]

[0149]

[0150] Where (x, y, w, h, θ) is the rotation coordinate representation of the detection target, which respectively represents the horizontal coordinate, vertical coordinate, width, height and rotation angle of the detection target center point, and (μ, ∑) represents the mean and covariance matrix of the Gaussian distribution.

[0151] In one embodiment of the present invention, in the KL divergence calculation module, a method for calculating the KL divergence of the Gaussian distribution representation of the detection target output by the student model and the teacher model includes:

[0152] The calculation formula of the KL divergence represented by the Gaussian distribution of the detection target output by the student model and the teacher model is:

[0153]

[0154] In the formula (x S ,y S , w S ,h S ,θ S ) and (x T ,y T , w T ,h T ,θ T ) (μ T ,∑ T ) represent the rotation coordinate representation and Gaussian distribution representation of the detection target output by the student model and the teacher model respectively, Δx = x S -x T , Δy=y S -y T , Δθ=θ S -θ T They respectively represent the difference in horizontal and vertical coordinates and the difference in rotation angle between the center point of the detected target output by the student model and the label real box.

[0155] The lightweight system for rotating target detection based on knowledge distillation of this embodiment is used to implement part of the embodiment of the aforementioned lightweight method for rotating target detection based on knowledge distillation. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part and will not be introduced in detail here.

[0156] In addition, since the lightweight system for rotating target detection based on knowledge distillation of this embodiment is used to implement the aforementioned lightweight method for rotating target detection based on knowledge distillation, its function corresponds to that of the above method and will not be repeated here.

[0157] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0158] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0161] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A lightweight method for rotating target detection based on knowledge distillation, characterized by: The following steps are involved: S1: Decode the coordinate encoding of the detection target output by the student model and the trained teacher model into the form of rotation coordinates to obtain the rotation coordinate representation of the detection target; Decode the coordinate encoding Y:(dx,dy,dw,dh,dθ) of the detection target output by the student model and the trained teacher model into the form of rotated coordinates in: x=x a +dx*w a y=y a +to*h a w=w a *have been dw h=h a *e dh θ=θ a +dθ In the formula, A:(x a ,y a ,w a ,h a ,θ a ) is the anchor frame preset by the model; S2: converting the rotation coordinate representation into a two-dimensional Gaussian distribution to obtain a Gaussian distribution representation of the detection target; Represent the rotated coordinates Convert to a two-dimensional Gaussian distribution The calculation formula is: μ=(x,y) T Where (x, y, w, h, θ) is the rotation coordinate representation of the detection target, which respectively represents the horizontal coordinate, vertical coordinate, width, height and rotation angle of the detection target center point, and (μ, ∑) represents the mean and covariance matrix of the Gaussian distribution; S3: Calculate the KL divergence of the Gaussian distribution representation of the detection target output by the student model and the teacher model to obtain a first KL divergence, and calculate the KL divergence between the Gaussian distribution representation of the detection target output by the student model and the Gaussian distribution representation of the detection target obtained by the label true frame to obtain a second KL divergence; The method for calculating the KL divergence of the Gaussian distribution representation of the detection target output by the student model and the teacher model includes: The calculation formula of the KL divergence represented by the Gaussian distribution of the detection target output by the student model and the teacher model is: In the formula, and Represent the rotation coordinate representation and Gaussian distribution representation of the detection target output by the student model and the teacher model, respectively, Δx = x S -x T ,Δy=y S -y T ,Δθ=θ S -θ T They represent the difference in horizontal and vertical coordinates and the difference in rotation angle between the center point of the detected target and the label real frame output by the student model respectively; The method for calculating the KL divergence between the Gaussian distribution representation of the detection target output by the student model and the Gaussian distribution representation of the detection target obtained by the label true frame includes: The calculation formula for the KL divergence between the Gaussian distribution representation of the detection target output by the student model and the Gaussian distribution representation of the detection target obtained by the label real box is: In the formula and They represent the rotation coordinate representation and Gaussian distribution representation of the detection target output by the student model and the detection target obtained by the label real box, respectively, Δx = x S -x G ,Δy=y S -y G ,Δθ=θ S -θ G They respectively represent the difference in horizontal and vertical coordinates and the difference in rotation angle between the center point of the detection target output by the student model and the detection target obtained by the label real frame; S4: normalizing the first KL divergence and the second KL divergence; Normalize the KL divergence between Gaussian distributions F(KL): Get the final KL divergence loss calculation formula between the coordinate parameter Gaussian distribution: S5: According to the normalized first KL divergence and the second KL divergence, the overall loss function of the teacher model in the process of knowledge distillation of the student model is calculated; During training, the overall loss function of the model is for: Among them, λ0, λ1, λ2 are weight parameters, which are set to 1, 15, and 5.5 respectively; is the positive sample region mask, which is determined by the intersection-over-union ratio of the anchor box and the true target box; C S ,C T ,C G The category confidence and label true category unique encoding output by the student and teacher models respectively; They are the rotation coordinates and label true coordinates output by the student and teacher models respectively; is the cross entropy loss; is the KL divergence loss between category distributions; and is the KL divergence between the normalized student-truth and student-teacher Gaussian distributions; S6: Prediction using the distilled student model.

2. According to claim 1, a lightweight method for rotating target detection based on knowledge distillation is characterized in that: In step S6, after prediction is performed using the distilled student model, the prediction box output by the student model is post-processed.

3. According to claim 2, a lightweight method for rotating target detection based on knowledge distillation is characterized in that: In step S6, the method for post-processing the prediction frame output by the student model includes: the specific steps of post-processing are as follows: Arrange all prediction boxes in each category in ascending or descending order by score; Select the prediction box with the highest score as the benchmark, and calculate the intersection and union ratio of other prediction boxes with it; The prediction boxes in each category are screened according to the intersection-and-union ratio, and the prediction boxes that do not meet the screening conditions are deleted, where the screening condition is that the intersection-and-union ratio of the prediction box to be screened and the reference prediction box is less than or equal to a preset threshold; Among the remaining prediction boxes that meet the screening conditions, select the prediction box with the highest score as the benchmark box, and repeat the above screening steps until all prediction boxes that meet the screening conditions are used as benchmarks for screening.

4. A lightweight system for rotating target detection based on knowledge distillation, characterized in that: The following steps are involved: A coordinate decoding module is used to decode the coordinate encoding of the detection target output by the student model and the trained teacher model into a rotation coordinate form, and obtain the rotation coordinate representation of the detection target; In the coordinate decoding module, the coordinate encoding Y: (dx, dy, dw, dh, dθ) of the detection target output by the student model and the trained teacher model is decoded into the form of rotation coordinates x=x a +dx*w a y=y a +to*h a w=w a *have been dw h=h a *e dh θ=θ a +dθ In the formula, A:(x a ,y a ,w a ,h a ,θ a ) is the anchor frame preset by the model; A coordinate conversion module, which is used to convert the rotation coordinate representation into a two-dimensional Gaussian distribution to obtain a Gaussian distribution representation of the detection target; In the coordinate conversion module, the method of converting the rotation coordinate representation into a two-dimensional Gaussian distribution includes: Represent the rotated coordinates Convert to a two-dimensional Gaussian distribution The calculation formula is: μ=(x,y) T Where (x, y, w, h, θ) is the rotation coordinate representation of the detection target, which respectively represents the horizontal coordinate, vertical coordinate, width, height and rotation angle of the detection target center point, and (μ, ∑) represents the mean and covariance matrix of the Gaussian distribution; A KL divergence calculation module is used to calculate the KL divergence of the Gaussian distribution representation of the detection target output by the student model and the teacher model to obtain a first KL divergence, and to calculate the KL divergence between the Gaussian distribution representation of the detection target output by the student model and the Gaussian distribution representation of the detection target obtained by the label true frame to obtain a second KL divergence; In the KL divergence calculation module, a method for calculating the KL divergence of the Gaussian distribution representation of the detection target output by the student model and the teacher model includes: The calculation formula of the KL divergence represented by the Gaussian distribution of the detection target output by the student model and the teacher model is: In the formula, and Represent the rotation coordinate representation and Gaussian distribution representation of the detection target output by the student model and the teacher model, respectively, Δx = x S -x T ,Δy=y S -y T ,Δθ=θ S -θ T They represent the difference in horizontal and vertical coordinates and the difference in rotation angle between the center point of the detected target and the label real frame output by the student model respectively; In the KL divergence calculation module, a method for calculating the KL divergence between the Gaussian distribution representation of the detection target output by the student model and the Gaussian distribution representation of the detection target obtained by the label true frame includes: The calculation formula for the KL divergence between the Gaussian distribution representation of the detection target output by the student model and the Gaussian distribution representation of the detection target obtained by the label real box is: In the formula and They represent the rotation coordinate representation and Gaussian distribution representation of the detection target output by the student model and the detection target obtained by the label real box, respectively, Δx = x S -x G ,Δy=y S -y G ,Δθ=θ S -θ G They respectively represent the difference in horizontal and vertical coordinates and the difference in rotation angle between the center point of the detection target output by the student model and the detection target obtained by the label real frame; A normalization processing module, used for performing normalization processing on the first KL divergence and the second KL divergence; In the normalization processing module, the KL divergence between Gaussian distributions is normalized F(KL): Get the final KL divergence loss calculation formula between the coordinate parameter Gaussian distribution: A loss function calculation module, which is used to calculate the overall loss function in the process of knowledge distillation of the teacher model to the student model according to the first KL divergence and the second KL divergence after normalization; The overall loss function of the model for: Among them, λ0, λ1, λ2 are weight parameters, which are set to 1, 15, and 5.5 respectively; is the positive sample region mask, which is determined by the intersection-over-union ratio of the anchor box and the true target box; C S ,C T ,C G The category confidence and label true category unique encoding output by the student and teacher models respectively; The rotated coordinates and true label coordinates output by the student and teacher models respectively; is the cross entropy loss; is the KL divergence loss between category distributions; and is the normalized KL divergence between the student-truth and student-teacher Gaussian distributions; The prediction module is used to make predictions using the distilled student model.

Images