A reliable identification method for drone aerial photography targets based on data and model uncertainty perception

Abstract

A method for credible identification of drone aerial photography targets with dual uncertainty perception of data and model includes the following steps: collecting drone aerial photography data sets, preprocessing the data, annotating the processed data, and constructing a training set, a validation set, and a test set; constructing a credible target detection model integrating the second-order probability modeling of AvUC loss regularization and improved focal loss; training the constructed credible target detection model with the training set, and selecting the best model with the validation set to obtain the trained credible target detection model; inputting the test image of the test set into the trained credible target detection model to obtain the positioning information and classification information of each target in the image, as well as the data uncertainty of the positioning information, the model uncertainty, and the overall uncertainty of the classification information, and reflecting the reliability of the model through uncertainty, thereby realizing credible target detection of drone aerial photography. This method provides an important auxiliary decision basis for risk-sensitive drone aerial photography target detection applications.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a reliable identification method for unmanned aerial vehicle aerial photography targets with dual uncertainty perception of data and models. Background Art

[0002] As drone aerial photography data collection becomes more convenient, drone-based target detection technology has developed rapidly, and has broad application prospects in smart cities, environmental monitoring, precision agriculture, and military positioning and strike. The rapid development of deep learning in recent years has also provided a variety of more effective solutions, among which the most representative are the two-stage target detection model represented by R-CNN and the single-stage target detection model represented by YOLO and RetinaNet. These models have greatly reduced the burden of manpower and improved detection efficiency.

[0003] However, existing work usually focuses on how to improve the prediction accuracy and speed of the model, ignoring the measurement of model reliability, which affects the user's recognition and acceptance of such models. Because in risk-sensitive scenarios such as drone aerial photography target detection, not only high accuracy and speed are required, but also the reliability measurement of prediction results is required to effectively avoid the threat caused by wrong predictions. Specifically, whether it is a single-stage model or a two-stage model, the target detection problem will be decomposed into classification and regression problems. For classification problems, the softmax activation function is used to output the prediction results and confidence; for regression problems, the regression value is directly output after the corresponding transformation. This results in the model being unable to effectively measure the prediction confidence, especially when facing out-of-distribution (OOD) data and data with large noise, the above model will still output extremely high confidence for its predicted erroneous results, which will have a serious impact on subsequent judgments.

[0004] At the same time, the drone aerial photography dataset also has the problem of class imbalance, that is, the number of positive samples (prediction boxes of objects in the box) in the dataset is significantly different. This will cause the classes with a large number of positive samples to dominate the training, and the classes with a small number of positive samples cannot be effectively trained and become difficult samples, affecting the accuracy of the object detector.

[0005] Existing target detection models cannot provide a reliability measure that characterizes the degree of uncertainty in prediction results. Therefore, it is necessary to invent a reliable identification method for drone aerial photography targets that perceives both data and model uncertainty, and propose a solution to the category imbalance problem based on uncertainty perception. Summary of the invention

[0006] The technical problem to be solved by the present invention is to propose a reliable identification method for drone aerial photography targets with dual uncertainty perception of data and model in view of the deficiencies in the prior art.

[0007] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0008] A reliable identification method for drone aerial photography targets based on data and model dual uncertainty perception specifically includes the following steps:

[0009] Step 1: Collect drone aerial photography data sets, preprocess the image data, annotate the processed image data, and construct training sets, validation sets, and test sets;

[0010] Step 2: Build a credible target detection model that integrates the second-order probability modeling of AvUC loss regularization and improved focal loss;

[0011] Step 3: Use the training set to train the constructed credible target detection model, and use the validation set to select the best model to obtain the trained credible target detection model;

[0012] Step 4: Input the test images of the test set into the trained trusted target detection model to obtain the positioning information and classification information of each target in the image, as well as the data uncertainty of the positioning information, the model uncertainty and the overall uncertainty of the classification information. The reliability of the model is reflected by the uncertainty. The higher the uncertainty, the less reliable the output result of the model for the prediction box, thereby realizing the trusted target detection of UAV aerial photography.

[0013] The advantages of the technical solution of the present invention are:

[0014] (1) Through second-order probability modeling, the present invention provides a reliability metric that characterizes the uncertainty of prediction results, which is not available in existing similar UAV aerial target detection methods. At the same time, the introduction of the AvUC loss regularization term in the loss function improves the reliability of uncertainty estimation, providing an important auxiliary decision-making basis for risk-sensitive UAV aerial target detection applications.

[0015] (2) Based on second-order probability modeling, the present invention proposes an improved focal loss to solve the problem of class imbalance in drone aerial photography datasets, thereby improving the accuracy of target detectors;

[0016] (3) Based on second-order probability modeling, the present invention proposes a zero-evidence loss function for negative sample training, so that negative samples can be well trained;

[0017] (4) Adopt the training strategy of frozen training and pre-training to make the training process faster and more effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 1 is a general flow chart of the method in an embodiment of the present invention.

[0019] Figure 2 The flowchart of collecting and constructing a data set in an embodiment of the present invention is shown in FIG.

[0020] Figure 3 The present invention provides a flowchart for constructing a trusted target detection model integrating second-order probability modeling with AvUC loss regularization and improved focal loss in an embodiment of the present invention.

[0021] Figure 4 The figure is a flowchart of the model training in the embodiment of the present invention.

[0022] Figure 5 The figure is a flow chart of model prediction in an embodiment of the present invention.

[0023] Figure 6 This is a diagram of the target detection network structure that integrates second-order probability modeling in an embodiment of the present invention.

[0024] Figure 7 4 is a diagram of the regression evidence perception network structure in an embodiment of the present invention.

[0025] Figure 8 This is a structural diagram of a classification evidence perception network in an embodiment of the present invention.

[0026] Fig. 9 Schematic diagram of the model running results in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0028] The UAV aerial photography trusted target detection method provided in this embodiment can obtain classification prediction information, positioning information, data uncertainty of positioning information and model uncertainty and overall uncertainty of classification information for a non-fixed number of detection targets in an image, and can be applied to various UAV aerial photography target detection scenarios.

[0029] The present invention discloses a method for detecting credible targets in drone aerial photography based on dual uncertainty perception of data and model. The overall process is as follows: Figure 1 As shown, the specific implementation includes the following steps:

[0030] like Figure 2 As shown, step 1 is to collect drone aerial photography data sets, preprocess the image data, annotate the processed image data, and construct training sets, validation sets, and test sets.

[0031] Step 1-1, obtain a drone aerial photography target detection dataset with labels (including positioning and classification labels) from public datasets, download other drone aerial photography images (unlabeled) from the Internet, and use drones to shoot in actual scenes (unlabeled) to collect drone aerial photography data.

[0032] Step 1-2: For the labeled data, select the categories that are suitable for the actual use scenarios of the model from all categories, delete the positioning and classification labels corresponding to the categories that are irrelevant to the actual use scenarios; annotate the unlabeled data, and merge it with the labeled data after annotating. After merging, convert the image format and unify it into jpeg compression format to form a drone aerial photography target detection dataset.

[0033] In steps 1-3, the data set is randomly shuffled and split into training set, validation set and test set in a ratio of 8:1:1. The training set is used for model training, the validation set is used for hyperparameter selection and optimal model selection, and the test set is used to evaluate model performance.

[0034] like Figure 3 As shown in Figure 2, step 2 is to build a credible target detection model that integrates the second-order probability modeling of AvUC loss regularization and the improved focal loss.

[0035] Step 2-1, since the essence of target detection is to regress the location of the predicted box and classify the category of the target in the image, first use evidence deep learning (EDL) to perform second-order probability modeling on the generalized classification and regression problems, and explicitly model the dual uncertainty of data and model. EDL regards the output of traditional neural networks as random variables, that is, for classification problems, the results of the model output are regarded as category probabilities, which obey the category distribution:

[0036]

[0037] Among them, K is the number of categories, S K is the space of K-dimensional category distribution parameters p, p k is the category probability of the kth category; for regression problems, EDL regards the output of the model as a Gaussian random variable that obeys a Gaussian distribution:

[0038] y~N(W T x,σ 2 ) (2)

[0039] Among them, W is the parameter of the neural network, x is the input data, σ 2is noise, and y is a Gaussian random variable. EDL performs second-order probability modeling on the category distribution and Gaussian distribution, that is, it is assumed that the parameter p of the category distribution obeys the second-order distribution of the category distribution - Dirichlet distribution, and the parameters μ, σ of the Gaussian distribution 2 A second-order distribution that follows a Gaussian distribution—the normal inverse gamma distribution (NIG).

[0040] Specifically, in the classification problem, EDL collects evidence of subjective opinions during the training process, and then parameterizes the evidence into parameters of the Dirichlet distribution based on subjective logic theory and DS evidence theory to perceive the uncertainty of the output results, and finally converts the uncertainty into a reliable measure of the output results. The Dirichlet distribution can be used to model the second-order probability distribution of the categorical distribution parameter p, and its probability density function is as follows:

[0041]

[0042] Among them, α is the parameter of Dirichlet distribution, α k Corresponding to the kth category, B(·) is the beta function. According to the DS evidence theory and subjective logic theory, the uncertainty of the prediction results and the belief scores of each category are calculated as follows:

[0043]

[0044] in, is the Dirichlet intensity, e k =α k -1 is the subjective opinion evidence of the neural network output for category k, and the uncertainty u and the belief score b of each category k Satisfies the following equation:

[0045]

[0046] The category probability of the predicted result can be calculated using the mean of the Dirichlet distribution:

[0047]

[0048] in, is the probability of predicting category k, category probability Can be converted into a one-hot vector The classification prediction result.

[0049] In regression problems, EDL uses the normal inverse gamma distribution (NIG) to perform second-order probability modeling on Gaussian random variables (obeying first-order Gaussian distribution). The normal inverse gamma distribution is the joint distribution of the conjugate prior distribution of the mean of the Gaussian distribution (second-order Gaussian distribution) and the conjugate prior distribution of the variance of the Gaussian distribution (inverse gamma distribution). The model uncertainty is obtained by calculating the variance of the second-order Gaussian distribution, and the data uncertainty is obtained by calculating the expectation of the inverse gamma distribution. The uncertainty of the regression output result is perceived through these two uncertainties, and finally the uncertainty is converted into a reliable measure of the regression output result. The NIG distribution can be used to model the Gaussian distribution parameters μ, σ 2 The second-order probability distribution of has the following probability density function:

[0050]

[0051] Among them, γ, v, α, β are the parameters of NIG distribution, and Γ(·) is the gamma function. The model uncertainty is obtained by calculating the variance of the second-order Gaussian distribution:

[0052]

[0053] The data uncertainty is obtained by calculating the expectation of the inverse gamma distribution:

[0054]

[0055] The prediction results of regression can be calculated using the expectation of the second-order Gaussian distribution:

[0056] E[μ]=γ (10)

[0057] In addition, the strength of evidence for NIG distribution can be calculated as follows:

[0058] Φ=2ν+α (11)

[0059] Step 2-2, such as Figure 6 As shown in the figure, Retinanet is used as the baseline target detection network. According to the EDL second-order probability modeling, the network architecture is modified to complete the task of extracting image features and second-order probability modeling.

[0060] Specifically, Retinanet is an anchor-frame-based object detector, that is, each prediction frame output by the model corresponds to an anchor frame one-to-one. In the feature maps of all scales output by Retinanet, each pixel corresponds to 3 sizes and 3 aspect ratios, with a total of 9 anchor frames. Retinanet consists of three modules, namely the detection head module, the feature fusion module and the output module. The backbone network of the detection head module is ResNet50, which is used to receive 600*600 image input and obtain feature maps of 3 scales, namely C3: 75*75*512, C4: 38*38*1024 and C5: 19*19*2048, where the first two dimensions are the width and height of the feature map, and the third dimension is the number of channels. The receptive fields corresponding to feature maps of different scales are different. The smaller the width and height of the feature map, the larger the receptive field, and the larger the width and height, the smaller the receptive field. The larger the receptive field is used to detect large objects in the image, and the smaller the receptive field is used to detect small objects in the image. The feature fusion module is an FPN feature pyramid network, which is used to fuse features from three scales of the detection head. P5 is obtained from C5 after two convolutions. C5 is convolved once and upsampled, and then added to the result of C4 after one convolution, and then convolved again to obtain P4. P3 is also obtained in a similar way to P4. Secondly, P6 is directly convolved from C5, and P7 is convolved from P6. So far, the feature fusion module obtains feature outputs of five scales, P3, P4, P5, P6, and P7, and the channel dimension of each output is 256. Among them, P6 and P7 have larger receptive fields, and P3 and P4 fuse features of different scales. The output module consists of a classification subnetwork and a regression subnetwork. The classification subnetwork and the regression subnetwork process the features of each scale output by the feature fusion module. Each classification and regression subnetwork performs five consecutive convolution operations. The channel dimension of the final output of the conventional Retinanet classification subnetwork is 9×the number of anchor boxes per pixel, which represents the category probability vector of the predicted box corresponding to all anchor boxes of the pixel. The channel dimension of the final output of each regression subnetwork is 9×4, which represents the normalized offset value of the center coordinates x and y of the predicted box corresponding to all anchor boxes of the pixel relative to the center coordinates of the anchor box:

[0061]

[0062]

[0063] And the logarithmic space ratio of the predicted box width and height w, h to the true anchor box width and height:

[0064]

[0065]

[0066] Among them, x pred,y pred ,w pred ,h pred Respectively represent the center coordinates and width and height of the prediction box, x anchor ,y anchor ,w anchor ,h anchor Respectively represent the center coordinates and width and height of the anchor box corresponding to the predicted box, Represents the output of the regression subnetwork.

[0067] For the object detection network that introduces EDL, the output module of Retinanet needs to be modified. That is, the channel dimension of the classification subnetwork output is 9×the number of anchor boxes per pixel, but it is no longer a category probability vector but a parameter vector of Dirichlet distribution. That is, the output of the model is activated by ReLU and added by 1, such as Figure 8 As shown. The channel dimension of the regression subnetwork output becomes the number of anchor boxes per pixel 9×16. It represents the second-order probability modeling of the four regression values. The second-order distribution of each regression value is represented by a 4-dimensional vector, that is, the center coordinate x relative to the center coordinate x of the anchor box anchor The normalized offset value of the NIG distribution parameter γ x ,v x ,α x ,β x ; The center coordinate y is relative to the center coordinate y of the anchor box anchor The normalized offset value of the NIG distribution parameter γ y ,v y ,α y ,β y ; NIG distribution parameter γ of the ratio of the predicted box width w to the anchor box width w ,v w ,α w ,β w and the NIG distribution parameter γ of the ratio of the predicted box height h to the anchor box height h ,v h ,α h ,β h Since v≥0,α≥1,β≥0, the four pairs of v,α,β output by the regression sub-network need to be activated by the SoftPlus function, and the four α parameters need to be increased by 1, such as Figure 7 shown.

[0068] In step 2-3, for the second-order distribution parameters of the output classification and regression, the second-order probability modeling fused with AvUC loss regularization and the loss function of improved focal loss are used to optimize the positive samples, and the 0-evidence loss function is used to optimize the negative samples.

[0069] In the object detection task, the prediction box is divided into three categories: positive samples, negative samples, and ignored. Among them, the positive sample refers to the sample whose maximum intersection-over-union ratio (IOU) between the anchor box corresponding to the prediction box and all the real boxes (labels) is greater than 0.5, that is, the prediction box of the object in the box, the negative sample refers to the sample whose maximum intersection-over-union ratio (IOU) between the anchor box corresponding to the prediction box and all the real boxes is less than 0.4, that is, the prediction box of the background in the box, and the ignored one refers to the prediction box whose maximum intersection-over-union ratio (IOU) between the anchor box corresponding to the prediction box and all the real boxes is between 0.4 and 0.5.

[0070] Based on the uncertainty perception of the second-order probability modeling of the model output results, the classification and regression of negative samples should generate 0 evidence. For classification, the Dirichlet distribution parameters of 0 evidence are all 1 vectors, that is, D(p i |<1,…,1>), the loss function is the KL divergence of the Dirichlet distribution of the negative sample output and the 0-evidence Dirichlet distribution:

[0071]

[0072] Among them, α i is the output Dirichlet distribution parameter, 1 is a full 1 vector, D(p i |1) is 0-evidence Dirichlet distribution, i.e. uniform distribution, Specifically:

[0073]

[0074] Among them, α ik are the parameters of the Dirichlet distribution for class k, Γ(·) is the gamma function, and ψ(·) is the digamma function.

[0075] For regression, the evidence strength of the NIG distribution of 0 evidence Φ = 2v + α should be 0, and the loss function is the L1 norm of the evidence strength of the NIG distribution:

[0076]

[0077] For positive samples, the number of different categories in the target also varies significantly. Categories with a larger number will dominate the loss function of positive samples and become easy samples, while categories with a smaller number will become difficult samples and cannot be effectively trained. Therefore, multi-class focal loss is introduced into the EDL classification loss function to adaptively weight categories of different difficulty in the loss function. Specifically, we first define the MSE Bayesian risk loss to learn the evidence of subjective opinions, which is used to model the second-order Dirichlet distribution:

[0078]

[0079] in, is the MSE operator, i represents the sample, c i Indicates a label. Specifically:

[0080]

[0081] The first term of this loss function is used to ensure the fit of the classification, and the second term is used to reduce the variance of the output. Then, the MSE Bayesian risk loss with the introduction of multi-class focal loss becomes

[0082]

[0083]

[0084] Among them, δ is the hyperparameter of focal loss. In order to make the wrong class label produce smaller evidence (even close to 0), the KL divergence regularization term needs to be added to the loss function:

[0085]

[0086] in is the Dirichlet distribution parameter corresponding to the evidence without the correct class label. This Dirichlet distribution should be equal to the Dirichlet distribution D(p i |<1,…,1>), that is, a uniform distribution that does not provide any evidence, and 1 is a vector of all 1s. Specifically:

[0087]

[0088] In order to make the model output smaller uncertainty for correctly classified samples and larger uncertainty for misclassified samples to make the uncertainty estimation more reliable, the AvUC loss is introduced, and its expression is as follows:

[0089]

[0090] in, is the annealing coefficient, epoch is the current number of training rounds, T is the total number of training rounds, and p i =max{α ik / S i}. In summary, the total loss function for positive sample classification is:

[0091]

[0092] Among them, λ eis the annealing coefficient, and epoch is the current number of training rounds, which is used to prevent the model from converging to a uniform distribution in the early stage. By optimizing the loss function shown in Formula 25, the model can output the uncertainty degree used to reflect the reliability of the model while outputting the category probability that represents the model's prediction results. Under the effect of AvUC loss, the uncertainty calibration of the model's prediction will be greatly improved, and the focal loss will greatly improve the model's accuracy.

[0093] For the regression task in the positive sample, the Type-2 maximum likelihood function of the Gaussian random variable is obtained according to the Type-2 maximum likelihood estimation. The negative logarithm of the likelihood function is taken and optimized to model the second-order normal inverse gamma distribution:

[0094]

[0095] Among them, i represents the sample, y i Indicates a label. Specifically:

[0096]

[0097] Where Ω = 2β i (1+v i ). Similar to classification, in order to make wrong predictions produce less evidence, a regularization term needs to be added to the loss function:

[0098]

[0099] In summary, the total loss function of positive sample regression is:

[0100]

[0101] Among them, λ is the regularization term weight.

[0102] In summary, the loss function of the credible target detection model that integrates the second-order probability modeling of AvUC loss regularization and the improved focal loss is:

[0103]

[0104] in, and is an indicator function, indicating whether the current prediction box is a positive sample or a negative sample. By optimizing this loss function, the optimization of the trusted target detection model that integrates the regularization of the AvUC loss and the improved focal loss can be completed.

[0105] like Figure 4As shown, in step 3, the constructed credible target detection model is trained with the training set, and the best model is selected with the validation set to obtain the trained credible target detection model;

[0106] Step 3-1, since the collected data comes from various sources and the image resolutions vary greatly, and the model requires the input image size to be 600*600 pixels, the image needs to be scaled. For the validation set and test set data, each image is scaled to the center of the 600*600 image, and the surrounding vacant parts are filled with gray in the RGB color space of [128,128,128]. At the same time, the coordinates of the real box (label) corresponding to each image are transformed in the same way. For the data of the training set, in order to make the model more robust and able to cope with various special situations, each image in the training set is subjected to data enhancement processing, that is, random color gamut transformation, random length and width scaling, random cropping, and random flipping are performed at the same time, and the size is limited to 600*600. At the same time, the coordinates of the real box (label) are transformed in the same way.

[0107] In step 3-2, the model weights pre-trained on the ImageNet dataset with Retinanet as the detection head are used as the initial weights of the model. The model is trained using the second-order probability modeling and the improved focal loss loss function proposed in step 2 that integrates the AvUC loss regularization. The frozen training strategy is adopted, that is, the training is frozen first, and the gradients of the weights of the model detection head are not calculated. Only the weights of the feature fusion module and the output module are updated. This prevents the feature extraction detection head with general value from being changed in the early stage of training. After the frozen training is completed, the unfrozen training is performed again, and all the parameters of the model participate in the calculation of the gradient, so that the performance of the model converges to the optimal state. It should be noted that the batch size of frozen training can be set slightly larger because there are fewer parameters involved in the training. The batch size of unfrozen training is set to half of that of frozen training because there are more parameters involved in the training. The number of iterations of frozen training is usually set to 50, and the number of iterations of unfrozen training varies with the task. During training, the Adam optimization algorithm is used to update the parameters of the model. The momentum parameter of the Adam algorithm is set to 0.9, the weight decay parameter is set to 0, and the cosine learning rate adjustment strategy is used to dynamically adjust the learning rate.

[0108] Step 3-3, calculate the loss of the current model on the validation set every 5 training rounds. The smaller the loss, the stronger the generalization ability on the validation set. When the validation set loss is less than the minimum of all previous validation set losses, save the parameters of the current model as the current best model until all training rounds are completed. Determine whether the loss on the validation set converges at the end of training. If so, end the training; if not, increase the number of iterations and retrain.

[0109] like Figure 5 As shown, in step 4, the test images of the test set are input into the trained trusted target detection model to obtain the positioning information and classification information of each target in the image, as well as the data uncertainty of the positioning information, the model uncertainty and the overall uncertainty of the classification information. The reliability of the model is reflected by the uncertainty, thereby realizing the trusted target detection of UAV aerial photography.

[0110] Step 4-1, input the pictures of the test set into the trained best model, and for each picture, obtain the feature maps of 5 output scales. The prediction box of each pixel relative to the 9 anchor boxes. The prediction box contains positioning information and classification information. The prediction box positioning information contains the NIG distribution parameters of 4 regression values, and the prediction box classification information is the Dirichlet distribution parameter. The NIG distribution parameters of the 4 regression values ​​are all used to obtain the data uncertainty of the regression through formula (9), the model uncertainty of the regression is obtained through formula (8), the predicted value of the regression is obtained through formula (10), and according to formula (12) to formula (15), the regression value is inversely transformed into the center coordinates and width and height of the prediction box at the original image scale. The Dirichlet distribution parameters are used to obtain the category probability through formula (6), and the overall uncertainty of the classification is obtained through formula (4).

[0111] In step 4-2, non-maximum suppression (NMS) is performed on each prediction box to delete duplicate prediction boxes of the same object in the box. First, the maximum value of the category probability and its category index in all prediction boxes are extracted. Then the prediction boxes with category probabilities lower than the preset confidence threshold are deleted. Finally, the NMS algorithm is used to traverse all categories. For example, for category k, all prediction boxes corresponding to category k are sorted from large to small according to category probability, starting from the largest one, and the IOU of the prediction box and all other prediction boxes corresponding to category k is calculated. When the IOU is greater than the preset IOU threshold, the category probability of the prediction box with a lower category probability is set to 0, and the prediction box is marked for deletion. After reordering, starting from the next prediction box, the above steps are repeated for all prediction boxes sorted after it until the last prediction box with a non-zero category probability is calculated. The prediction box that is not marked for deletion after NMS processing is the final result.

[0112] Step 4-3: Export the positioning information, classification information, data uncertainty of the positioning information, model uncertainty, and overall uncertainty of the classification information of all prediction boxes after NMS screening. Take one of the prediction boxes as an example, recorded as in They represent the positioning information of the prediction box, namely the coordinates and width and height of the center point. represents the predicted class probability, represents the predicted category index, The data uncertainty and model uncertainty of the coordinates and width and height of the center point are represented respectively. The subscript a represents data uncertainty, and the subscript e represents model uncertainty. cls Represents the overall uncertainty of the classification. The average of the four data uncertainty values ​​of the prediction box positioning gives the overall data uncertainty u of the positioning. a , averaging the four model uncertainty values ​​of the prediction box positioning gives the overall model uncertainty u of the positioning e .Will The predicted box positioning information represented is drawn at the corresponding position of the input image, and the thickness of the border is used to represent the overall model uncertainty u of the positioning e The thicker the circle, the less certain it is. The size of the circle at the four corners of the positioning box indicates the overall data uncertainty of the positioning. The larger the circle, the less certain it is, and the more important it is for the user to review. Output in the form of a text box above the prediction box Corresponding category name, category probability and classification uncertainty u cls , use font size to indicate the uncertainty of classification. The larger the font, the more uncertain it is, and the user needs to review it carefully. The higher the uncertainty of classification and regression, the less reliable the model's output for this prediction box is. Users should review the results of such prediction boxes carefully to avoid possible errors.

[0113] The results of running the model are as follows Fig. 9 shown.

[0114] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A reliable identification method for drone aerial photography targets based on data and model uncertainty perception, characterized by: The method specifically comprises the following steps: Step 1: Collect drone aerial photography data sets, preprocess the image data, annotate the processed image data, and construct training sets, validation sets, and test sets; Step 2: Build a credible target detection model that integrates the second-order probability modeling of AvUC loss regularization and improved focal loss; The step 2 specifically includes the following sub-steps: Step 2-1, use evidence deep learning (EDL) to perform second-order probability modeling on the output of model classification and regression, thereby explicitly modeling the dual uncertainty of data and model; For classification problems, the model output is regarded as the category probability, which obeys the category distribution and is formally expressed by the following formula: Among them, K is the number of categories, S K is the space of K-dimensional category distribution parameters K, p k is the category probability of the kth category; For regression problems, the output of the model is considered to be a Gaussian random variable that obeys a Gaussian distribution and is formally expressed as follows: y~N(W T x,σ 2 ) (2) Among them, W is the parameter of the neural network, x is the input data, σ 2 for noise; EDL performs second-order probability modeling on the category distribution and Gaussian distribution, that is, it is assumed that the parameter p of the category distribution obeys the second-order distribution of the category distribution, namely the Dirichlet distribution, and the parameters μ, σ of the Gaussian distribution 2 The second-order distribution that obeys the Gaussian distribution is the normal inverse gamma distribution NIG; Step 2-2, transform the classification and regression output modules of the Retinanet object detection network that introduces EDL; using Retinanet as the baseline object detection network, according to the EDL second-order probability modeling, transform the network architecture to complete the task of extracting image features and second-order probability modeling; Step 2-3: For the second-order distribution parameters of the output classification and regression, the 0-evidence loss function is used to optimize the negative samples, and the second-order probability modeling fused with AvUC loss regularization and the loss function of the improved focal loss are used to optimize the positive samples. Step 2-3: For the second-order distribution parameters of the output classification and regression, the second-order probability modeling fused with AvUC loss regularization and the loss function of improved focal loss are used to optimize the positive samples, and the zero-evidence loss function is used to optimize the negative samples. In the target detection task, the prediction box is divided into three categories: positive samples, negative samples, and ignored samples. Among them, the positive sample refers to the sample whose maximum intersection-over-union ratio (IOU) between the anchor box corresponding to the prediction box and all the real boxes is greater than 0.5, that is, the prediction box of the object in the box; the negative sample refers to the sample whose maximum intersection-over-union ratio (IOU) between the anchor box corresponding to the prediction box and all the real boxes is less than 0.4, that is, the prediction box of the background in the box; the ignored sample refers to the prediction box whose maximum intersection-over-union ratio (IOU) between the anchor box corresponding to the prediction box and all the real boxes is between 0.4 and 0.5; Based on the uncertainty perception of the second-order probability modeling of the model output results, the classification and regression of negative samples should generate 0 evidence; for classification, the Dirichlet distribution parameters of 0 evidence are all 1 vectors, that is, D(p i |<1,…,1>), the loss function is the KL divergence of the Dirichlet distribution of the negative sample output and the 0-evidence Dirichlet distribution: Among them, α i is the output Dirichlet distribution parameter, 1 is a full 1 vector, D(p i |1) is 0-evidence Dirichlet distribution, i.e. uniform distribution, Specifically: Among them, α ik are the parameters of the Dirichlet distribution for class k, Γ(·) is the gamma function, ψ(·) is the digamma function; For regression, the evidence strength of the NIG distribution of 0 evidence Φ = 2ν + α should be 0, and the loss function is the L1 norm of the evidence strength of the NIG distribution: For positive samples, since the number of different categories in the target is also significantly different; the category with a larger number will dominate the loss function of the positive sample and become an easy sample, while the category with a smaller number will become a difficult sample and cannot be effectively trained; therefore, multi-class focal loss is introduced into the EDL classification loss function to adaptively weight the weights of categories of different difficulty in the loss function; specifically, the MSE Bayesian risk loss is first defined to learn the evidence of subjective opinions for modeling the second-order Dirichlet distribution: in, is the MSE operator, i represents the sample, c i Indicates a label; Specifically: The first term of this loss function is used to ensure the fit of the classification, and the second term is used to reduce the variance of the output; then, the MSE Bayesian risk loss with multi-class focal loss is introduced to become Among them, δ is the hyperparameter of focal loss; in order to make the wrong class label produce smaller evidence, the KL divergence regularization term needs to be added to the loss function: in is the Dirichlet distribution parameter corresponding to the evidence without the correct class label. This Dirichlet distribution should be equal to the Dirichlet distribution D(p i |<1,…,1>), that is, a uniform distribution that does not provide any evidence, and 1 is a vector of all 1s; Specifically: In order to make the model output smaller uncertainty for correctly classified samples and larger uncertainty for misclassified samples to make the uncertainty estimation more reliable, the AvUC loss is introduced, and its expression is as follows: in, is the annealing coefficient, epoch is the current number of training rounds, T is the total number of training rounds, and p i =max{α ik / S i }; In summary, the total loss function for positive sample classification is: Among them, λ e is the annealing coefficient, epoch is the current training round number, which is used to prevent the model from converging to a uniform distribution in the early stage; For the regression task in the positive sample, the Type-2 maximum likelihood function of the Gaussian random variable is obtained according to the Type-2 maximum likelihood estimation. The negative logarithm of the likelihood function is taken and optimized to model the second-order normal inverse gamma distribution: Among them, i represents the sample, y i Indicates a label; Specifically: Where Ω = 2β i (1+ν i ); Similar to classification, in order to make wrong predictions produce less evidence, a regular term needs to be added to the loss function: In summary, the total loss function of positive sample regression is: Among them, λ is the regularization term weight; In summary, the loss function of the credible target detection model that integrates the second-order probability modeling of AvUC loss regularization and the improved focal loss is: in, and is an indicator function, indicating whether the current prediction box is a positive sample or a negative sample. By optimizing this loss function, the optimization of the second-order probability modeling integrating AvUC loss regularization and the trusted target detection model with improved focal loss can be completed. Step 3: Use the training set to train the constructed credible target detection model, and use the validation set to select the best model to obtain the trained credible target detection model; Step 4: Input the test images of the test set into the trained trusted target detection model to obtain the positioning information and classification information of each target in the image, as well as the data uncertainty of the positioning information, the model uncertainty and the overall uncertainty of the classification information. The reliability of the model is reflected by the uncertainty. The higher the uncertainty, the less reliable the output result of the model for the prediction box, thereby realizing the trusted target detection of UAV aerial photography.

2. According to claim 1, a reliable identification method for drone aerial photography targets with dual uncertainty perception of data and model, characterized in that: The step 1 specifically includes the following sub-steps: Step 1-1, obtain a labeled drone aerial photography target detection dataset from a public dataset, download other drone aerial photography images from the Internet, and use a drone to shoot in actual usage scenarios; Step 1-2: For the labeled data, select the categories that are applicable to the actual use scenarios of the model and remove the labels that are irrelevant to the actual use scenarios; label the unlabeled data and merge it with the labeled data after labeling; after merging, convert the image format and unify it into JPEG compression format; In steps 1-3, the data set is randomly shuffled and divided into training set, validation set and test set in a ratio of 8:1:

1. The training set is used for model training, the validation set is used for hyperparameter selection and selection of the best model, and the test set is used to test model performance.

3. The method for credible identification of drone aerial photography targets with dual uncertainty perception of data and model according to claim 2 is characterized by: In step 2-1, in the classification problem, EDL collects evidence of subjective opinions during the training process, and then parameterizes the evidence into parameters of the Dirichlet distribution based on subjective logic theory and DS evidence theory to perceive the uncertainty of the output results and convert the uncertainty into a reliable measure of the output results. The specific method is: The Dirichlet distribution is used to model the second-order probability distribution of the categorical parameter p, and its probability density function is as follows: Among them, α is the parameter of Dirichlet distribution, α k Corresponding to the kth category, is the space of K-dimensional category distribution parameters p, p k is the category probability of the kth category, B(·) is the beta function; according to the DS evidence theory and subjective logic theory, the uncertainty of the prediction results and the belief scores of each category are calculated as follows: in, is the Dirichlet intensity, e k =α k -1 is the subjective opinion evidence of the neural network output for category k; and the uncertainty u and the belief score b of each category k Satisfies the following equation: The category probability of the predicted result is calculated using the mean of the Dirichlet distribution: in, is the probability of predicting category k, category probability Can be converted into a one-hot vector The classification prediction results.

4. The method for credible identification of drone aerial photography targets with dual uncertainty perception of data and model according to claim 3 is characterized by: In step 2-1, in the regression problem, EDL uses the normal inverse gamma distribution NIG to perform second-order probability modeling on the Gaussian random variable. The normal inverse gamma distribution is the conjugate prior distribution of the mean of the Gaussian distribution, i.e., the second-order Gaussian distribution, and the conjugate prior distribution of the variance of the Gaussian distribution, i.e., the inverse gamma distribution. The model uncertainty is obtained by calculating the variance of the second-order Gaussian distribution, and the data uncertainty is obtained by calculating the expectation of the inverse gamma distribution. The uncertainty of the regression output result is perceived through these two uncertainties, and finally the uncertainty is converted into a reliable measure of the regression output result. The NIG distribution is used to model the Gaussian distribution parameters μ, σ 2 The second-order probability distribution of has the following probability density function: Among them, γ, ν, α, β are the parameters of NIG distribution, Γ(·) is the gamma function; the model uncertainty is obtained by calculating the variance of the second-order Gaussian distribution: The data uncertainty is obtained by calculating the expectation of the inverse gamma distribution: The prediction results of regression are calculated using the expectation of the second-order Gaussian distribution: E[μ]=γ (10) In addition, the strength of evidence for NIG distribution was calculated by the following formula: Φ=2ν+α (11).

5. The method for credible identification of drone aerial photography targets with dual uncertainty perception of data and model according to claim 4 is characterized by: The specific method of step 2-2 is: The prediction box positioning information output by the Retinanet regression subnetwork is the normalized offset value of the center coordinates x and y of each prediction box relative to the center coordinates of the anchor box, and the ratio of the width and height w and h of the prediction box relative to the width and height of the anchor box: Among them, x pred ,y pred 、w pred 、h pred Respectively represent the center coordinates and width and height of the prediction box, x anchor ,y anchor 、w anchor 、h anchor Represent the center coordinates and width and height of the anchor box respectively. Represents the output of the regression subnetwork; For the object detection network that introduces EDL, the output module of Retinanet needs to be modified; that is, the final number of channels for the classification subnetwork is the number of anchor boxes A × the number of categories, but since each value represents the Dirichlet distribution parameter of each predicted box of that class, the output of the model needs to be activated by the ReLU function and +1 to obtain; The final number of channels for the regression subnetwork is the number of anchor boxes A×16; among the 16 parameters, every 4 represent the second-order probability modeling of 1 positioning information, denoted as γ, v, α, β; there are 4 positioning information in total, namely the normalized offset value of the center coordinates x, y of the predicted box relative to the center coordinates of the anchor box, and the logarithmic space ratio of the width and height w, h of the predicted box relative to the width and height of the anchor box; since v≥0, α≥1, β≥0, it is necessary to pass the SoftPlus activation function on the 4 pairs of v, α, β output by the regression subnetwork, and add 1 to the 4 α parameters.

6. The method for credible identification of drone aerial photography targets with dual uncertainty perception of data and model according to claim 5 is characterized by: The method of step 3 is specifically as follows: Step 3-1, for the validation set and test set data, calculate the ratio of the actual size of the image to the size W*H of the input model, scale the input image to the center of the W*H image, fill the surrounding vacant parts with gray using the RGB color space of [128,128,128], and perform the same transformation on the coordinates of the real box in the label; The training set images are randomly transformed in color space, randomly scaled, randomly cropped, and randomly flipped, and the images are limited to the size of W*H. At the same time, the coordinates of the real box in the label are transformed in the same way. Step 3-2, use the model weights of Retinanet with ResNet detection head pre-trained on the ImageNet dataset as the initial weights of the model, and train the model using the loss function of the trusted target detection model that integrates the second-order probability modeling of AvUC loss regularization and the improved focal loss proposed in step 2; The frozen training strategy is adopted, that is, the training is frozen first, and the gradient of the weight of the model detection head is not calculated. Only the weights of the feature fusion module and the output module are updated; After the freeze training is completed, the unfreeze training begins. All parameters of the model are involved in the calculation of gradients. All parameters are fine-tuned to make the performance of the model converge to the optimal state. Step 3-3, calculate the loss of the current model on the validation set every 5 training rounds. The smaller the loss, the stronger the generalization ability on the validation set. When the validation set loss is less than the minimum of all previous validation set losses, save the parameters of the current model as the current best model until all training rounds are completed; determine whether the loss on the validation set converges at the end of training, if so, end the training; if not, increase the number of iterations and retrain.

7. The method for credible identification of drone aerial photography targets with dual uncertainty perception of data and model according to claim 6 is characterized by: The method of step 4 is specifically as follows: Step 4-1, input the pictures of the test set into the trained best model. For each picture, obtain the prediction box of each pixel relative to A anchor boxes under 5 output scales. The prediction box contains positioning information and classification information. The prediction box positioning information contains the NIG distribution parameters of 4 regression values, and the prediction box classification information is the Dirichlet distribution parameters. The NIG distribution parameters of the 4 regression values ​​are all used to obtain the regression data uncertainty through formula (9), the regression model uncertainty is obtained through formula (8), and the regression prediction value is obtained through formula (10). According to formulas (12) to (15), the center coordinates and width and height of the anchor box are combined for inverse transformation to obtain the center coordinates and width and height of the prediction box. The Dirichlet distribution parameters are used to obtain the category probability through formula (6), and the overall classification uncertainty is obtained through formula (4). The class with the largest category probability among all prediction boxes is extracted as the classification result of the prediction box, and the category probability and category index of this class are obtained. According to the category probability, the prediction box with a category probability lower than the preset confidence threshold is deleted. Step 4-2, perform non-maximum suppression (NMS) processing on each prediction box; use the NMS algorithm to delete repeated prediction boxes for the same object in the image; In step 4-3, the positioning information and classification information of the prediction box after NMS screening, as well as the data uncertainty of the positioning information, the model uncertainty and the overall uncertainty of the classification information are processed to realize the reliable target detection of UAV aerial photography.

8. The method for credible identification of drone aerial photography targets with dual uncertainty perception of data and model according to claim 7 is characterized by: The specific method of step 4-3 is: The positioning information, classification information, data uncertainty of the positioning information, model uncertainty and overall uncertainty of the classification information of all prediction boxes after NMS screening are derived. Taking one of the prediction boxes as an example, it is recorded as u cls ,in They represent the positioning information of the prediction box, namely the coordinates and width and height of the center point. represents the predicted class probability, represents the predicted category index, represents the data uncertainty and model uncertainty of the coordinates and width and height of the center point, respectively. The subscript a represents data uncertainty, and the subscript e represents model uncertainty. cls represents the overall uncertainty of the classification; The four data uncertainty values ​​of the prediction box positioning are averaged to obtain the overall data uncertainty u of the positioning a ; The four model uncertainty values ​​of the prediction box positioning are averaged to obtain the overall model uncertainty u of the positioning e ; Will The predicted box positioning information represented is drawn at the corresponding position of the input image, and the thickness of the border is used to represent the overall model uncertainty u of the positioning e The thicker the circle, the less certain it is. The size of the circles at the four corners of the positioning box represents the overall data uncertainty of the positioning, and the larger the circle, the less certain it is. Will Corresponding category name, category probability and classification uncertainty u cls , output in the form of a text box above the prediction box, and use the font size to indicate the uncertainty of the classification. The larger the font, the more uncertain it is.

Images