SAR ship open set detection method and device based on uncertainty learning

Abstract

The invention discloses an SAR ship open set detection method and device based on uncertainty learning. The method comprises the following steps: extracting data representation of a ship in an SAR image based on a Faster R-CNN backbone network, a region proposal network and a region-of-interest alignment module; respectively extracting distribution parameters represented by the data through two parallel encoders, and performing resampling based on the distribution parameters to obtain sampling features of the ship in the SAR image; classifying the sampling features through a classifier to obtain the category of the ship in the SAR image; according to the method, the data representation extracted by the Faster R-CNN is resampled, so that the data representation can be transferred from a point space to a probability space, the uncertainty in the classification process can be reduced, and the influence of cognitive uncertainty on the detection precision in the detection process is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of synthetic aperture radar ship detection, and particularly relates to a SAR ship open-set detection method and device based on uncertainty learning. Background Art

[0002] Synthetic Aperture Radar (SAR) ship detection algorithms are of great significance for tasks such as maritime monitoring, ship target positioning and identification, and can provide accurate target position and identity information for the implementation of tasks such as monitoring, striking, and rescue. In practical applications, synthetic aperture radar has the advantages of being unaffected by conditions such as weather and light, and can continuously monitor ship targets in the marine environment. The most important task of ship detection algorithms is to provide users with stable and accurate target position and identity information, and the key technology among them is to distinguish foreground and background. In actual ship detection, due to the coherent imaging mechanism of SAR, the signals reflected by multiple scatterers in the resolution unit produce random intensity changes when the phases are superimposed, and there is a large amount of inherent speckle noise in SAR images. In practical applications, these speckle noises will cause false alarms in ship detection algorithms.

[0003] In terms of ship detection, traditional ship detection algorithms are based on the closed-set assumption - that is, the training set contains all target categories that may appear in the test set. Traditional ship detection methods detect the actual scene by learning the target categories and position information in the training set. However, in the real world, categories that have not appeared in the training set will continuously appear, which brings great uncertainty and confusion to the prediction of ship detection algorithms. Summary of the Invention

[0004] The purpose of the present invention is to provide a SAR ship open-set detection method and device based on uncertainty learning to reduce the impact of cognitive uncertainty on detection accuracy during the detection process.

[0005] The present invention adopts the following technical solutions: A SAR ship open-set detection method based on uncertainty learning, comprising the following steps:

[0006] Extract the data representation of ships in the SAR image based on the backbone network, region proposal network and region of interest alignment module of Faster R-CNN;

[0007] Extract the distribution parameters of the data representation through two parallel encoders respectively, and perform resampling based on the distribution parameters to obtain the sampled features of ships in the SAR image;

[0008] Classify the sampled features through a classifier to obtain the categories of ships in the SAR image; the category set of the classifier includes known categories, unknown categories and background categories;

[0009] Among them, the encoder, classifier, backbone network, region proposal network, and region of interest alignment module of Faster R-CNN are jointly trained to optimize the parameters.

[0010] Furthermore, resampling based on distribution parameters includes:

[0011] Sampling random numbers from a normal distribution;

[0012] Generating sampling features based on the random numbers and distribution parameters.

[0013] Furthermore, generating sampling features based on the random numbers and distribution parameters includes:

[0014] x′ = μ + a·σ,

[0015] where x′ represents the sampling feature, μ and σ respectively represent the mean and variance of the data representation distribution, a represents the random number sampled from the normal distribution, and a ∈ N(0, 1).

[0016] Furthermore, the encoder is composed of a single-layer convolution and an L2 norm layer.

[0017] Furthermore, the classification loss of the classifier, cross-entropy loss, uncertainty loss of the encoder, box regression loss, and region proposal network loss in Faster R-CNN are included in the joint training.

[0018] Furthermore, the uncertainty loss is:

[0019] L IE = -α·p(x′)log p(x′),

[0020] where L IE represents the uncertainty loss, α represents the hyperparameter, and p(x′) represents the probability of the sampling feature.

[0021] Furthermore, the classification loss is:

[0022] L UE = -log(p(U*)),

[0023] where L UE represents the classification loss, and p(U*) represents the probability of mislabeling the sampling feature with the true class as a known class and labeling it as an unknown class.

[0024] Furthermore, the calculation method of p(U*) is:

[0025]

[0026] where S uClassification data indicating that the sampled feature is marked as an unknown class, S j Classification data indicating that the sampled feature is marked as a non-known class, C j Indicates the j-th class in the class set, and C′ represents the known class.

[0027] Furthermore, the cross-entropy loss is:

[0028] L cls = -log(p(C′)),

[0029] where L cls represents the cross-entropy loss, p(C′) represents the probability of correctly marking the sampled feature with the true class being the known class, and C′ represents the known class.

[0030] Another technical solution of the present invention: A SAR ship open-set detection device based on uncertainty learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above method is implemented.

[0031] The beneficial effects of the present invention are as follows: By resampling the data representation extracted by Faster R-CNN without losing gradients, the present invention can transfer the data representation from the point space to the probability space, reduce the uncertainty in the classification process, thereby reducing the impact of epistemic uncertainty on the detection accuracy in the detection process, and can identify unknown-class ships through epistemic uncertainty. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of a SAR ship open-set detection method based on uncertainty learning in an embodiment of the present invention;

[0033] Figure 2 It is a schematic diagram of the entropy-aware module in an embodiment of the present invention;

[0034] Figure 3 It is a schematic diagram of the unknown-aware module in an embodiment of the present invention;

[0035] Figure 4 It is a schematic diagram of the detection result in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0036] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Nowadays, deep learning algorithms, with their powerful representation capabilities, can map high-dimensional data to a series of low-dimensional outputs. However, these mappings are often blind and unreliable. Therefore, the estimation of uncertainty and reasonable decision-making are crucial in practical applications.

[0038] In the deep learning method based on uncertainty learning, attention is paid to the data uncertainty caused by sensor noise and the cognitive uncertainty caused by lack of knowledge. Therefore, the uncertainty learning method can be regarded as a special form of uncertainty estimation method with a supervised form, where the supervision is induced by preset prior knowledge. The ship detection method is supervised by setting an ideal prediction probability distribution (i.e., the higher the uncertainty, the more the probability distribution tends to be a uniform distribution) or an estimated value.

[0039] Traditional ship detection algorithms achieve ship detection in real-world scenarios by learning the location and category information of targets in the training set, without considering the situation of new categories appearing in real-world scenarios, which brings difficulties to the deployment of ship detection in real-world scenarios. In recent years, with the development of deep learning technology, some open-set recognition methods have been proposed and extended to some detection methods. However, these methods are for general computer vision tasks. In contrast, detecting ship targets in SAR images is more challenging. On the one hand, the inherent speckle noise in SAR images leads to data uncertainty. On the other hand, the difficulty in obtaining a large amount of training data leads to model cognitive difficulties, bringing huge cognitive uncertainty to the SAR ship detection task.

[0040] Uncertainty learning aims to estimate the uncertainty in deep learning. Most of the uncertainties proposed currently estimate the uncertainty by using the ensemble of multiple models or multiple sets of hyperparameters, but the overly complex processing process does not meet the real-time requirements of ship detection.

[0041] Therefore, the present invention discloses a SAR ship open-set detection method based on uncertainty learning, including the following steps: extracting the data representation of ships in the SAR image based on the backbone network, region proposal network, and region of interest alignment module of Faster R-CNN; respectively extracting the distribution parameters of the data representation through two parallel encoders, and resampling based on the distribution parameters to obtain the sampled features of ships in the SAR image; classifying the sampled features through a classifier to obtain the categories of ships in the SAR image; the category set of the classifier includes known categories, unknown categories, and background categories; among them, the encoders, classifier, and the backbone network, region proposal network, and region of interest alignment module of Faster R-CNN are optimized by joint training.

[0042] The present invention can transfer the data representation from the point space to the probability space by resampling the data representation extracted by Faster R-CNN without losing gradients, which can reduce the uncertainty in the classification process, thereby reducing the impact of cognitive uncertainty on the detection accuracy in the detection process.

[0043] Specifically, the present invention is equipped with two modules: an entropy-aware module and an unknown-aware module. In the entropy-aware module, entropy is used to measure the uncertainty of data and reduce the entropy to further optimize the data representation, thereby mitigating the negative impact of data uncertainty. In the unknown-aware module, the prediction of known classes and the estimation of epistemic uncertainty are jointly interactively optimized to reduce the epistemic uncertainty of known classes and detect unknown classes. Therefore, the method of the present invention can accurately detect unknown-class ships while accurately identifying known-class ships.

[0044] The present invention is based on uncertainty learning in deep learning, and simultaneously focuses on the data uncertainty caused by data and the epistemic uncertainty brought by model cognition in deep learning, realizing the open-set SAR ship detection task in an open scenario, which can effectively overcome the false alarms caused by speckle noise in SAR images and accurately locate and identify known-class and unknown-class ships. For the problem of false alarms caused by speckle noise in SAR images, the present invention proposes an entropy-aware module to optimize the data representation by reducing entropy. In addition, for the epistemic uncertainty caused by insufficient model cognition, the present invention proposes an unknown-aware module to jointly interactively optimize the prediction of known classes and the estimation of epistemic uncertainty, thereby reducing the epistemic uncertainty of known-class ships and obtaining an understanding of the unknown. Different from conventional sampling-based uncertainty estimation methods, the method adopted by the present invention is a real-time method and can be applied to detection tasks with high requirements for real-time performance.

[0045] In the actual sea condition environment, the types of ships are complex, and in actual applications, it is difficult for existing data sets to cover all ship types. Therefore, it is a challenging and widely applicable task to detect unknown-class ships on the premise of correctly detecting known-class ships. Traditional ship detection algorithms are based on the closed-set hypothesis and detect the actual scene by learning the target classes and location information in the training set, and cannot accurately detect and identify unknown-class ships in the scene. The present invention proposes an entropy-aware module and an unknown-aware module to respectively focus on the data uncertainty and epistemic uncertainty in ship detection. The SAR ship detection method based on uncertainty learning is as Figure 1 shown.

[0046] In the present invention, Faster R-CNN is selected as the baseline, which includes a backbone network, a region proposal network, and a region of interest alignment module, and the detection head is improved by the entropy-aware module and the unknown-aware module of the present invention. First, the image extracts multi-level features through the backbone network. Subsequently, the region proposal network selects the regions where the targets may exist and gives the location information. The region of interest alignment module maps the regions where the targets may exist selected by the region proposal network to the features extracted by the backbone network, and then the features are sent into the entropy-aware module and the unknown-aware module.

[0047] The proposed entropy-aware module solves the negative impact of data uncertainty on data representation. Based on the data representation optimized by the entropy-aware module, the unknown-aware module reduces the cognitive uncertainty of known classes and obtains the perception of the unknown. In the unknown-aware module, the probabilities of unknown classes and known classes are predicted. Regarding joint training, the classification loss and cross-entropy loss of the classifier, the uncertainty loss of the encoder, and the box regression loss and region proposal network loss in Faster R-CNN are included in the present invention.

[0048] The structure of the entropy-aware module is as Figure 2 shown. Based on the consensus that lower entropy corresponds to lower uncertainty and higher entropy corresponds to higher uncertainty, the present invention optimizes data representation in the entropy-aware module.

[0049] Specifically, resampling based on distribution parameters includes: sampling random numbers from a normal distribution; generating sampling features based on the random numbers and distribution parameters.

[0050] More specifically, the data representation x of ships in the SAR image is first converted into the form of a probability distribution by an encoder in the entropy-aware module. The encoder consists of a simple single-layer convolutional layer and an L2 norm layer. That is, by sending the data representation x into two encoders with the same structure but different parameters respectively, the mean and variance of its distribution, that is, the distribution parameters μ and σ, can be obtained. Then sampling can be performed from the distribution, the entropy of each data representation can be calculated, and the data representation can be optimized. It is important to emphasize that the sampling process is non-differentiable, so the reparameterization trick needs to be used to enable the model to still utilize the gradient.

[0051] Specifically, the present invention adopts the reparameterization trick. First, sample a random number a ∈ N(0,1) from a normal distribution. Subsequently, generate x′ as the corresponding sampling feature, that is, x′ = μ + a·σ. Through the reparameterization process, the loss of gradient when going from the point space to the probability space can be prevented.

[0052] Then, taking the sampling feature x′ as an observation, the probability of the sampling feature is obtained: The entropy of each sampling feature can be calculated using the probability p(x′), and the calculation formula is as formula (1).

[0053] E(x) = -α·p(x′)log p(x′) (1)

[0054] To optimize the representation to increase the difference between the inherent noise and the target, the present invention uses E(x) as the loss function L IE to penalize the data representation with high uncertainty (i.e., the degree of chaos of the data) during training, that is, L IE = -α·p(x′)log p(x′), where α is a hyperparameter that can be adjusted according to the learning situation of the model.

[0055] The structure of the unknown perception module is as shown in Figure 3 , lacking sufficient training data sets makes it difficult for the model to learn effectively, resulting in a high degree of cognitive uncertainty. To estimate the cognitive uncertainty, the present invention first establishes a new classifier setting. Different from the classifier setting of the closed-set detection method, the present invention expands the classifier dimension so that its categories include k known classes, an unknown class C for cognitive uncertainty u and a background class C bg , that is, the unknown class is the k + 1-th category, and the background class is the k + 2-th category. However, due to the lack of unknown class C u label supervision to teach the network when to give low or high uncertainty scores during training, it is difficult to learn such a classifier.

[0056] Therefore, the present invention uses a form of conditional probability to supervise the model's ability to estimate uncertainty by reducing uncertainty in the training of known classes.

[0057] First, the true category of the target (i.e., the target to be detected corresponding to the sampled feature) is labeled as C′. At the same time, since all true categories are known, C′ also represents a known class. Then, when the recognized category is incorrect, the corresponding category label is labeled as C′ n , the recognition result of the unknown class is labeled as U, and the probability of U occurring conditional on C′ n occurring can be calculated, as shown in formula (2).

[0058]

[0059] Among them, the numerator p(UC′ n ) represents the probability that the recognition is incorrect and the recognized class is the unknown class at the same time, and the denominator p(C′ n ) represents the probability of incorrect recognition. For simplicity, this conditional probability is denoted as p(U*), that is, p(U*) represents the probability of mislabeling the sampled feature with a known true category as the unknown class. Since C′ n must occur when U appears, it can be inferred that p(UC′ n ) = p(U). Therefore, formula (2) can be equivalent to equation (3).

[0060]

[0061] Combining formula (3) with Softmax gives the calculation formula for p(U*), as shown in formula (4).

[0062]

[0063] Among them, S uClassification data indicating that the sampled feature is marked as an unknown class, S j Classification data indicating that the sampled feature is marked as a non-known class, C j Indicates the j-th class in the class set, and C′ represents the known class. This uncertainty p(U*) will be used as the probability of the occurrence of the unknown class. The higher the uncertainty, the higher the probability that the class is unknown. The goal during training is to optimize p(U*) and minimize the epistemic uncertainty of the known classes. Therefore, the present invention designs the classification loss L UE As shown in formula (5).

[0064] L UE =-log(p(U*)) (5)

[0065] This classification loss will be jointly optimized with the cross-entropy loss L cls to estimate the probability and uncertainty of the known classes, L cls See formula (6).

[0066] L cls =-log(p(C′)) (6)

[0067] where p(C′) represents the probability of correctly marking the sampled feature with the true class being the known class, and C′ represents the known class.

[0068] In addition, to verify the accuracy of the method of the present invention, a set of open-set validation datasets was first constructed based on SRSDD-v1.0. The SRSDD-v1.0 dataset includes 2,884 ships in six categories: oil tankers, bulk carriers, fishing boats, law enforcement boats, dredgers, and container ships. The SRSDD-v1.0 dataset was divided into a training set and a test set. The training set only contains known classes, while the test set contains known classes and unknown classes. The dataset adopted a multi-task setting to verify SAR open-set ship detection.

[0069] For task one (denoted as T 1 ), the dataset for this task selects container ships as the unknown class from the dataset, and uses all datasets containing container ship marking information as the test set to ensure that the training set does not contain any marking information about the unknown class.

[0070] For task two (denoted as T 2 ), T 1 is extended by considering container ships and law enforcement boats as unknown classes, and the dataset with container ship and law enforcement boat marking information is used as the test set.

[0071] In addition, to comprehensively evaluate the reliability of the present invention, additional tasks with higher score thresholds were created for tasks T 1 and T 2 , respectively denoted as T1* and T 2* In the low-threshold task T 1 and T 2 the fractional threshold is set to where n is the number of classes. In the high-threshold task T 1* and T 2* the fractional threshold is set to 0.5. When the confidence of the predicted target is lower than the threshold, the prediction is considered inaccurate.

[0072] To comprehensively verify the present invention and quantify the performance metrics, first calculate the average precision AP, closed-set metric AP k , open-set metric AP u , and average metric AP K∪U for each class. All of the above verification metrics are compared with other existing well-known open-set detection methods, and the comparison methods include: Dropout Sampling (DS), PROSER, ODL, and OpenDet. The verification results are shown in Table 1, and the detection visualization is as shown in Figure 4 . As can be seen from Table 1, the method of the present invention has higher detection accuracy than the existing open-set detection methods in different task modes and different metrics.

[0073] Table 1 Open-set detection accuracy of the present invention and comparative methods under multiple tasks

[0074]

[0075]

[0076] In summary, the unknown-aware module designed by the present invention not only targets the recognition of unknown classes but also optimizes the recognition of known classes. Through the joint optimization of the estimation of known and unknown classes, while reducing the cognitive uncertainty of known-class targets, the algorithm's detection and recognition capabilities for unknown classes are improved.

[0077] Therefore, the uncertainty-based SAR intelligent open-set detection method of the present invention is used to solve the challenging open-set ship detection problem in SAR images. This method not only overcomes the false alarms in SAR ship detection caused by speckle noise in SAR images but also can accurately detect known classes and simultaneously accurately detect unknown classes.

[0078] The present invention is based on uncertainty learning in deep learning, and simultaneously focuses on the data uncertainty caused by data in deep learning and the cognitive uncertainty brought by model cognition. It realizes the open-set SAR ship detection task in an open scenario, can effectively overcome the false alarms caused by speckle noise in SAR images, and accurately locate and identify known and unknown category ships. For the problem of false alarms caused by speckle noise in SAR images, the present invention proposes an entropy-aware module to optimize data representation by reducing entropy. In addition, for the cognitive uncertainty caused by insufficient model cognition, the present invention proposes an unknown-aware module to jointly and interactively optimize the prediction of known categories and the estimation of cognitive uncertainty, thereby reducing the cognitive uncertainty of known category ships and obtaining an understanding of the unknown. Different from conventional sampling-based uncertainty estimation methods, the method adopted by the present invention is a real-time method and can be applied to detection tasks with high requirements for real-time performance.

[0079] The present invention also discloses a SAR ship open-set detection device based on uncertainty learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned method is implemented.

[0080] The present invention also discloses an embodiment that provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments are implemented.

[0081] The present invention also provides a computer program product. When the computer program product runs on a data storage device, the data storage device can be made to implement the steps in the above-mentioned various method embodiments when executed.

[0082] When the integrated unit module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present invention, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in the computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be realized. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate forms, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the storage device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc.

[0083] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0084] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in the present invention can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

[0085] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0086] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Claims

1. A SAR ship open set detection method based on uncertainty learning, characterized in that: The following steps are involved: The data representation of ships in SAR images is extracted based on the Faster R-CNN backbone network, region proposal network and region of interest alignment module; Extracting distribution parameters represented by the data respectively through two parallel encoders, and resampling based on the distribution parameters to obtain sampling features of the ship in the SAR image; The sampling features are classified by a classifier to obtain the category of the ship in the SAR image; the category set of the classifier includes a known class, an unknown class and a background class; The encoder, the classifier, the backbone network of Faster R-CNN, the region proposal network and the region of interest alignment module are jointly trained to optimize parameters.

2. The SAR ship open set detection method based on uncertainty learning as claimed in claim 1, characterized in that: Resampling based on the distribution parameters includes: Sample random numbers from a normal distribution; The sampling feature is generated based on the random number and a distribution parameter.

3. The SAR ship open set detection method based on uncertainty learning as claimed in claim 2, characterized in that: Generating the sampling feature based on the random number and the distribution parameter comprises: x′=μ+a·σ, Wherein, x′ represents the sampling feature, μ and σ represent the mean and variance of the data characterization distribution respectively, and a represents a random number sampled from a normal distribution, a∈N(0,1).

4. A SAR ship open set detection method based on uncertainty learning as claimed in claim 2 or 3, characterized in that: The encoder consists of a single convolution layer and an L2 norm layer.

5. The SAR ship open set detection method based on uncertainty learning as claimed in claim 4, characterized in that: The joint training includes the classification loss and cross entropy loss of the classifier, the uncertainty loss of the encoder, and the box regression loss and region proposal network loss in FasterR-CNN.

6. The SAR ship open set detection method based on uncertainty learning as claimed in claim 5, characterized in that: The uncertain loss is: L IE =-α·p(x′)logp(x′), Among them, L IE represents uncertainty loss, α represents a hyperparameter, p(x′) represents the probability of the sampled feature, 7. The SAR ship open set detection method based on uncertainty learning as claimed in claim 5, characterized in that: The classification loss is: L UE =-log(p(U*)), Among them, L UE Represents the classification loss, and p(U*) represents the probability of mislabeling the sampled features whose true categories are known categories and labeling them as unknown categories.

8. The SAR ship open set detection method based on uncertainty learning as claimed in claim 7, characterized in that: The calculation method of p(U*) is: Among them, S u represents the classification data that marks the sampled features as unknown classes, S j Indicates the classification data that marks the sampled features as non-known classes, C j represents the jth category in the category set, and C′ represents the known category.

9. The SAR ship open set detection method based on uncertainty learning as claimed in claim 5 or 7, characterized in that: The cross entropy loss is: L cls =-log(p(C′)), Among them, L cls represents the cross entropy loss, p(C′) represents the probability of correctly labeling the sampled features whose true category is a known category, and C′ represents the known category.

10. A SAR ship open set detection device based on uncertainty learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images