A remote sensing image target detection method based on relation learning

Abstract

The present invention provides a remote sensing image target detection method based on relational learning, which belongs to the field of remote sensing images. The method comprises: for remote sensing images, obtaining candidate frames generated by a region generation network, dividing the candidate frames into foreground frames, reasoning frames and background frames; adopting a double-headed reasoning strategy, respectively constructing graph reasoning modules for classification and positioning tasks, respectively constructing visual relationship graphs and position relationship graphs between candidate frames for classification and positioning tasks through the constructed graph reasoning modules, performing relational reasoning simultaneously according to the constructed visual relationship graphs and position relationship graphs, and obtaining reasoning features of each candidate frame; splicing the candidate frame features and reasoning features generated by the region generation network, and performing feature fusion through a multi-layer perceptron, and classifying and positioning the candidate frames again. By adopting the present invention, the potential relationship information in the foreground and background of remote sensing images can be mined, thereby improving the detection accuracy of remote sensing image targets.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing images, and in particular to a remote sensing image target detection method based on relationship learning. Background Art

[0002] The ultimate goal of the target detection algorithm is to train a model similar to the human brain that can autonomously identify various targets in different scenarios. The various existing models simulate the learning process of the human brain in different ways. In order to enable the model to more clearly understand what "detection" is, the target detection model divides "detection" into two parallel subtasks: classification and positioning. In the deep learning target detection model, the classification and positioning tasks share a backbone network, process the feature map of the image in parallel, and jointly optimize the entire network with classification accuracy and detection box accuracy as indicators. Generally speaking, the existing deep learning target detection models can be divided into two categories according to whether candidate boxes are generated: two-stage models and one-shot models.

[0003] Most remote sensing image target detection models follow a two-stage detection approach: (1) generating candidate boxes and (2) classifying and locating the candidate boxes. Traditional methods use a sliding window method to generate candidate boxes and use the Histogram of Oriented Gradient (HOG) to extract features. Since early target detection models relied on manual feature extraction, the generalization of features was not strong, resulting in these models only having good detection effects on certain categories (such as ships, ports, and waters). However, for multi-category, complex background data sets, the detection results are often unsatisfactory. Since deep learning has achieved good results in the field of natural images, researchers have gradually transferred deep learning technology to the field of remote sensing images. Given that the fields in which remote sensing images are applicable have high requirements for detection accuracy, most methods use a two-stage model to process remote sensing images. However, in order to achieve real-time detection, people have also tried to use the detection idea of ​​a one-shot model.

[0004] Existing remote sensing image target detection methods are all based on deep learning methods. Based on the general two-stage model and one-shot model, the original model is modified to address the problems of complex remote sensing image scenes, large image scale, large rotation angle changes of objects, and multi-scale objects. In view of the excellent results achieved by Faster R-CNN in natural scenes, many remote sensing image target detection models have adopted the detection ideas of the Faster R-CNN framework. For example, Li et al. proposed a rotation-insensitive RPN, RI-RPN (Rotation Insensitive RPN). The network introduces anchors with rotation angles, which can effectively deal with the problem of angle changes of remote sensing targets. Xu et al. proposed DeformableFaster R-CNN to solve the problem of variable shapes and different scales of remote sensing targets. The model combines DCN (Deformable Convolutional Networks) and Faster R-CNN to improve the detection accuracy of the model for remote sensing images. Based on Faster R-CNN, SCRDet proposed a random orientation-oriented target detection framework and an effective feature fusion strategy to address the problems of uncertain orientation and complex scenes of remote sensing image targets. RoI Transformer proposes a module for the Two-stage model that can convert horizontal candidate boxes into rotated candidate boxes, which greatly improves the model's detection accuracy for rotated targets in remote sensing images.

[0005] Although the detection idea of ​​the two-stage model can bring good performance to the remote sensing image target detection model, in order to achieve real-time detection, some studies have tried to use the idea of ​​the one-shot model to detect remote sensing images. For example, Tang et al. referred to the idea of ​​SSD (Single Shot MultiBox Detector) and designed a model that can detect randomly oriented vehicles, AO-SSD (Arbitrary-Oriented SSD). The model first generates anchors of different aspect ratios and sizes at various positions in the feature map, and then predicts the offset of each anchor to the real box. The RI-SSD (Rotation Invariant SSD) proposed by Liu et al. replaces the original anchor in SSD with an anchor with a rotation angle, thereby predicting targets with rotation angles and giving the model rotation invariance. Liu et al. designed a model for detecting ships in any orientation, AO-YOLO (Arbitrary-Oriented YOLO), based on the structure of YOLO_v2. In 2018, Van et al. proposed YOLT (You Only Look Twice) based on the YOLO model to address the problems of dense and large-scale targets in remote sensing images.

[0006] Although remote sensing image target detection methods based on deep learning have achieved good detection results, they are migrated from general target detection algorithms (such as Faster R-CNN, YOLO, SSD). Due to the large number of occluded and blurred targets, small inter-class gaps and complex backgrounds in remote sensing images, complex networks are also difficult to extract effective target features, resulting in low detection accuracy. But is the complex background in remote sensing images really useless for target detection? This is not necessarily the case. Although the background of remote sensing images is more complex than that of natural images, the complex background also brings richer information. However, most of the existing remote sensing image target detection methods ignore the potential relationship information in remote sensing images, resulting in the inability of remote sensing image target detection methods to achieve very accurate results. Summary of the invention

[0007] The embodiment of the present invention provides a remote sensing image target detection method based on relationship learning, which can mine potential relationship information in the foreground and background of remote sensing images, thereby improving the detection accuracy of remote sensing image targets. The method includes:

[0008] S101, for the remote sensing image, obtain a candidate frame generated by a region generation network, and divide the candidate frame into a foreground frame, an inference frame, and a background frame; wherein the background frame containing local or global information of the real frame is called the inference frame;

[0009] S102, constructing a corresponding reasoning category for each category, and classifying and locating the foreground box, the reasoning box, and the background box for the first time;

[0010] S103, adopting a dual-headed reasoning strategy, constructing graph reasoning modules for classification and positioning tasks respectively, using the constructed graph reasoning module to use the weights of the classifier and regressor obtained in S102 as the visual semantic information and position semantic information of the category respectively, constructing a visual relationship graph and a position relationship graph between candidate boxes for classification and positioning tasks respectively, and performing relationship reasoning simultaneously based on the constructed visual relationship graph and position relationship graph to obtain the visual reasoning features and position reasoning features of each candidate box;

[0011] S104, concatenating the candidate frame features obtained in S101 and the corresponding visual reasoning features and position reasoning features obtained in S103, and performing feature fusion through a multi-layer perceptron to classify and locate the candidate frame again.

[0012] Furthermore, for the remote sensing image, obtaining a candidate frame generated by using a region generation network, and dividing the candidate frame into a foreground frame, an inference frame, and a background frame includes:

[0013] For remote sensing images, candidate boxes generated by the region generation network are obtained, and the candidate boxes are divided into foreground boxes, inference boxes and background boxes according to the intersection-over-union ratio between the candidate boxes and the true boxes.

[0014] Furthermore, the dual-headed reasoning strategy is adopted to construct graph reasoning modules for classification and positioning tasks respectively. The weights of the classifier and the regressor obtained in S102 are used as the visual semantic information and the position semantic information of the category through the constructed graph reasoning module. A visual relationship graph and a position relationship graph between candidate boxes are constructed for the classification and positioning tasks respectively. Relational reasoning is performed simultaneously based on the constructed visual relationship graph and position relationship graph, and the visual reasoning features and position reasoning features of each candidate box are obtained, including:

[0015] The first graph reasoning module is constructed for the classification task. The weight of the classifier obtained in S102 is used as the visual semantic information of the category through the first graph reasoning module. The classification score of the candidate box output by the classifier is used to map the visual semantic information of the category to the candidate box to obtain the visual node feature V cls ; According to the obtained V cls , use the graph autoencoder to adaptively learn the visual relationship between candidate boxes and obtain the visual node adjacency matrix A cls ; According to the obtained V cls and A cls , construct the visual relationship graph G between the candidate boxes cls = {V cls , A cls}; According to the constructed visual relationship graph G cls= {V cls , A cls}, using the first graph convolutional neural network to propagate graph information, realize the learning of reasoning features, and obtain the visual reasoning features of each candidate box;

[0016] A second graph reasoning module is constructed for the positioning task. The weight of the regressor obtained in S102 is used as the location semantic information of the category through the second graph reasoning module. The classification score of the candidate box output by the classifier is used to map the location semantic information of the category into the candidate box to obtain the location node feature V loc ; According to the obtained V loc , use the graph autoencoder to adaptively learn the position relationship between candidate boxes and obtain the position node adjacency matrix A loc ; According to the obtained V loc and A loc , construct the position relationship graph G between the candidate boxes loc = {V loc , A loc}; According to the constructed position relationship graph G loc = {V loc , A loc}, use the second graph convolutional neural network to propagate graph information, realize the learning of reasoning features, and obtain the position reasoning features of each candidate box;

[0017] Among them, the first graph reasoning module and the second graph reasoning module respectively target the classification and positioning tasks of target detection, and learn the reasoning features of their respective tasks in parallel.

[0018] Furthermore, the visual node feature V cls It is expressed as:

[0019] V cls =softmax(cls_socre)W cls

[0020] Among them, W cls Represents the weight of the classifier, cls_socre represents the classification score of the candidate box output by the classifier, and the softmax function is used to convert the value of cls_socre to between 0 and 1, indicating the probability p that the current candidate box belongs to each category. cls .

[0021] Furthermore, the position node feature V loc It is expressed as:

[0022] V loc =softmax(cls_socre)W loc_reshape

[0023] Among them, W loc_reshapeRepresents the semantic information of each category, represented by the weight W of the regressor loc After dimension transformation, cls_socre represents the classification score of the candidate box output by the classifier. The softmax function is used to convert the value of cls_socre to between 0 and 1, indicating the probability p of the current candidate box belonging to each category. cls .

[0024] Further, the V obtained according to cls , use the graph autoencoder to adaptively learn the visual relationship between candidate boxes and obtain the visual node adjacency matrix A cls include:

[0025] Use graph autoencoder to transform visual node features V cls Encode it into a latent variable Z, and use matrix multiplication as a decoder to learn the relationship information ε between the candidate boxes:

[0026] ε=ZZ T , with Z = encoder(V)

[0027] Where V = V cls , e i，j ∈ε represents the similarity between nodes, encoder represents the encoder;

[0028] Sample ε, that is, set the largest first k values ​​in each column of ε to 1 and the rest to 0, and get a sparse adjacency matrix A cls :

[0029]

[0030] Among them, A cls =A i，j , i, j∈[0, r-1], r represents the number of candidate boxes, and k represents the number of parameters that need to be regressed.

[0031] Furthermore, the visual relationship graph G constructed according to cls = {V cls , A cls}, the first graph convolutional neural network is used to propagate graph information and learn the reasoning features. The visual reasoning features of each candidate box include:

[0032] According to the constructed visual relationship graph G cls = {V cls , A cls}, using the first graph isomorphic network to perform V cls and A cls The information is propagated to obtain the visual reasoning features of each candidate box.

[0033] Furthermore, the position relationship graph G constructed according to cls = {V cls , A cls}, the second graph convolutional neural network is used to propagate graph information and learn the inference features. The position inference features of each candidate box are obtained, including:

[0034] According to the constructed position relationship graph G loc = {V loc , A loc}, using the second graph isomorphic network to perform V loc and A loc The information is propagated to obtain the position reasoning features of each candidate box.

[0035] The remote sensing image target detection method based on relationship learning described in the embodiment of the present invention has at least the following beneficial effects:

[0036] 1) Some existing remote sensing image target detection models ignore the mining of background information, resulting in many background features in remote sensing images being abandoned, making it impossible for the model to use the relationship between the foreground and the background to improve the representation ability of features. In order to mine the potential relationship information in the foreground and background of remote sensing images, this embodiment constructs a corresponding reasoning category for each category to distinguish between local information and global information of different categories, and mines the background features in the image by adding detection of some background boxes (i.e., reasoning boxes), and provides a basis for the graph reasoning module; in this way, the reasoning box detection can alleviate the imbalance problem of positive and negative samples in remote sensing image target detection on the one hand, and effectively mine the visual information and location information contained in the foreground and background on the other hand; and the increase in reasoning categories can enrich the categories in remote sensing images, thereby improving the effectiveness of relationship learning;

[0037] 2) Since target detection has two subtasks, classification and positioning, and the two subtasks focus on different information, this embodiment uses a dual-headed reasoning strategy. Specifically, the weights of the classifier and regressor obtained from the first classification and positioning are used as the visual semantic information and position semantic information of the category, respectively. Without adding any prior knowledge, a relationship graph between two candidate boxes is constructed for the classification and positioning tasks, including: a visual relationship graph and a position relationship graph; then, based on the constructed visual relationship graph and position relationship graph, a graph convolutional neural network is used to perform relationship reasoning at the same time, and reasoning features for classification and positioning are obtained respectively; finally, the two types of reasoning features and the original candidate box features are spliced, and a multi-layer perceptron is used for feature fusion to obtain new candidate box features for re-classification and positioning. In this way, the existing remote sensing image target detection model can be endowed with the ability of relationship reasoning, fully tapping the potential visual relationship information and position relationship information in the remote sensing image, and improving the performance of the remote sensing image target detection model from the perspective of relationship learning, thereby improving the detection accuracy of blurred, occluded and other targets.

[0038] 3) In order to perform effective relational reasoning, in this embodiment, a graph reasoning module is designed from three aspects: graph node feature representation, graph relationship learning, and information propagation. The graph reasoning module attempts to construct a visual relationship graph and a position relationship graph between the foreground box, the reasoning box, and the background box, thereby realizing relational reasoning. Specifically: First, in order to ensure the richness and representativeness of the node features, the present invention uses the weights of the classifier and regressor obtained by the first classification and positioning as the visual semantic information and position semantic information of the category, respectively, and uses the classification score cls_socre of the candidate box to map the semantic information of the category to the candidate box, thereby obtaining a graph node feature representation based on the candidate box. Secondly, in order to be able to adaptively learn the relationships between nodes, this embodiment uses a graph autoencoder to learn the direct connection relationship between each node, and obtains the adjacency matrix A between the nodes. cls and A loc Finally, a graph isomorphism network is used to propagate graph information and learn reasoning features for classification and positioning, including visual reasoning features and position reasoning features.

[0039] (4) This embodiment is lightweight and has good portability. It can be added to any two-stage or one-shot model and endowed with the ability of relational learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A schematic diagram of a flow chart of a remote sensing image target detection method based on relationship learning provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of candidate frames with different intersection-over-union ratios provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of multiple types of golf course targets provided by an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of a fuzzy aircraft target provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of the workflow of the graph reasoning module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0047] Remote sensing images record surface information. There is a certain overlap and correlation between the target area (foreground) and the background area (background). Therefore, the background contains local features or global features of the foreground. When identifying targets in complex remote sensing scenes, the rational use of background information can enhance target features and help the target detection model make more accurate detections. For example, targets around ships are likely to be ports or ships, and targets on the road are likely to be cars. Therefore, mining the potential relationship information between candidate boxes and using the relationship information for inference detection can effectively improve the accuracy of remote sensing image target detection. In view of the above situation, this embodiment provides a remote sensing image target detection method based on relationship learning, such as Figure 1 As shown, the following steps may be specifically included:

[0048] S101, for the remote sensing image, obtaining a candidate frame generated by using a region generation network, and dividing the candidate frame into a foreground frame, an inference frame, and a background frame;

[0049] In this embodiment, Faster R-CNN can be used as the benchmark model. First, the image features of the remote sensing image are extracted through the backbone network of Faster R-CNN, and the candidate boxes are generated by the region generation network, and the candidate boxes are divided into foreground boxes, inference boxes, and background boxes according to the intersection-over-union ratio between the candidate boxes and the real boxes.

[0050] In the target detection task, the intersection-over-union ratio is often used to describe the overlap of two candidate boxes. The intersection-over-union ratio refers to the ratio of the overlapping area of ​​two candidate boxes to the combined area. Figure 2 As shown, (a) shows the foreground frame and the true frame, (b) shows the inference frame, and (c) shows the background frame. The solid frame is the true frame of the tennis court, and the dotted frame is the generated candidate frame. In general target detection models, only the foreground frame is classified and located, and all background frames are classified as background frames and are not located. This is because the gap between the background frame and the true frame is too large to perform accurate regression. Due to the large amount of background in remote sensing images, there is an extremely serious imbalance problem between positive and negative samples during the detection process, which greatly reduces the efficiency of target detection. Comparison Figure 2 By looking at (b) and (c) in the figure, we can find that although the IoU ratio between the background box and the real box in (b) is less than 0.5, it contains local or global information of the real box, and the IoU ratio between the candidate box and the real box in (c) is almost 0, which contains a lot of redundant information. In this embodiment, the background box containing local or global information of the real box is called the inference box, and the background box containing a lot of redundant information is the real background box. During the training process, the IoU ratio is used as a threshold to distinguish the foreground box, the inference box, and the background box.

[0051] S102, constructing a corresponding reasoning category for each category, and classifying and locating / regressing the foreground box, the reasoning box, and the background box for the first time;

[0052] In this embodiment, after the candidate boxes obtained in S101 are classified and located for the first time using Faster R-CNN, the weights of the classifier and regressor obtained by the first classification and location are extracted.

[0053] In this embodiment, the inference frame often contains local or global information of the real frame, and the inference frames of different categories are also different. In order to mine the potential relationship information in the foreground and background of remote sensing images, this embodiment constructs corresponding inference categories for each category to distinguish local information and global information of different categories. During the training and testing process, a visual relationship diagram and a position relationship diagram between the foreground frame, inference frame, and background frame will be constructed; during the training process, the background frame, foreground frames of different categories, and inference frames will be classified and located, so as to subsequently mine the visual node features V contained in the foreground and background of remote sensing images. cls and position node feature V loc; During the test, in order to ensure the validity of the prediction box, the inference box was not detected. The inference box can be regarded as a "hard example", which is discarded in general target detection models. The inference box detection can effectively alleviate the imbalance problem of positive and negative samples in the remote sensing image target detection task, generate more "hard examples" during the training process, and improve the robustness of the model. In addition, by adding the detection of the inference category, the visual information contained in the foreground and background of the remote sensing image can be mined (referring to: V cls ) and location information (referring to: V loc ), providing a basis for reasoning about the relationship between background and foreground.

[0054] S103, adopts a dual-headed reasoning strategy, and constructs a graph reasoning module (GRM) for classification and positioning tasks respectively. Through the constructed graph reasoning module, the weights of the classifier and regressor obtained in S102 are used as the visual semantic information and position semantic information of the category respectively, and a visual relationship graph and a position relationship graph between candidate boxes are constructed for classification and positioning tasks respectively. Relational reasoning is performed simultaneously based on the constructed visual relationship graph and position relationship graph to obtain the visual reasoning features and position reasoning features of each candidate box;

[0055] The target detection task consists of two subtasks: classification and positioning. Classification focuses on the visual information contained in the candidate frame, while positioning focuses on the position information contained in the candidate frame. In this embodiment, it is hoped that the accuracy of the classification and positioning tasks can be improved by relational reasoning. However, since classification and positioning focus on different information, a dual-headed reasoning strategy is used to handle them separately. That is, the visual relationship and position relationship between the foreground frame, the reasoning frame, and the background frame are used to perform reasoning learning on the classification task and the positioning task respectively, thereby improving the accuracy of classification and positioning;

[0056] 1) Visual relationship analysis

[0057] In the problem of large-scale target detection, people often use visual relationship reasoning to solve the long-tail distribution problem in the data set. These methods often use the visual features between candidate boxes of different categories to build a relationship graph for reasoning detection, which is suitable for situations where there are a large number of categories in a single image. However, not every remote sensing image has a rich variety of categories, and there are still many blurred and obscured targets. In order to more effectively perform visual relationship reasoning in remote sensing images, in this embodiment, target categories are added through reasoning box detection, and each category has a corresponding reasoning class (category of reasoning box). In this way, the visual features of the target can be enhanced and visual relationship reasoning can be achieved by constructing the visual relationship between the foreground box, the reasoning box, and the background box. Figure 3 Demonstrates the idea of ​​visual relationship reasoning. Figure 3The tennis court target in (a) is small and is easy to be missed or misdetected; 3(b) shows the visual relationship between the tennis court foreground frame and other candidate frames. Among them, the solid line frame represents the tennis court foreground frame and the baseball field foreground frame, and the dotted line frame represents the inference frame around the foreground frame. This relationship can be used to infer the category of the basketball court and improve the classification accuracy of the basketball court. The inference frame of the basketball court contains local or global information of the basketball court, and other courts such as tennis courts and baseball fields often appear nearby. These visual information can help the model infer that the category of the foreground frame is more likely to be a basketball court. Through the above analysis, it can be found that by constructing a visual relationship between the foreground frame and the inference frame and performing inference prediction, the classification accuracy of remote sensing targets can be improved.

[0058] 2) Position relationship analysis

[0059] Blurred and occluded targets in remote sensing images can also make it difficult to accurately locate targets, because the location information provided by low-quality target frames is not rich enough. In addition, many targets in remote sensing images are large in size and difficult to regress, resulting in inaccurate positioning. However, since there is no significant positional relationship between the candidate frames of each category, there are few methods for reasoning about the position information of the candidate frames. In visual relationship analysis, this embodiment performs reasoning by mining the visual relationship between the foreground frame, the reasoning frame, and the background frame. Similarly, the foreground frame and the reasoning frame of the same category also have a significant positional relationship in relative position, and this relationship can also be used to reason about the positional relationship between the candidate frames. Figure 4 Demonstrates the idea of ​​positional relationship reasoning. Figure 4 In (a), the aircraft target is blurry and its color is similar to the ground, making it difficult to locate. Figure 4 (b) shows the relationship between the aircraft foreground frame and the reasoning frames at different positions. The solid line frame represents the aircraft foreground frame, and the dotted line frame represents the reasoning frames at different positions of the aircraft. The reasoning frame can be used to infer the position of the aircraft, thereby improving the positioning accuracy. For example, the right wing of the aircraft is almost integrated with the ground, making it difficult to determine the position of the aircraft, but the position of the aircraft can be inferred by the position of the nose, tail and left wing. Through the above analysis, it is found that there is an obvious positional relationship between the foreground frame and the reasoning frame in the remote sensing image. Using this relationship for position reasoning can improve the detection accuracy of the model.

[0060] In order to perform visual relationship reasoning and position relationship reasoning at the same time, in this embodiment, a graph reasoning module is constructed for classification and positioning tasks respectively to learn the visual relationship and position relationship between candidate boxes. And in order to perform effective relationship reasoning, in this embodiment, the graph reasoning module is designed from three aspects: graph node feature representation, graph relationship learning, and relationship reasoning. The workflow of the graph reasoning module is as follows: Figure 5As shown, first, in order to obtain node information based on candidate boxes, the weights of the classifier and regressor obtained in S102 are used as the visual semantic information of the category (also called category semantic features) and position semantic information, respectively, and the classification score cls_socre of each candidate box output by the classifier in S102 is used to realize the mapping of category semantic information to candidate box information. In short, cls_socre is used as a mapping matrix to map the category semantic information to each candidate box one by one, so as to obtain the visual node feature V cls and position node feature V loc ; Then, the graph autoencoder is used to adaptively learn the visual relationship and position relationship between the candidate boxes to obtain the visual node adjacency matrix A cls and the position node adjacency matrix A loc ; In this way, a visual relationship graph G between candidate boxes is constructed cls = {V cls , A cls} and position relationship diagram G loc = {V loc , A loc Since the background still occupies the vast majority of remote sensing images, G cls = {V cls , A cls} and G loc = {V loc , A loc} are two sparse graphs. Finally, a graph convolutional neural network is used to propagate graph information, simulate the reasoning process, and obtain the reasoning features of each candidate box.

[0061] In this embodiment, for better understanding, the graph reasoning module is explained from three aspects: graph node feature representation, graph relationship learning, and relationship reasoning:

[0062] 1) Graph Node Feature Representation

[0063] Some existing methods directly use the candidate box features extracted by the backbone network as node features on the relationship graph and perform information propagation. However, there are many blurred and highly occluded candidate boxes in remote sensing images, which leads to insufficient node feature information and low information propagation efficiency. In order to solve this problem, this embodiment draws on the method in semi-supervised / unsupervised classification and uses the weight of the classifier as the visual semantic information of the category. This is because the weight of the classifier will contain high-level semantic information of the category during the continuous training and updating process. Target detection requires classification and positioning at the same time, so the weights of the original classifier and regressor can be used to construct visual node features respectively. and location node features Among them, r represents the number of candidate boxes, d cls and d locRespectively represent V cls and V loc The characteristic dimension of Represents the set of real numbers.

[0064] For visual relationship reasoning, first obtain the weight of the classifier obtained in S102 Among them, c represents the number of categories. Since an inference class is constructed for each category, there are a total of (2×c+1) categories including the background class, m cls Represents the dimension of the classifier weight. At this point, the visual semantic information of each category is obtained. Next, we need to construct the visual node feature V cls The classification score output by the classifier is the mapping matrix, and cls_socre is used to map the visual semantic information of the category into the candidate box to obtain the visual node feature V cls ;

[0065] V cls =softmax(cls_socre)W cls

[0066] Among them, the softmax function is used to convert the value of cls_socre to between 0 and 1, indicating the probability p that the current candidate box belongs to each category cls . cls As the weight of each class for the candidate box, W is multiplied by the matrix cls Mapped to each candidate frame, and finally obtain the visual node feature V cls .

[0067] For positional relationship reasoning, most methods use manually constructed position features as the position information of candidate boxes. For example, in SGRN (Spatial-aware Graph Relation Network), the distance and angle between the center points of the candidate boxes are calculated as position information. However, manually constructed features are often not very representative and require a lot of computing time. To solve this problem, in this embodiment, the weights of the regressor obtained in S102 are used. Construct the position node feature, where k represents the number of parameters that need to be regressed. For general target detection tasks, the value of k is 4, including the coordinates of the center point of the candidate box (x, y) and the length and width of the candidate box (w, h). Since the target orientation in the remote sensing image is not always horizontal, this embodiment adds an estimation of the candidate box rotation angle θ, so the value of k is 5. W loc It can be regarded as the high-level semantic information of each category for each regression parameter. Similarly, cls_socre is used to map it to each candidate box. Before that, W locPerform dimension transformation to obtain semantic information of each category Then use cls_socre to map:

[0068] V loc =softmax(cls_socre)W loc_reshape

[0069] In this way, the feature representation of the graph node based on the candidate box is obtained: the visual node feature V cls and position node feature V loc .

[0070] 2) Graph Relationship Learning

[0071] To construct a visual relationship graph G between candidate boxes cls = {V cls , A cls} and position relationship diagram G loc = {V loc , A loc}, we also need to obtain the adjacency matrix A between the nodes cls and A loc In order to adaptively learn the relationship between nodes, the present invention refers to the idea of ​​graph autoencoder. For the convenience of description, in this embodiment, the node features on the two relationship graphs are uniformly recorded as V, and the adjacency matrix is ​​uniformly recorded as A.

[0072] The present invention draws on the idea of ​​graph autoencoders, uses a learnable graph autoencoder to encode V into a latent variable Z, and uses matrix multiplication as a decoder to learn the relationship information ε between nodes, that is:

[0073] ε=ZZ T , with Z = encoder(V)

[0074] Among them, the encoder is represented by a two-layer fully connected layer. l represents the feature dimension of Z. Through matrix multiplication, Z is decoded into ε, e i，j ∈ε can be regarded as the similarity between nodes.

[0075] Since there are a lot of redundant background frames in remote sensing images, it is impossible to connect all candidate frame nodes. These redundant connections will reduce the efficiency of the graph reasoning module. To solve this problem, in this embodiment, ε is sampled, that is, the first k largest values ​​in each column of ε are set to 1, and the rest are set to 0. Thus, a sparse adjacency matrix A is obtained:

[0076]

[0077] Among them, i, j∈[0, r-1]. The above method can effectively avoid redundant edge connections in the relationship graph and ensure the rationality and efficiency of relationship reasoning.

[0078] 3) Relational Reasoning

[0079] In order to obtain inference features for classification and regression, the visual relationship graph G cls = {V cls , A cls} and position relationship diagram G loc = {V loc , A loc} Perform subsequent analysis and adopt appropriate information propagation strategies to learn inference features, thereby enhancing the accuracy of target classification and positioning. Since graph convolutional neural networks have demonstrated extremely strong feature extraction capabilities in the graph field, this embodiment uses graph convolutional neural networks for information propagation. Graph convolutional neural networks are divided into two categories: graph convolutional neural networks based on the spectral domain and graph convolutional neural networks based on the spatial domain. Since the former can only perform direct learning, requiring all graphs to share an adjacency matrix, and the visual relationship graph and position relationship graph constructed in the target detection task have different graph structures, this embodiment uses a graph convolutional neural network based on the spatial domain to further analyze the graph. The structure of the graph convolutional neural network is as follows: Figure 5 As shown, Figure 5 The categories such as airplane, car, ball, etc. represent the semantic features between categories represented by the weights of the classifier or regressor, X a , X b , X c , X k Represents the input node features, Z a , Z b , Z c , Z k , Z1, Z2, ..., Z r In this embodiment, a graph convolutional neural network (GIN) with the same representation capability as graph isomorphism is used for information propagation. In this embodiment, the information propagation method of GIN is used to perform relationship reasoning between candidate boxes.

[0080] GIN provides three ways to aggregate adjacent nodes: sum, average, and maximum. It has been proven that the sum aggregation method allows GIN to obtain the maximum representation capability. Therefore, in this embodiment, summation is used as the aggregation method, and a multi-layer perceptron is used to realize the mapping of bottom-level node features to high-level node features:

[0081]

[0082] Among them, MLP() represents the multi-layer perceptron algorithm. represents the feature of node i in the t-th layer of graph convolution, is the set of neighbor nodes of node i.

[0083] In this embodiment, the formula By propagating node information, we can finally obtain the visual reasoning features of each candidate box. and location inference features d cls and d loc Respectively represent f' cls and f' loc Dimension.

[0084] S104, concatenating the candidate frame features obtained in S101 and the corresponding visual reasoning features and position reasoning features obtained in S103, and performing feature fusion through a multi-layer perceptron to classify and locate the candidate frame again.

[0085] In this embodiment, the candidate frame feature f obtained in S101 and its corresponding inference feature f' cls 、f' loc The two layers are spliced ​​and feature fused using the fully connected layers to obtain the enhanced candidate box feature f new , classify and locate the candidate boxes again.

[0086] This embodiment can be added to any Two-stage or One-shot model without any prior knowledge.

[0087] Finally, in order to verify the effectiveness of the remote sensing image target detection method based on relational learning provided by an embodiment of the present invention, experiments were conducted on the DOTA dataset to verify the effectiveness of each strategy; wherein the DOTA dataset is a public remote sensing dataset with a rotation angle, which contains 2806 remote sensing images, each with a size ranging from 800×800 pixels to 4000×4000 pixels, and annotated with 188282 instances of different scales, directions, and aspect ratios, with a total of 15 categories, see Table 1.

[0088] Table 1 DOTA dataset categories and their subscripts

[0089] <![CDATA[C11]]> <![CDATA[C12]]> <![CDATA[C13]]> <![CDATA[C14]]> <![CDATA[C15]]> <![CDATA[C16]]> <![CDATA[C17]]> <![CDATA[C18]]> airplane Baseball Field bridge Track and Field Small vehicles Large vehicles Boat tennis court <![CDATA[C19]]> <![CDATA[C110]]> <![CDATA[C111]]> <![CDATA[C112]]> <![CDATA[C113]]> <![CDATA[C114]]> <![CDATA[C115]]> Basketball Court water tank football field a roundabout swimming pool port Helicopters

[0090] In the experiment, this embodiment was introduced into four benchmark models: two-stage models Faster R-CNN and CascadeR-CNN, and one-shot models RetinaNet and YOLO_v3, and this embodiment was compared with the above benchmark models one by one. The results in Table 2 show that this embodiment can improve the mean average precision (mAP) of different models by 2% to 3%, which means that this embodiment can give different benchmark models the ability to reason about relations, and further improve the detection accuracy of remote sensing image target detection models. The strategies used in each experiment in Table 2 are described as follows:

[0091] Base-line: Benchmark models used in experiments, such as Faster R-CNN, Cascade R-CNN, RetinaNet, and YOLO_v3.

[0092] SGRN: Reproduces the idea of ​​SGRN (Spatial-aware Graph Relation Network) based on the baseline model.

[0093] DH-GRN: A dual-headed graph reasoning network is added based on the baseline model.

[0094] Table 2 Detection results of DOTA dataset

[0095]

[0096] The remote sensing image target detection method based on relationship learning described in the embodiment of the present invention has at least the following beneficial effects:

[0097] 1) Some existing remote sensing image target detection models ignore the mining of background information, resulting in many background features in remote sensing images being abandoned, making it impossible for the model to use the relationship between the foreground and the background to improve the representation ability of features. In order to mine the potential relationship information in the foreground and background of remote sensing images, this embodiment constructs a corresponding reasoning category for each category to distinguish between local information and global information of different categories, and mines the background features in the image by adding detection of some background boxes (i.e., reasoning boxes), and provides a basis for the graph reasoning module; in this way, the reasoning box detection can alleviate the imbalance problem of positive and negative samples in remote sensing image target detection on the one hand, and effectively mine the visual information and location information contained in the foreground and background on the other hand; and the increase in reasoning categories can enrich the categories in remote sensing images, thereby improving the effectiveness of relationship learning;

[0098] 2) Since target detection has two subtasks, classification and positioning, and the two subtasks focus on different information, this embodiment uses a dual-headed reasoning strategy. Specifically, the weights of the classifier and regressor obtained from the first classification and positioning are used as the visual semantic information and position semantic information of the category, respectively. Without adding any prior knowledge, a relationship graph between two candidate boxes is constructed for the classification and positioning tasks, including: a visual relationship graph and a position relationship graph; then, based on the constructed visual relationship graph and position relationship graph, a graph convolutional neural network is used to perform relationship reasoning at the same time, and reasoning features for classification and positioning are obtained respectively; finally, the two types of reasoning features and the original candidate box features are spliced, and a multi-layer perceptron is used for feature fusion to obtain new candidate box features for re-classification and positioning. In this way, the existing remote sensing image target detection model can be endowed with the ability of relationship reasoning, fully tapping the potential visual relationship information and position relationship information in the remote sensing image, and improving the performance of the remote sensing image target detection model from the perspective of relationship learning, thereby improving the detection accuracy of blurred, occluded and other targets.

[0099] 3) In order to perform effective relational reasoning, in this embodiment, a graph reasoning module is designed from three aspects: graph node feature representation, graph relationship learning, and information propagation. The graph reasoning module attempts to construct a visual relationship graph and a position relationship graph between the foreground box, the reasoning box, and the background box, thereby realizing relational reasoning. Specifically: First, in order to ensure the richness and representativeness of the node features, the present invention uses the weights of the classifier and regressor obtained by the first classification and positioning as the visual semantic information and position semantic information of the category, respectively, and uses the classification score cls_socre of the candidate box to map the semantic information of the category to the candidate box, thereby obtaining a graph node feature representation based on the candidate box. Secondly, in order to be able to adaptively learn the relationships between nodes, this embodiment uses a graph autoencoder to learn the direct connection relationship between each node, and obtains the adjacency matrix A between the nodes. cls and A loc Finally, a graph isomorphism network is used to propagate graph information and learn reasoning features for classification and positioning, including visual reasoning features and position reasoning features.

[0100] (4) This embodiment is lightweight and has good portability. It can be added to any two-stage or one-shot model and endowed with the ability of relational learning.

[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A remote sensing image target detection method based on relationship learning, characterized in that: include: S101, for the remote sensing image, obtain a candidate frame generated by using a region generation network, and divide the candidate frame into a foreground frame, an inference frame, and a background frame; wherein the background frame containing local or global information of the real frame is called the inference frame; S102, construct corresponding reasoning categories for each category, and classify and locate the foreground box, reasoning box and background box for the first time; wherein, positioning is also called regression, and after the first classification and positioning, extract the weights of the classifier and regressor obtained by the first classification and positioning; S103, adopt a two-headed reasoning strategy, construct graph reasoning modules for classification and positioning tasks respectively, and use the weights of the classifier and regressor obtained in S102 as the visual semantic information and position semantic information of the category through the constructed graph reasoning module, respectively, and construct a visual relationship graph and a position relationship graph between candidate boxes for classification and positioning tasks respectively, and perform relationship reasoning simultaneously based on the constructed visual relationship graph and position relationship graph to obtain the visual reasoning features and position reasoning features of each candidate box; S104, concatenating the candidate frame features obtained in S101 and the corresponding visual reasoning features and position reasoning features obtained in S103, and performing feature fusion through a multi-layer perceptron to classify and locate the candidate frame again.

2. The method for remote sensing image target detection based on relationship learning according to claim 1, characterized in that: The method of obtaining a candidate frame generated by a region generation network for a remote sensing image and dividing the candidate frame into a foreground frame, an inference frame and a background frame includes: For remote sensing images, candidate boxes generated by the region generation network are obtained, and the candidate boxes are divided into foreground boxes, inference boxes and background boxes according to the intersection-over-union ratio between the candidate boxes and the true boxes.

3. The method for remote sensing image target detection based on relationship learning according to claim 1, characterized in that: The dual-headed reasoning strategy is adopted to construct graph reasoning modules for classification and positioning tasks respectively. The weights of the classifier and the regressor obtained in S102 are used as the visual semantic information and the position semantic information of the category through the constructed graph reasoning module. A visual relationship graph and a position relationship graph between candidate boxes are constructed for the classification and positioning tasks respectively. Relational reasoning is performed simultaneously based on the constructed visual relationship graph and position relationship graph, and the visual reasoning features and position reasoning features of each candidate box are obtained, including: The first graph reasoning module is constructed for the classification task. The weight of the classifier obtained in S102 is used as the visual semantic information of the category through the first graph reasoning module. The classification score of the candidate box output by the classifier is used to map the visual semantic information of the category to the candidate box to obtain the visual node feature V cls ; According to the obtained V cls , use the graph autoencoder to adaptively learn the visual relationship between candidate boxes and obtain the visual node adjacency matrix A cls ; According to the obtained V cls and A cls , construct the visual relationship graph G between the candidate boxes cls = {V cls , A cls }; According to the constructed visual relationship graph G cls = {V cls , A cls }, using the first graph convolutional neural network to propagate graph information, realize the learning of reasoning features, and obtain the visual reasoning features of each candidate box; A second graph reasoning module is constructed for the positioning task. The weight of the regressor obtained in S102 is used as the location semantic information of the category through the second graph reasoning module. The classification score of the candidate box output by the classifier is used to map the location semantic information of the category into the candidate box to obtain the location node feature V loc ; According to the obtained V loc , use the graph autoencoder to adaptively learn the position relationship between candidate boxes and obtain the position node adjacency matrix A loc ; According to the obtained V loc and A loc , construct the position relationship graph G between the candidate boxes loc = {V loc , A loc }; According to the constructed position relationship graph G loc = {V loc , A loc }, use the second graph convolutional neural network to propagate graph information, realize the learning of reasoning features, and obtain the position reasoning features of each candidate box; Among them, the first graph reasoning module and the second graph reasoning module respectively target the classification and positioning tasks of target detection, and learn the reasoning features of their respective tasks in parallel.

4. The method for remote sensing image target detection based on relationship learning according to claim 3, characterized in that: The visual node feature V cls It is expressed as: V cls =softmax(cls_socre)W cls Among them, W cls Represents the weight of the classifier, cls_socre represents the classification score of the candidate box output by the classifier, and the softmax function is used to convert the value of cls_socre to between 0 and 1, indicating the probability p that the current candidate box belongs to each category. cls .

5. The method for remote sensing image target detection based on relationship learning according to claim 3, characterized in that: The position node feature V loc It is expressed as: V loc =softmax(cls_socre)W loc_reshape Among them, W loc_reshape Represents the semantic information of each category, represented by the weight W of the regressor loc After dimension transformation, cls_socre represents the classification score of the candidate box output by the classifier. The softmax function is used to convert the value of cls_socre to between 0 and 1, indicating the probability p of the current candidate box belonging to each category. cls .

6. The method for remote sensing image target detection based on relationship learning according to claim 3, characterized in that: The V cls , use the graph autoencoder to adaptively learn the visual relationship between candidate boxes and obtain the visual node adjacency matrix A cls include: Use graph autoencoder to transform visual node features V cls Encode it into a latent variable Z, and use matrix multiplication as a decoder to learn the relationship information ε between the candidate boxes: ε=ZZ T ,with Z=encoder(V) Where V = V cls , e i，j ∈ε represents the similarity between nodes, encoder represents the encoder; Sample ε, that is, set the largest first k values ​​in each column of ε to 1 and the rest to 0, and get a sparse adjacency matrix A cls : Among them, A cls =A i，j , i, j∈[0, r-1], r represents the number of candidate boxes, and k represents the number of parameters that need to be regressed.

7. The method for remote sensing image target detection based on relationship learning according to claim 3, characterized in that: The visual relationship graph G constructed according to cls = {V cls , A cls }, the first graph convolutional neural network is used to propagate graph information and learn the reasoning features. The visual reasoning features of each candidate box include: According to the constructed visual relationship graph G cls = {V cls , A cls }, using the first graph isomorphic network to perform V cls and A cls The information is propagated to obtain the visual reasoning features of each candidate box.

8. The method for remote sensing image target detection based on relationship learning according to claim 3, characterized in that: The position relationship graph G constructed according to loc = {V loc , A loc }, the second graph convolutional neural network is used to propagate graph information and learn the inference features. The position inference features of each candidate box are obtained, including: According to the constructed position relationship graph G loc = {V loc , A loc }, using the second graph isomorphic network to perform V loc and A loc The information is propagated to obtain the position reasoning features of each candidate box.

Images