Remote sensing image target detection method and system based on consistency relation reasoning

Abstract

The invention provides a remote sensing image target detection method and system based on consistency relation reasoning, and relates to the technical field of target detection, in order to solve the problem that the detection precision of an existing remote sensing image target detection method is reduced during a target detection task under the condition that visual features are limited. Remote sensing images in the data set are preprocessed; constructing a target detection network based on consistency relation reasoning; training the constructed target detection network based on the preprocessed remote sensing image; performing detection precision evaluation on the trained target detection network, if the preset precision is not reached, returning to the previous step for re-training until the preset precision is reached, and outputting the target detection network; and target detection of the remote sensing image is realized based on the output target detection network. According to the method provided by the invention, the performance of target detection is improved by utilizing a visual relationship, and reliable reasoning can be carried out under the condition of not depending on visual features.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a remote sensing image target detection method and system based on consistency relationship reasoning. Background Art

[0002] The remote sensing target detection task aims to accurately locate the position of the object of interest and determine its category by rotating the bounding box. In recent years, the remote sensing image target detection task has become one of the current research hotspots and plays an important role in many research fields. At present, the existing remote sensing image target detection algorithms have made major breakthroughs in rotation invariant feature extraction, rotation bounding box representation, and continuity of angle regression, effectively improving the detection performance of detectors for targets with arbitrary orientations, large aspect ratios, and dense distribution. However, most of these existing methods focus on locating and classifying objects of interest based on regional features, ignoring the context-related information between objects. Under conditions where visual features are limited, such as small target size, blurred images, or severe occlusion, the detection performance drops sharply.

[0003] Introducing contextual relationship reasoning can significantly improve the performance of target detectors. Existing methods based on relationship reasoning can be divided into two categories depending on whether external prior knowledge is introduced. Among them, the first type of methods integrate external prior knowledge by defining a knowledge graph within the network. However, they only consider regional features and cannot achieve reliable relationship reasoning when visual features are limited. The second type of methods aggregates spatial or semantic priors through contextual information fusion operations, and constructs contextual relationships between objects through graph structures to improve the detector's ability to locate objects. Since this method constructs multiple local graphs for information propagation, its performance is greatly affected by the way the relationship graph is initialized.

[0004] In summary, the existing remote sensing image target detection methods have obvious shortcomings when dealing with target detection tasks under conditions with limited visual features. Further research is needed on target detection methods based on relational reasoning to improve the accuracy and robustness of target detection in complex environments. Summary of the invention

[0005] In order to solve the problem that the existing remote sensing image target detection algorithms have degraded performance when dealing with target detection tasks with limited visual features such as small target size, blurred image or severe occlusion, the present invention proposes a remote sensing image target detection method based on consistency relationship reasoning under limited visual feature conditions, which uses visual relationships to improve the performance of target detection.

[0006] The technical solution adopted by the present invention is: a remote sensing image target detection method based on consistency relationship reasoning under limited visual feature conditions, and the specific steps of the method are as follows: Step 1: Obtain a remote sensing image dataset and preprocess the remote sensing images in the dataset; Step 2: Build an object detection network based on consistency relationship reasoning; Step 3: Based on the preprocessed remote sensing image, train the target detection network constructed in step 2; Step 4: Evaluate the detection accuracy of the trained target detection network. If the preset accuracy is not reached, return to step 3 and retrain until the preset accuracy is reached, and output the target detection network. Step 5: Implement target detection of remote sensing images based on the target detection network output from step 4.

[0007] Furthermore, the specific process of step 1 includes: Step 1.1: Select remote sensing image datasets DOTA and DIOR-R; Step 1.2: Preprocess the selected data sets for different detection tasks; Among them, different detection tasks include remote sensing image target detection tasks and remote sensing image occluded target reasoning tasks; For the remote sensing image object detection task, no preprocessing is performed on the DOTA and DIOR-R datasets; For the remote sensing image occluded target reasoning task, 25% of the total number of objects in each image in the DOTA and DIOR-R datasets are selected and the selected targets are divided into m × m , randomly select 75% of the image blocks and apply random cloud occlusion, and record the data sets after applying random cloud occlusion as Occluded DOTA and Occluded DIOR-R respectively.

[0008] Furthermore, the object detection network includes a region proposal network, a relationship proposal network and a consistency reasoning network, wherein: Region Proposal Network, which uses a trained two-stage remote sensing image object detector as a candidate box extractor to detect unoccluded objects and locate the regions of occluded objects; The relation proposal network is used to construct coarse relation pairs based on the detection results of the region proposal network, and select the coarse relation pairs with the greatest possibility of interaction as relation suggestions; The consistent relation reasoning network is used to build distribution consistency features based on the spatial and semantic consistency of object distribution learned by relation proposals, thereby reconstructing the high-order semantic information of occluded objects and achieving target detection.

[0009] Furthermore, the workflow of the target detection network in step 2 includes the following steps: Step 2.1: The region proposal network uses the remote sensing image object detector to extract unobstructed objectsv Candidate box and occluded objects o Candidate box , and based on RoI Align Network Based and multi-scale features output by remote sensing image object detectors P Extract unobstructed objects v Regional characteristics , and according to Predicting the class labels of unobstructed objects ,in: , ; In the formula, ( x , y ) is the center point coordinate of the candidate box, ( w , h ) are the length and width of the candidate box respectively, is the angle information of the candidate box, For the i An unobstructed object, For the i occluded object, is the number of unobstructed objects, is the number of occluded objects, For the i The category of unobstructed objects; Step 2.2: For each occluded object , based on the nearest neighbor principle n Unobstructed objects Construct a coarse relationship pair, then use the relationship suggestion network to obtain the interactivity score of the interactive existence feature, and then n Select from the coarse relation pairs k The coarse relationship pairs with the highest interaction scores are taken as relationship suggestions; Step 2.3: Use the consistent relationship reasoning network to strengthen the relationship pair region features in the relationship proposal, and then build the consistent predicate features based on the strengthened relationship pair region features and relationship proposals. Use the consistent predicate features to perform spatial and semantic interactive reasoning, and output distribution consistency features. Finally, perform occluded object detection based on the distribution consistency features and output the detection results.

[0010] Furthermore, the specific steps of step 2.2 include: Step 2.2.1: Select n and the occluded object The closest unobstructed object in space Construct coarse relationship pairs, each of which contains relationship pair regional features , spatial characteristics and the category vector ,in, D v is the dimension of the relationship pair feature, N p The number of relationship pairs, D s , D l are the dimensions of spatial features and category vectors respectively; Step 2.2.2: Based on the constructed coarse relation pairs, obtain the spatial encoding of relation pair dependencies ,Will As position code, use and Interaction obtains interactive existence characteristics : , ; In the formula, represents the relational encoder, They represent the unobstructed objects, obstructed objects, and the intersection and union areas of two objects in the rough relationship pair, respectively. i and j Respectively represent the numbers of unobstructed objects and obstructed objects; Concat Indicates a connection operation; Next, Input into a multi-layer perceptron network and get The interactivity score is then used Sigmoid The activation function normalizes the obtained interactivity score to the interval [0, 1]: ; In the formula, represents the normalized interactivity score; Step 2.2.3: Select from the coarse relationship pairs constructed in step 2.2.1 k Interactivity score The highest coarse relationship pairs are used as relationship suggestions.

[0011] Furthermore, the specific steps of step 2.3 include: Step 2.3.1: Construct regional features for all relationship pairs in each relationship proposal The self-attention mechanism with spatial prior knowledge strengthens the regional features of relationship suggestions through this self-attention mechanism: ; ; ; In the formula, N p is the number of relation pairs in the relation proposal; They are query value, key value and value respectively; Representing spatial prior knowledge; D q , D k , D v The number of channels representing query value, key value and value respectively; represents the Kronecker product; Step 2.3.2: Aggregate all attention weighted values v j To calculate the regional features of each enhanced relationship proposal : ; In the formula, For value v The attention weight of each attention mechanism Through the following Softmax The function is normalized to obtain: ; In the formula, is the attention weight, The calculation formula is: ; In the formula, is the learnable embedding matrix, It is a two-layer MLP network used to encode spatial features; Step 2.3.3: Get the position encoding of the relation proposal region by the following formula : ; in, , Unobstructed objects v and occluding objects o The length and width of the candidate box, and Unobstructed objects v and occluding objects o Angle information of Step 2.3.4: Position encoding of relation proposal regions and enhanced regional features of relation proposals based on semantic prior knowledge , construct consistent predicate features : ; in, ε is a small constant; Step 2.3.5: Reconstruct the high-order semantic features of the occluded object by constructing the interaction between the consistent predicate features and the spatial and semantic priors of the relation proposal through the cross-attention mechanism. The cross-attention mechanism is: ; ; ; In the formula, Represents the spatial transformation position encoding between two objects in the relation proposal; Represents the category embedding corresponding to the unoccluded target, through sampling get; Next, the spatial and semantic features of each key are used to interact with the corresponding query value, perform spatial and semantic interaction reasoning, and output the attention weight : ; in, Indicates i The query value in the criss-cross attention mechanism; Indicates j The key value in the cross-attention mechanism; Step 2.3.6: Aggregate all attention weights The weighted values ​​are used to obtain the distribution consistency feature, which is then combined with a two-layer MLP network. Sigmoid The activation function obtains the category of the occluded target, and then combines the center point coordinates, length, width, and angle information of the candidate box of the occluded object to output the detection result of the occluded object.

[0012] A second aspect of the present invention provides a remote sensing image target detection system based on consistency relationship reasoning, the system is implemented using a remote sensing image target detection method based on consistency relationship reasoning, and the system comprises: The data set acquisition module is used to acquire the remote sensing image data set and preprocess the remote sensing images in the data set; Detection network building module, used to build an object detection network based on consistency relationship reasoning; The network training module is used to train the constructed target detection network based on the preprocessed remote sensing images; The accuracy evaluation module is used to evaluate the detection accuracy of the trained target detection network. If the preset accuracy is not reached, the network training module is used to retrain until the preset accuracy is reached, and the target detection network is output; The target detection module is used to realize target detection of remote sensing images based on the target detection network output by the accuracy assessment module.

[0013] Preferably, the detection network construction module includes: A region proposal network building unit is used to build a region proposal network, which uses a trained two-stage remote sensing image object detector as a candidate box extractor to detect unobstructed objects and locate the region of occluded objects; A relationship proposal network building unit, used to build a relationship proposal network, which builds coarse relationship pairs based on the detection results of the region proposal network and selects the coarse relationship pairs that are most likely to interact as relationship suggestions; The consistent relation reasoning network building unit is used to build a consistent relation reasoning network. The consistent relation reasoning network builds distribution consistency features based on the spatial and semantic consistency of object distribution learned by relation proposals, thereby reconstructing high-order semantic information of occluded objects and realizing target detection.

[0014] Therefore, compared with the prior art, the present invention has the following beneficial effects: First, the present invention converts the visual relationship learning task into a scene graph completion task, and proposes a visual relationship learning framework based on consistency reasoning to learn the spatial and semantic consistency of object distribution, thereby giving the target detection algorithm the ability of visual relationship reasoning.

[0015] Second, the visual relationship learning method for consistency reasoning proposed in the present invention performs spatial dependency encoding through a self-attention mechanism to refine relationship features, and selects the relationship pairs with the highest degree of interaction for further reasoning.

[0016] Third, the query key interaction mechanism proposed in the present invention learns the distribution consistency characteristics of objects by integrating spatial and semantic prior knowledge and reconstructs the semantic features of occluded objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is the structural diagram of the proposed remote sensing image target detection network based on consistency relationship reasoning.

[0018] Figure 2 Flowchart for constructing a consistency relation reasoning network.

[0019] Figure 3 Diagram of the mask-reconstruction based relation learning framework.

[0020] Figure 4 Figure 2 shows an example of target detection results using the baseline method and the consistency reasoning-based target detection algorithm on the DOTA dataset and the DIOR-R dataset. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. In the description of the present invention, it should also be noted that, unless otherwise clearly specified and limited, these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art will make various changes or modifications to the present invention, and these equivalent forms also fall within the scope limited by the appended claims of the application.

[0022] Step 1: Obtain a remote sensing image dataset and preprocess the remote sensing images in the dataset for different detection tasks; Step 2: Construct an object detection network based on consistent relation reasoning, which includes a relation proposal network, a region proposal network, and a consistent reasoning network, as shown in the attached figure. Figure 1 As shown; The main workflow of the object detection network is as follows: first, a trained two-stage remote sensing image object detector is used as a region proposal network to extract candidate boxes, detect unobstructed objects and locate the regions of occluded objects; then, the relation proposal network constructs coarse relation pairs based on the detection results of the detector, and selects the coarse relation pairs with the highest probability of interaction as relation proposals, which are used for subsequent reasoning; Finally, the consistent relation reasoning network builds distribution consistency features based on relation proposals by learning the spatial and semantic consistency of object distribution, reconstructs the high-order semantic information of occluded objects, and finally realizes target detection. Step 3: Combine with Figure 3 , train the target detection network based on consistency relationship reasoning constructed in step 2; input the remote sensing image preprocessed in step 1 into the target detection network constructed in step 2 for network training, calculate and record the loss function, and stop training after reaching the preset epoch; Step 4: Evaluate the detection accuracy of the trained target detection network. If the preset detection accuracy is not reached, return to step 3 to retrain the network until the preset detection accuracy is met, and output the target detection network. Step 5: Based on the target detection network output in step 4, perform target detection on the remote sensing image and output the detection results.

[0023] Furthermore, the specific operation steps of step 1 include: Step 1.1: Select remote sensing image datasets DOTA and DIOR-R; DOTA is a challenging remote sensing image directional target detection dataset, containing 2806 satellite optical images with sizes ranging from 800×800 to 4000×4000 pixels, and a total of 188282 annotated target instances; these instances are divided into 15 common categories, including plane (PL), baseball diamond (BD), bridge (BR), ship (SH), harbor (HA), roundabout (RA), storage tank (ST), tennis court (TC), small vehicle (SV), large vehicle (LV), basketball court (BC), ground track field (GTF), soccer ball field (SBF), swimming pool (SP), helicopter (HC); DIOR-R is a comprehensive dataset for large-scale remote sensing image detection, containing 23,463 images and a total of 192,518 instances. The dataset contains 20 common target categories, including airplane (APL), airport (APO), baseball field (BF), basketball court (BC), bridge (BR), chimney (CH), expressway service area (ESA), expressway toll station (ETS), dam (DAM), golf field (GF), ground track field (GTF), harbor (HA), overpass (OP), ship (SH), stadium (STA), storage tank (ST), tennis court (TC), train station (TS), vehicle (VE), windmill (WM); Step 1.2: Preprocess DOTA and DIOR-R for different detection tasks; 1) Remote sensing image target detection task: The rotation target detection task is a conventional target detection task. It uses the conventional settings of the DOTA dataset and the DIOR-R dataset. No special processing of the objects in the image is required. That is, there is no need to preprocess the remote sensing image dataset for this task.

[0024] 2) Remote sensing image occluded target reasoning task: For the remote sensing images in the DOTA and DIOR-R datasets, 25% of the total number of objects in each remote sensing image are selected and the selected targets are segmented into m × m 75% of the image blocks are randomly selected to apply random cloud occlusion to avoid excessive calculation. m The value range of is [8, 16], and the datasets after random cloud occlusion are denoted as Occluded DOTA and Occluded DIOR-R respectively.

[0025] Furthermore, the target detection steps of the target detection network based on consistency relationship reasoning constructed in step 2 include: Step 2.1: Use the two-stage target detector in the prior art as the remote sensing image target detector, and use the trained remote sensing image target detector as the candidate box extractor to extract unobstructed objects. v Candidate box and occluded objects o Candidate box ,in, , , They are the center point coordinates, length, width, and angle information of the candidate box. For the i An unobstructed object, For the i occluded object, N v is the number of unobstructed objects, N o is the number of occluded objects; RoI Align Network Based B v And multi-scale features based on the output of remote sensing image target detector P , extract unobstructed objects v Regional characteristics F v , and according to F v Predicting the class labels of unobstructed objects ,in, For the i The category of unobstructed objects; Step 2.2: Construct rough relationship pairs; For each occluded object, select the nearest neighbor to the occluded object according to the nearest neighbor principle. n Unoccluded objects are used to construct coarse relationship pairs, and each coarse relationship pair contains the relationship pair region features. , spatial characteristics and the category vector ,in, is the dimension of the relationship pair feature, is the number of relationship pairs, are the dimensions of spatial features and category vectors respectively; the relationship between the regional features is achieved through RoI Align The visual features of the region where the coarse relationship pairs are extracted by the network; the spatial features are the position encoding of the geometric attributes in the coarse relationship pairs; the category vector is the category label of the unobstructed object High-dimensional embedding of Step 2.3: Based on the constructed coarse relationship pairs, obtain the interactive existence features and calculate the interactive scores; First, based on the constructed coarse relationship pairs, the spatial encoding of the relationship pair dependencies is constructed : ; in, Respectively represent the unobstructed objects and the obstructed objects in the coarse relation pair, I , U Respectively represent the intersection and union areas of unobstructed objects and obstructed objects; Concat Indicates a connection operation; Secondly, the relationship between regional characteristics The key, query, and value used for the self-attention mechanism in the relational encoder, As position code, use Interaction with position encoding to obtain interactive presence characteristics : (1); in, represents the relational encoder, They represent the unobstructed object, the obstructed object, and the intersection and union regions of two objects in the coarse relation pair respectively; Finally, Input into a two-layer MLP network to obtain the interactive existence feature The interactivity score is then Sigmoid The activation function normalizes the interactivity score to the interval [0, 1]. This process can be expressed as: (2); in, represents the normalized interactivity score; Step 2.4: Select from the coarse relationship pairs constructed in step 2.2 k Interactivity score The highest coarse relation pair is used as a relation suggestion for subsequent relation reasoning; Step 2.5: Refer to the attached Figure 2 , build a consistent relationship reasoning network, and use the network to build each relationship suggestion k Regional characteristics of relationship pairs Prior knowledge of space The self-attention mechanism strengthens the relationship between regional features , the self-attention mechanism is: (3); In the formula, N p is the number of relation pairs in the relation proposal; They are query value, key value and value respectively; Representing spatial prior knowledge; D q , D k , D v The number of channels representing query value, key value and value respectively; represents the Kronecker product; By aggregating the weighted values ​​of all attention v j To calculate the regional features of each enhanced relationship proposal : (4); in, W v For value v The attention weight of each attention mechanism α ij Through the following Softmax The function is normalized to obtain: (5); in, is the attention weight, It can be obtained by the following formula: (6); In the formula, is a learnable embedding matrix, It is a two-layer MLP network used to encode spatial features; Step 2.6: Construct semantic prior knowledge and position encoding of relation proposal regions; Semantic prior knowledge is the category label for all objects of interest High-dimensional vector encoding, is the position encoding of the relation proposal region; Based on the position information of unobstructed objects and occluded objects, the position encoding of the relationship proposal area is constructed ; (7); in, For unobstructed objects v The candidate box, For the occluded object o The candidate box, , , and is the position information of the unobstructed object and the obstructed object, and For unobstructed objects v The length and width of the candidate box, For the occluded object o The length and width of the candidate box, and They are the angle information of unobstructed objects and obstructed objects respectively; Step 2.7: Based on semantic prior knowledge , Position encoding of relation proposal regions and regional characteristics suggested by the strengthened relationship , construct consistent predicate features : (8); in, ε For adding A small constant on the vector to avoid taking the logarithm to zero; Step 2.8: Predicate features including visual, spatial and semantic features obtained in step 2.7 , through the cross-attention mechanism expressed by formula (9), the spatial and semantic prior interactions between the consistent predicate features and the relation suggestions (the filtered coarse relation pairs) are constructed to reconstruct the high-order semantic features of the occluded objects; the cross-attention mechanism is: (9); In the formula, Represents the spatial transformation position encoding between two targets in the relation proposal region, which is used to interact with the pairwise position encoding in the predicate feature; It is the category embedding corresponding to the unoccluded target, through sampling get; The spatial and semantic features of each key are used to interact with the corresponding query value to perform spatial and semantic interaction reasoning to achieve reasoning and output the attention weights : (10); in, Indicates i The query value in the criss-cross attention mechanism; Indicates j The key value in the cross-attention mechanism; Step 2.9: Aggregate all attention weights The weighted values ​​are used to obtain the distribution consistency feature, which is then passed through a two-layer MLP network and combined Sigmoid The activation function obtains the category of the occluded target, and combines the position information of the occluded object obtained in step 2.1 (i.e. the center point coordinates, length, width, and angle information of the candidate box) to output the detection result of the occluded object; Furthermore, in step 4, for the target detection network based on consistency relationship reasoning after parameter adjustment, a remote sensing image target detection task is constructed through the test sets of the DOTA dataset and the DIOR-R dataset, and the detection accuracy is evaluated; a remote sensing image occluded target reasoning task is constructed through the test sets of the Occluded DOTA dataset and the Occluded DIOR-R dataset, and the reasoning accuracy under the conditions of limited target visual features is evaluated.

[0026] Embodiment: In order to further illustrate the effect of the remote sensing image target detection method (Consistency Reasoning Transformer, hereinafter referred to as CRTr method) proposed in the present invention, a remote sensing image target detection task is constructed on the DOTA dataset (Table 1, Table 3) and the DIOR-R dataset (Table 2) to evaluate the detection accuracy; a remote sensing image occluded target reasoning task is constructed on the Occluded DOTA dataset (Table 4) and the Occluded DIOR-R dataset (Table 5) to evaluate the reasoning accuracy under the condition of limited target visual features; 1. Remote sensing image target detection task: The baseline models used in the experiment are Faster RCNN-O and Gliding Vertex algorithms.

[0027] 1) DOTA: Table 1 shows the performance comparison of the proposed CRTr method with existing rotated object detection algorithms on the DOTA dataset. After training for 12 epochs, the CRTr method with ResNet50 as the feature extraction network achieved 76.81% mAP, surpassing many current advanced methods such as S 2 A-Net, Gliding Vertex, KFIoU and OrientedRCNN methods. In addition, without using additional training techniques, the CRTr method with ResNet101 as the feature extraction network achieved 77.44% mAP. The performance of this method exceeds all the advanced rotation object detection methods listed and reaches the state-of-the-art level. Figure 4 The target detection results of the baseline method (i.e., Faster RCNN-O) and the target detection method based on consistency reasoning proposed by the present invention on the DOTA dataset and the DIOR-R dataset are shown in the figure. The white dotted box is the target detection result before the implementation of consistency reasoning, and the white solid box is the target detection result after the implementation of consistency reasoning. It can be seen from the results in the figure that the present method improves the target detection performance through consistency relationship reasoning, and has a higher detection accuracy rate than the baseline method. Therefore, the quantitative and qualitative results prove the superiority of the proposed method.

[0028] Table 1 Comparison of detection performance with the most advanced rotation target detection methods on the DOTA dataset: ; 2) DIOR-R: The performance comparison with existing methods on the DIOR-R dataset is shown in Table 2. On the DIOR-R dataset, the ResNet50-based CRTr method achieved 64.02% mAP, which is better than other two-stage and one-stage detection methods. Experimental results show that the proposed method improves 4.48% mAP over the baseline method Faster RCNN-O, and outperforms RetinaNet-O method, FCOS-O method, and Gliding Vertex method by 6.47% mAP, 4.22% mAP, and 3.98% mAP, respectively. It is worth noting that the proposed method achieves the best performance in 9 target categories. Especially in those more challenging categories, such as airports, dams, and bridges, there is a huge improvement compared with existing methods. The detection results on the DIOR-R dataset further demonstrate the generalization and stability of the proposed CRTr method.

[0029] Table 2 Comparison of detection performance with the most advanced rotation object detection methods on the DIOR-R dataset: ; 3) Comparison with other inference-based detectors: Table 3 compares the CRTr method with existing reasoning-based detectors to further verify the advancedness of the proposed method. CRTr based on ResNet101 achieves 75.90% mAP, which has better detection performance than other detectors. DRGN endows the object detection model with the ability to reason by propagating visual and spatial embeddings. Compared with DRGN, the CRTr method achieves a 5.20% performance improvement (75.90% vs. 70.70%). Even compared with the state-of-the-art PCI method and CTM method, the proposed method improves 1.20% and 1.69% mAP based on ResNet101, respectively. In addition, the classic reasoning-based method Reasoning RCNN is reproduced on MMRotate. Under the same setting, the CRTr method improves the detection performance by 0.88% mAP over Reasoning RCN (75.90% vs. 74.82%). In the conventional rotation object detection task, the proposed method uses easy-to-detect objects to infer difficult-to-detect objects, and has better detection accuracy than other detectors for some easily confused categories, such as small vehicles, large vehicles, basketball courts, and helicopters. In addition, Figure 4 The visualization results of target detection before and after consistency reasoning are shown. As shown in the figure, when faced with challenges such as blurred image features, shadow occlusion, and small targets, the proposed method can achieve more consistent detection results through visual relationship reasoning. The experimental results prove the powerful reasoning ability of the CRTr method.

[0030] Table 3 Comparison of detection performance on the DOTA dataset with other reasoning-based methods: ; 2. Remote sensing image occluded target reasoning task: The baseline models used in the experiment are Faster RCNN-O and Gliding Vertex algorithms.

[0031] 1) Occluded DOTA: Table 4 shows the comparison of the detection performance of the CRTr method and the current state-of-the-art methods on Occluded DOTA. The data in brackets show that the performance of existing methods degrades when dealing with occluded objects.

[0032] Table 4 Comparison of detection performance with advanced rotation target detection methods on the Occluded DOTA dataset: ; The results show that when dealing with occlusion in the experimental setting, all methods experience a performance drop of about 9%, such as Faster RCNN-O performance drops by 9.10%, RoI Transformer performance drops by 10.59%, and R 3 Det performance decreased by 8.72%, S 2 The performance of A-Net decreased by 8.93%, and that of CFL decreased by 9.86%. Even the most advanced two-stage method OrientedR-CNN only achieved 73.25% mAP, a 9.77% performance decrease. However, the baseline method using CRTr (with ResNet50 as the feature extraction network) achieved a very large gain (79.76% vs 73.25%). When ResNet101 was used as the feature extraction network, the proposed algorithm achieved 80.52% detection performance, which was 6.61% better than the ReDet algorithm using a powerful feature extraction network (ReR50-ReFPN). Experiments show that the proposed method can detect occluded objects through reasoning experiments under the condition of limited visual features, and the proposed method can be integrated into existing object detection algorithms as an easy-to-use component.

[0033] 2) Occluded DIOR-R: The Occluded DIOR-R dataset contains rich categories and semantic relations. As shown in Table 5, on the Occluded DIOR-R dataset, the proposed method achieves 61.54% and 63.14% mAP when using ResNet50 and ResNet101, respectively, which is better than other two-stage and one-stage methods. The superior results on the Occluded DIOR-R dataset demonstrate the reliability and superiority of the proposed method.

[0034] Table 5 Comparison of detection performance with advanced rotation target detection methods on the Occluded DIOR-R dataset: ; Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A remote sensing image target detection method based on consistency relationship reasoning, characterized in that: The following steps are involved: Step 1: Obtain a remote sensing image dataset and preprocess the remote sensing images in the dataset; Step 2: Build an object detection network based on consistency relationship reasoning; Step 3: Based on the preprocessed remote sensing image, train the target detection network constructed in step 2; Step 4: Evaluate the detection accuracy of the trained target detection network. If the preset accuracy is not reached, return to step 3 and retrain until the preset accuracy is reached, and output the target detection network. Step 5: Implement target detection of remote sensing images based on the target detection network output from step 4.

2. The remote sensing image target detection method based on consistency relationship reasoning as claimed in claim 1, characterized in that: The specific process of step 1 includes: Step 1.1: Select remote sensing image datasets DOTA and DIOR-R; Step 1.2: Preprocess the selected data sets for different detection tasks; Among them, different detection tasks include remote sensing image target detection tasks and remote sensing image occluded target reasoning tasks; For the remote sensing image object detection task, no preprocessing is performed on the DOTA and DIOR-R datasets; For the remote sensing image occluded target reasoning task, 25% of the total number of objects in each image in the DOTA and DIOR-R datasets are selected and the selected targets are divided into m × m , randomly select 75% of the image blocks and apply random cloud occlusion, and record the data sets after applying random cloud occlusion as Occluded DOTA and Occluded DIOR-R respectively.

3. The method for remote sensing image target detection based on consistency relationship reasoning as claimed in claim 2, characterized in that: The target detection network includes a region proposal network, a relationship proposal network and a consistency reasoning network, wherein: Region Proposal Network, which uses a trained two-stage remote sensing image object detector as a candidate box extractor to detect unoccluded objects and locate the regions of occluded objects; The relation proposal network is used to construct coarse relation pairs based on the detection results of the region proposal network, and select the coarse relation pairs with the greatest possibility of interaction as relation suggestions; The consistent relation reasoning network is used to build distribution consistency features based on the spatial and semantic consistency of object distribution learned by relation proposals, thereby reconstructing the high-order semantic information of occluded objects and achieving target detection.

4. The method for remote sensing image target detection based on consistency relationship reasoning as claimed in claim 3, characterized in that: The workflow of the target detection network described in step 2 includes the following steps: Step 2.1: The region proposal network uses the remote sensing image object detector to extract unobstructed objects v Candidate box and occluded objects o Candidate box , and based on RoI Align Network Based and multi-scale features output by remote sensing image object detectors P Extract unobstructed objects v Regional characteristics , and according to Predicting the class labels of unobstructed objects ,in: ， ； In the formula, ( x , y ) is the center point coordinate of the candidate box, ( w , h ) are the length and width of the candidate box respectively, is the angle information of the candidate box, For the i An unobstructed object, For the i occluded object, is the number of unobstructed objects, is the number of occluded objects, For the i The category of unobstructed objects; Step 2.2: For each occluded object , based on the nearest neighbor principle n Unobstructed objects Construct a coarse relationship pair, then use the relationship suggestion network to obtain the interactivity score of the interactive existence feature, and then n Select from the coarse relation pairs k The coarse relationship pairs with the highest interaction scores are taken as relationship suggestions; Step 2.3: Use the consistent relationship reasoning network to strengthen the relationship pair region features in the relationship proposal, and then build the consistent predicate features based on the strengthened relationship pair region features and relationship proposals. Use the consistent predicate features to perform spatial and semantic interactive reasoning, and output distribution consistency features. Finally, perform occluded object detection based on the distribution consistency features and output the detection results.

5. The method for remote sensing image target detection based on consistency relationship reasoning as claimed in claim 4, characterized in that: The specific steps of step 2.2 include: Step 2.2.1: Select n and the occluded object The closest unobstructed object in space Construct coarse relationship pairs, each of which contains relationship pair regional features , spatial characteristics and the category vector ,in, D v is the dimension of the relationship pair feature, N p The number of relationship pairs, D s , D l are the dimensions of spatial features and category vectors respectively; Step 2.2.2: Based on the constructed coarse relation pairs, obtain the spatial encoding of relation pair dependencies ,Will As position code, use and Interaction obtains interactive existence characteristics : ， ； In the formula, represents the relational encoder, They represent the unobstructed objects, obstructed objects, and the intersection and union areas of two objects in the rough relationship pair, respectively. i and j Respectively represent the numbers of unobstructed objects and obstructed objects; Concat Indicates a connection operation; Next, Input into a multi-layer perceptron network and get The interactivity score is then used Sigmoid The activation function normalizes the obtained interactivity score to the interval [0, 1]: ； In the formula, represents the normalized interactivity score; Step 2.2.3: Select from the coarse relationship pairs constructed in step 2.2.1 k Interactivity score The highest coarse relationship pairs are used as relationship suggestions.

6. The method for remote sensing image target detection based on consistency relationship reasoning as claimed in claim 4, characterized in that: The specific steps of step 2.3 include: Step 2.3.1: Construct regional features for all relationship pairs in each relationship proposal The self-attention mechanism with spatial prior knowledge strengthens the regional features of relationship suggestions through this self-attention mechanism: ； ； ； In the formula, N p is the number of relation pairs in the relation proposal; They are query value, key value and value respectively; Representing spatial prior knowledge; D q , D k , D v The number of channels representing query value, key value and value respectively; represents the Kronecker product; Step 2.3.2: Aggregate all attention weighted values v j To calculate the regional features of each enhanced relationship proposal : ; In the formula, For value v The attention weight of each attention mechanism Through the following Softmax The function is normalized to obtain: ; In the formula, is the attention weight, The calculation formula is: ; In the formula, is the learnable embedding matrix, It is a two-layer MLP network used to encode spatial features; Step 2.3.3: Get the position encoding of the relation proposal region by the following formula : ; in, , Unobstructed objects v and occluding objects o The length and width of the candidate box, and Unobstructed objects v and occluding objects o Angle information of Step 2.3.4: Position encoding of relation proposal regions and enhanced regional features of relation proposals based on semantic prior knowledge , construct consistent predicate features : ； in, ε is a small constant; Step 2.3.5: Reconstruct the high-order semantic features of the occluded object by constructing the interaction between the consistent predicate features and the spatial and semantic priors of the relation proposal through the cross-attention mechanism. The cross-attention mechanism is: ； ； ； In the formula, Represents the spatial transformation position encoding between two objects in the relation proposal; Represents the category embedding corresponding to the unoccluded target, through sampling get; Next, the spatial and semantic features of each key are used to interact with the corresponding query value, perform spatial and semantic interaction reasoning, and output the attention weight : ； in, Indicates i The query value in the criss-cross attention mechanism; Indicates j The key value in the cross-attention mechanism; Step 2.3.6: Aggregate all attention weights The weighted values ​​are used to obtain the distribution consistency feature, which is then combined with a two-layer MLP network. Sigmoid The activation function obtains the category of the occluded target, and then combines the center point coordinates, length, width, and angle information of the candidate box of the occluded object to output the detection result of the occluded object.

7. A remote sensing image target detection system based on consistency relationship reasoning, characterized in that: The method according to any one of claims 1 to 6 is implemented, the system comprising: The data set acquisition module is used to acquire the remote sensing image data set and preprocess the remote sensing images in the data set; Detection network building module, used to build an object detection network based on consistency relationship reasoning; The network training module is used to train the constructed target detection network based on the preprocessed remote sensing images; The accuracy evaluation module is used to evaluate the detection accuracy of the trained target detection network. If the preset accuracy is not reached, the network training module is used to retrain until the preset accuracy is reached, and the target detection network is output; The target detection module is used to realize target detection of remote sensing images based on the target detection network output by the accuracy assessment module.

8. The remote sensing image target detection system based on consistency relationship reasoning as claimed in claim 7, characterized in that: The detection network building module includes: A region proposal network building unit is used to build a region proposal network, which uses a trained two-stage remote sensing image object detector as a candidate box extractor to detect unobstructed objects and locate the region of occluded objects; A relationship proposal network building unit, used to build a relationship proposal network, which builds coarse relationship pairs based on the detection results of the region proposal network and selects the coarse relationship pairs that are most likely to interact as relationship suggestions; The consistent relation reasoning network building unit is used to build a consistent relation reasoning network, which builds distribution consistency features based on the spatial and semantic consistency of object distribution learned by relation proposals, thereby reconstructing high-order semantic information of occluded objects and realizing target detection.

Images