Remote sensing image target detection method based on fusion network

Abstract

The application provides a remote sensing image target detection method based on a fusion network, which comprises the following steps: constructing a remote sensing knowledge graph, taking the remote sensing knowledge graph as a graph structure to establish a graph neural network, and training the graph neural network; constructing a Faster-RCNN model introducing the graph neural network and training the Faster-RCNN model, and fine-tuning Fast-RCNN model parameters; acquiring a remote sensing image, inputting the remote sensing image into the Faster-RCNN model, extracting features by using the graph neural network, obtaining a first feature map, and performing position prediction and feature recommendation on the first feature map to obtain position information and recommendation information; integrating the first feature map, the position information and the recommendation information to obtain a second feature map; and performing target detection classification on the second feature map to obtain a target detection result. By introducing the graph neural network based on the knowledge graph into the Faster-RCNN model as a feature extraction and recommendation network, the application reduces the generation of unnecessary candidate boxes, improves the calculation speed, and obtains an accurate target detection result.

Description

Technical Field

[0001] This invention relates to remote sensing image processing technology, specifically to a remote sensing image target detection method based on a fusion network. Background Technology

[0002] Remote sensing technology is an important tool for humans to conduct scientific investigations of their living environment. Remote sensing satellites, equipped with multi-source sensors, can actively or passively acquire spatial and physical information about various targets in the Earth's sphere, such as vegetation, lakes, oceans, and the atmosphere, and present this information in the form of two-dimensional remote sensing images. The ground feature information contained in remote sensing images can be used in fields such as military reconnaissance, land resource surveys, urban development spatial monitoring, and disaster assessment and early warning. These fields are all closely related to national defense and security and national economic development.

[0003] Object detection is one of the fundamental problems in computer vision recognition tasks, with wide applications in various fields. Object detection in remote sensing images has broad application prospects in military applications, urban planning, and environmental management. Unlike object detection in natural images, targets in remote sensing images are much smaller, and their size and orientation vary (e.g., playgrounds, vehicles, bridges, etc.). Furthermore, the visual appearance of target instances varies due to occlusion, shadows, lighting, resolution, and viewpoint variations. Therefore, object detection in remote sensing images is significantly more difficult than object detection in natural images.

[0004] In recent years, some studies have introduced deep convolutional neural networks (CNNs) into object detection, enabling the automatic learning of robust and expressive feature representations from data. This approach has achieved significant progress in both speed and accuracy. Among these, candidate region extraction-based and regression-based object detection algorithms are the most classic deep convolutional neural network-based object detection algorithms. Candidate region extraction-based algorithms first extract candidate regions from a given image, then classify and regress each extracted region, offering advantages in object detection accuracy. Regression-based object detection algorithms, on the other hand, utilize a single, holistic convolutional neural network, reconstructing the object detection problem into a regression problem to directly predict the object's category and location, thus offering advantages in object detection speed.

[0005] Although current object detection algorithms have achieved good results in object detection in natural images, there is still room for improvement in object detection in remote sensing images. The computational speed and resource waste are still problems that urgently need to be solved. Summary of the Invention

[0006] To address the aforementioned technical issues, this invention introduces a graph neural network built on a knowledge graph into the Faster-RCNN model as a feature extraction and recommendation network, thereby reducing the generation of unnecessary candidate boxes, improving computational speed, and resulting in more accurate target detection results.

[0007] This invention provides a remote sensing image target detection method based on a fusion network, the method comprising:

[0008] S1 Constructs a remote sensing knowledge graph, uses the remote sensing knowledge graph as a graph structure to build a graph neural network, and trains the graph neural network.

[0009] S2 constructs and trains a Faster-RCNN model that incorporates a graph neural network, and fine-tunes the Fast-RCNN model parameters;

[0010] S3 acquires remote sensing images, inputs them into the Faster-RCNN model, uses graph neural networks to extract features, obtains the first feature map, and performs location prediction and feature recommendation on the first feature map to obtain location information and recommendation information;

[0011] S4 integrates the first feature map, location information, and recommendation information to obtain the second feature map;

[0012] S5 performs object detection and classification on the second feature map to obtain the object detection result.

[0013] In a specific embodiment of the present invention, step S1 includes:

[0014] Acquire remote sensing knowledge, including textual and image knowledge, extract knowledge from textual and image knowledge respectively, identify entities and establish relationships between entities;

[0015] By structuring entities and relations, multiple triples are obtained;

[0016] Connect the triples to form a remote sensing knowledge graph;

[0017] A graph neural network is built using remote sensing knowledge graphs as the graph structure, where the graph structure includes nodes and edges, with entities being nodes and relationships being edges;

[0018] Obtain the first training sample and train the graph neural network to complete the remote sensing knowledge graph;

[0019] The graph neural network is optimized using the completed remote sensing knowledge graph and the first training samples.

[0020] In a specific embodiment of the present invention, step S2 includes:

[0021] A Fast-RCNN model was built, and a graph neural network was introduced into the Fast-RCNN model as a feature extraction unit;

[0022] Obtain the second training sample, input the first and second training samples into the Fast-RCNN model respectively, perform cross-comparison training, and adjust the Fast-RCNN model parameters.

[0023] In a specific embodiment of the present invention, step S3, "performing location prediction and feature recommendation on the first feature map to obtain location information and recommendation information," includes:

[0024] The first feature map is matched with the remote sensing knowledge graph, and the location of the first feature map is predicted based on the node information of the remote sensing knowledge graph to obtain the location information.

[0025] Feature recommendation is performed on the first feature map based on the edge information of the remote sensing knowledge graph to obtain recommendation information.

[0026] In a specific embodiment of the present invention, step S4 includes:

[0027] The first feature map, location information, and recommendation information are input into the pooling layer of the Fast-RCNN model;

[0028] Based on the recommendation information, the first feature map is filtered to obtain candidate feature maps;

[0029] Candidate boxes are generated in the candidate feature map based on the location information, resulting in a second feature map.

[0030] In a specific embodiment of the present invention, step S5 includes:

[0031] The category of each candidate box is calculated based on the fully connected layers and logistic regression algorithm in the Fast-RCNN model, and the category detection results are obtained.

[0032] The position offset of each candidate box is obtained by using the bounding box regression algorithm, thus obtaining the precise position of the candidate box;

[0033] The precise location of the candidate bounding box and the category detection result are used as the final object detection result.

[0034] The beneficial effects of this invention are as follows: This invention provides a remote sensing image target detection method based on a fusion network. The method includes: constructing a remote sensing knowledge graph; using the remote sensing knowledge graph as a graph structure to build a graph neural network and training the graph neural network; constructing and training a Faster-RCNN model incorporating the graph neural network, and fine-tuning the Fast-RCNN model parameters; acquiring a remote sensing image, inputting it into the Faster-RCNN model, using the graph neural network for feature extraction to obtain a first feature map, and performing location prediction and feature recommendation on the first feature map to obtain location information and recommendation information; integrating the first feature map, location information, and recommendation information to obtain a second feature map; and performing target detection and classification on the second feature map to obtain the target detection result. This invention, by introducing a graph neural network built based on a knowledge graph into the Faster-RCNN model as a feature extraction and recommendation network, reduces the generation of unnecessary candidate boxes, improves computation speed, and obtains accurate target detection results. Furthermore, after detecting remote sensing images, the knowledge graph can automatically learn new knowledge, which is then fed back to the graph neural network to improve the network structure, automatically and continuously optimizing the model. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart of the remote sensing image target detection method based on a fusion network according to an embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0038] Please see Figure 1 This invention provides a remote sensing image target detection method based on a fusion network, the method comprising:

[0039] S1 Constructs a remote sensing knowledge graph, uses the remote sensing knowledge graph as a graph structure to build a graph neural network, and trains the graph neural network.

[0040] S2 constructs and trains a Faster-RCNN model that incorporates a graph neural network, and fine-tunes the Fast-RCNN model parameters;

[0041] S3 acquires remote sensing images, inputs them into the Faster-RCNN model, uses graph neural networks to extract features, obtains the first feature map, and performs location prediction and feature recommendation on the first feature map to obtain location information and recommendation information;

[0042] S4 integrates the first feature map, location information, and recommendation information to obtain the second feature map;

[0043] S5 performs object detection and classification on the second feature map to obtain the object detection result.

[0044] The process of constructing a remote sensing knowledge graph is as follows:

[0045] (1) Remote sensing ontology design

[0046] An ontology abstracts real-world objects into concepts and provides explicit and standardized descriptions of these concepts through attributes and attribute constraints. An ontology consists of classes (also called concepts), subclasses, attributes, and attribute constraints. A class is an abstraction of a concept within a specific domain, while attributes describe the class. Ontologies can be extended and constrained to construct remote sensing knowledge graphs. Ontologies structure and organize knowledge and data, assigning semantic relationships to the data. It is important to note that concepts and entities are easily confused; under the same ontology, a remote sensing entity may also be an abstract concept.

[0047] Ontologies structure and organize knowledge and data, assigning semantic relationships to the data. It's important to note that concepts and entities are easily confused; under the same ontology, a remote sensing entity may also be an abstract concept. Furthermore, remote sensing satellites conduct periodic observations of the Earth's surface, obtaining observational results at different temporal levels; therefore, temporal relationships are needed to describe geographic entities. Temporal relationships at the point-in-time level include "earlier than," "later than," and "simultaneous," while at the time-time level, they include "period" and "overlapping" relationships.

[0048] The construction of a remote sensing ontology follows a coarse-to-fine process. After determining the domain and scope of the remote sensing knowledge graph, key concepts in the graph can be obtained. Then, concepts are expanded based on these, gradually completing the ontology concepts. Before constructing the ontology, it is necessary to analyze the needs to determine whether there are relevant ontologies or classification standards. If so, ontology mapping, expansion, or direct use can be performed based on them.

[0049] (2) Remote sensing knowledge extraction

[0050] Acquire remote sensing knowledge, including textual and image knowledge.

[0051] Textual knowledge describes the information in remote sensing images in words, extracts structured information from the text, and establishes a mapping relationship between entities in the remote sensing knowledge graph and classes in the ontology. Text knowledge extraction employs lexical mining techniques to identify important phrases and words in the remote sensing domain. Lexical mining techniques include domain phrase mining methods based on statistical indicators, which can be unsupervised or supervised learning. This embodiment uses an unsupervised learning-based domain phrase mining method, with the following steps: a) Candidate phrase generation: Candidate phrases refer to high-frequency N-character or word sequences (N-Grames). First, a minimum threshold for the occurrence of N-Grames is set, which is proportional to the size of the text knowledge. N-Grames with occurrences greater than or equal to the threshold are obtained as candidate phrases through frequent pattern mining; b) Statistical feature calculation: Statistical indicator characteristics of candidate phrases are calculated based on text knowledge, such as TF-IDF, PMI, left neighbor entropy, and right neighbor entropy; c) Quality scoring: The values ​​of the statistical indicators are fused to obtain the final score of the candidate phrase, which is used to evaluate the quality of the phrase; d) Sorting output: All candidate phrases are sorted from high to low scores, and the top K phrases or phrases selected based on the threshold are taken as output. After phrase mining, synonyms and abbreviations can also be mined to obtain remote sensing vocabulary.

[0052] Then, entity extraction and relation extraction are performed on the remote sensing vocabulary to identify entities and establish specific relationships between entities, including:

[0053] Entity extraction

[0054] Entity extraction, also known as entity recognition, aims to extract entity information elements from textual knowledge, including building names, feature names, street names, geographical locations, time, and date. Entity extraction requires first identifying and locating entities from the textual knowledge, and then classifying the identified entities into predefined categories. Entity extraction methods can be rule-based, statistical model-based, or deep learning-based. This embodiment selects a deep learning-based method for entity extraction.

[0055] Compared to traditional statistical models, deep learning-based methods directly use word vectors from the text as input, achieving end-to-end named entity recognition through neural networks, eliminating reliance on manually defined features. The neural networks used for named entity recognition can be convolutional neural networks, recurrent neural networks, or neural networks incorporating attention mechanisms. Different neural network structures act as encoders in the named entity recognition process, obtaining a new vector representation for each word based on the initial input and contextual information, and finally outputting the annotation results for each word through a CRF model.

[0056] Relation Extraction

[0057] Relation extraction refers to the process of extracting semantic relationships between two or more entities from textual knowledge. Relation extraction is closely related to entity extraction; generally, after identifying entities in the text, the possible relationships between them are extracted. Relation extraction methods can be template-based, supervised learning-based, or weakly supervised learning-based. This embodiment uses a weakly supervised learning-based relation extraction method. First, a small number of instances are used as an initial seed set. Then, a template for relation extraction is learned on the seed set. More instances are then extracted using the template and added to the seed set. Through continuous iteration, a large number of relation instances are extracted from the textual knowledge.

[0058] Image knowledge extraction can employ either a top-down or bottom-up approach. This embodiment adopts a top-down construction method, using manual annotation to label image entities and establish relationships between entities within the image.

[0059] (3) Establishment of triplet

[0060] The extracted entities and relations are structured and linked pairwise to obtain multiple triples in the form of: Entity 1 - Relation - Entity 2. After connecting all the triples based on relations, a structural graph is formed, namely the remote sensing knowledge graph.

[0061] A graph neural network (GNN) is built using the remote sensing knowledge graph as the graph structure. The GNN can be a graph convolutional network, graph attention network, graph autoencoder, graph generative network, etc. The graph structure includes nodes and edges, with entities as nodes and relationships as edges. A first training sample is obtained to train the GNN. The graph structure of the GNN is used to identify isomorphic subgraphs and predict missing relationships to complete the remote sensing knowledge graph. The completed remote sensing knowledge graph and the first training sample are then used to optimize the GNN. Specifically, the first training sample is input into the GNN, and knowledge is extracted from the first training sample using the remote sensing knowledge graph to obtain the entities of the first training sample. For each entity, a vector representation and a weight matrix related to the relationship type are calculated to output information. The output information is aggregated to generate a new entity representation, and the relationships between entities are reconstructed based on the entity vector representation, i.e., the entity labels are predicted. The predicted entity labels are compared with the labels of corresponding entities in the remote sensing knowledge graph to adjust the GNN parameters.

[0062] A Fast-RCNN model is built, and a graph neural network is introduced into the Fast-RCNN model as a feature extraction unit; a second training sample is obtained, and the first and second training samples are respectively input into the Fast-RCNN model for cross-comparison training, and the parameters of the Fast-RCNN model are adjusted.

[0063] Remote sensing images are acquired and input into the Fast-RCNN model. Feature extraction is performed on the remote sensing images through a graph neural network to obtain a first feature map. Then, the first feature map is matched with the remote sensing knowledge graph in the graph neural network. The location of the first feature map is predicted based on the nodes with location information in the remote sensing knowledge graph. Each corresponding entity has a specific location coordinate value, thus obtaining the location information of the first feature map.

[0064] Feature recommendation is performed on the first feature map based on the edge information of the remote sensing knowledge graph to obtain recommendation information.

[0065] The first feature map, location information, and recommendation information are input into the pooling layer of the Fast-RCNN model; the first feature map is filtered based on the recommendation information to obtain candidate feature maps; candidate boxes are generated in the candidate feature maps based on the location information. The candidate boxes are rectangular boxes centered on entities with location coordinates, thus obtaining the second feature map.

[0066] The class of each candidate box is calculated using the fully connected layers and logistic regression algorithm in the Fast-RCNN model to obtain the class detection result; the position offset of each candidate box is obtained using the bounding box regression algorithm to obtain the precise position of the candidate box; the precise position of the candidate box and the class detection result are used as the final object detection result.

[0067] The beneficial effects of this invention are as follows: This invention provides a remote sensing image target detection method based on a fusion network. The method includes: constructing a remote sensing knowledge graph; using the remote sensing knowledge graph as a graph structure to build a graph neural network and training the graph neural network; constructing and training a Faster-RCNN model incorporating the graph neural network, and fine-tuning the Fast-RCNN model parameters; acquiring a remote sensing image, inputting it into the Faster-RCNN model, using the graph neural network for feature extraction to obtain a first feature map, and performing location prediction and feature recommendation on the first feature map to obtain location information and recommendation information; integrating the first feature map, location information, and recommendation information to obtain a second feature map; and performing target detection and classification on the second feature map to obtain the target detection result. This invention, by introducing a graph neural network built based on a knowledge graph into the Faster-RCNN model as a feature extraction and recommendation network, reduces the generation of unnecessary candidate boxes, improves computation speed, and obtains accurate target detection results. Furthermore, after detecting remote sensing images, the knowledge graph can automatically learn new knowledge, which is then fed back to the graph neural network to improve the network structure, automatically and continuously optimizing the model.

Claims

1. A remote sensing image target detection method based on a fusion network, characterized in that, The method includes: S1 Constructs a remote sensing knowledge graph, uses the remote sensing knowledge graph as a graph structure to build a graph neural network, and trains the graph neural network. S2 constructs and trains a Faster-RCNN model that incorporates a graph neural network, and fine-tunes the Fast-RCNN model parameters; S3 acquires remote sensing images, inputs them into the Faster-RCNN model, uses graph neural networks to extract features, obtains the first feature map, and performs location prediction and feature recommendation on the first feature map to obtain location information and recommendation information; S4 integrates the first feature map, location information, and recommendation information to obtain the second feature map; S5 performs target detection and classification on the second feature map to obtain the target detection result; Step S1 includes: Acquire remote sensing knowledge, including textual and image knowledge, extract knowledge from textual and image knowledge respectively, identify entities and establish relationships between entities; By structuring entities and relations, multiple triples are obtained; Connect the triples to form a remote sensing knowledge graph; A graph neural network is built using remote sensing knowledge graphs as the graph structure, where the graph structure includes nodes and edges, with entities being nodes and relationships being edges; Obtain the first training sample and train the graph neural network to complete the remote sensing knowledge graph; The graph neural network is optimized using the completed remote sensing knowledge graph and the first training samples; Step S2 includes: A Fast-RCNN model was built, and a graph neural network was introduced into the Fast-RCNN model as a feature extraction unit; Obtain the second training sample, input the first training sample and the second training sample into the Fast-RCNN model respectively, perform cross-comparison training, and adjust the Fast-RCNN model parameters; In step S3, position prediction and feature recommendation are performed on the first feature map to obtain position information and recommendation information, including: The first feature map is matched with the remote sensing knowledge graph, and the location of the first feature map is predicted based on the node information of the remote sensing knowledge graph to obtain the location information. Feature recommendation is performed on the first feature map based on the edge information of the remote sensing knowledge graph to obtain recommendation information; Step S4 includes: The first feature map, location information, and recommendation information are input into the pooling layer of the Fast-RCNN model; Based on the recommendation information, the first feature map is filtered to obtain candidate feature maps; Candidate boxes are generated in the candidate feature map based on the location information, resulting in a second feature map.

2. The remote sensing image target detection method based on fusion network according to claim 1, characterized in that, Step S5 includes: The category of each candidate box is calculated based on the fully connected layers and logistic regression algorithm in the Fast-RCNN model, and the category detection results are obtained. The position offset of each candidate box is obtained by using the bounding box regression algorithm, thus obtaining the precise position of the candidate box; The precise location of the candidate bounding box and the category detection result are used as the final object detection result.

Images