Remote sensing image scene graph generation method based on semantic topology correction and prototype learning

Abstract

The application relates to the technical field of remote sensing image processing, and particularly discloses a remote sensing image scene graph generation method based on semantic topology correction and prototype learning, which comprises the following steps: S1, acquiring a remote sensing image; S2, performing target detection and feature initialization on the remote sensing image to obtain an initialization graph, wherein the nodes of the graph are objects, and the edges of the graph are the relationships between the objects; S3, performing message passing between the nodes and the edges based on the initialization graph through an object-to-object, relationship-to-relationship and object-relationship interactive message passing mode, and obtaining a fine feature graph through iterative message aggregation and feature updating; wherein the object-to-object adopts an additive fusion strategy combined with global context and image-level semantic priori; and S4, obtaining a remote sensing image scene graph according to the fine feature graph. The application actively corrects damaged topology connections and feature representations by designing the additive fusion strategy combined with global context and semantic priori.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a method for generating scene graphs of remote sensing images based on semantic topology correction and prototype learning. Background Technology

[0002] Scene graph generation technology aims to parse unstructured images into structured semantic graphs containing nodes and relational edges, outputting triples in the form of "subject-verb-object". In the field of remote sensing image processing, mainstream solutions typically employ a two-stage processing flow: the first stage is object detection, and the second stage is relational reasoning. During message passing, existing technologies are mainly divided into vision-based methods and knowledge-aware methods. Vision-based methods, such as IMP, rely on extracted visual features as association weights for message passing between nodes. Knowledge-aware methods, represented by Motif, introduce external knowledge and utilize statistical probability or linguistic priors to assist reasoning. Statistical priors are added as biases to the relational reasoning results, or directly used as multiplicative gating weights to adjust the refined relational feature strength.

[0003] The existing methods mainly have the following problems: First, there is a severe problem of error cascading propagation, lacking an active correction mechanism during message passing. The complex backgrounds, drastic changes in object scale, high inter-class similarity, and high intra-class diversity of remote sensing images often lead to noisy features or incorrect classification labels in the first-stage target detector. Vision-based methods over-rely on these unreliable visual features for graph inference, blindly aggregating messages from all neighboring nodes with the same message passing weight during message passing, causing errors in the target detection stage to propagate directly to the relation prediction stage. While knowledge-aware methods introduce prior knowledge, they typically only apply this prior as a static bias to the relation prediction result after message passing, failing to actively correct the damaged feature representations containing detection noise during the intermediate graph update process. This results in the final inference result still being severely interfered with by the detection noise in the first stage, and the generated scene graph containing numerous incorrect connections caused by feature noise.

[0004] Second, the rigid use of prior knowledge makes it extremely sensitive to classification errors in the object detection stage. While existing knowledge-aware methods incorporate linguistic or statistical priors, they typically employ multiplicative gating mechanisms, directly controlling visual feature intensity using prior information as multiplicative gating weights. This mechanism generates incorrect prior gating weights when object label prediction is incorrect, leading to the erroneous suppression of originally effective visual contextual information and even blocking correct message passing paths. This passive fusion approach lacks fault tolerance and cannot dynamically correct connection errors in the graph structure when semantics and vision are inconsistent.

[0005] Third, the feature space distribution is chaotic, with issues of long-tail distribution and background bias. In remote sensing scene map generation tasks, head predicate relations account for a large proportion, while tail rare class samples are extremely scarce, resulting in a severe long-tail distribution problem. This makes it difficult for the model to learn tail samples, and features of the same relation category are spatially dispersed, with blurred classification boundaries, making it difficult to accurately identify long-tail relations. Furthermore, due to the extremely sparse relation annotations in the dataset, a large number of potential positive samples are unlabeled, and the statistical prior knowledge is heavily biased towards the background class, easily masking the few effective positive sample relations. This causes the model to tend to ignore visual evidence and directly predict that the predicate relation is unrelated, resulting in low recall. Summary of the Invention

[0006] This invention aims to address the problems in two-stage scene graph generation frameworks. These problems stem from the cascading amplification of perceptual noise and classification errors in the first-stage object detection phase during message passing, and the lack of fault tolerance in existing knowledge-aware methods' multiplicative gating mechanisms, which generate incorrect gating weights when faced with incorrect labels. These issues result in the erroneous suppression of effective visual context and inference failure. Furthermore, the long-tailed distribution of data and sparse annotations lead to a discrete feature space distribution and difficulty in identifying rare relationships. To address these issues, this invention provides a remote sensing image scene graph generation method based on semantic topology correction and prototype learning. By designing an additive fusion strategy that combines global context and semantic priors, it actively corrects damaged topological connections and feature representations. Simultaneously, it utilizes a prototype learning strategy to align predicted instance features to stable semantic prototypes, standardizing the feature space distribution. This allows for the generation of structurally consistent and accurate scene graphs even when object detection results are unreliable or exhibit a long-tailed distribution.

[0007] This invention provides a method for generating scene graphs from remote sensing images based on semantic topology correction and prototype learning. The technical solution adopted is as follows: including the following steps: S1: Acquire remote sensing images; S2: Perform target detection and feature initialization on the remote sensing image to obtain an initial graph, where the nodes of the graph are objects and the edges of the graph are the relationships between objects. S3: Message passing between nodes and edges is achieved through object-to-object, relation-to-relationship, and object-relationship interactions. Through iterative message aggregation and feature updates, a refined feature map is obtained. Specifically, object-to-object interactions employ an additive fusion strategy combining global context and image-level semantic priors. Based on nodes and edges, construct a fully connected interaction weight matrix initialized to all 1s and excluding self-connections. The fully connected interaction weight matrix is ​​used to maintain global information flow among all objects in the image. Based on the object category and semantic prior matrix of the node, the semantic prior path weight matrix is ​​obtained. The semantic prior path weight matrix is ​​used to introduce logical constraint error correction signals between categories. Based on the fully connected interaction weight matrix and the semantic prior path weight matrix, the aggregation result of the neighbor node messages received by the node is calculated. In step S3, in each iteration, message passing is first performed between objects and between relationships, updating the object features of the nodes. Then, message passing is calculated between relationships and between objects and between objects, updating the relationship features of the edges. Among these, object-relationship interactions include message passing from objects to relationships and message passing between relationships and objects. S4: Obtain the scene map of the remote sensing image based on the fine feature map.

[0008] Furthermore, the semantic prior matrix is ​​constructed by statistically analyzing the co-occurrence frequencies of all object categories in the remote sensing image dataset.

[0009] Furthermore, the process by which a node receives message aggregation from its neighboring nodes can be represented as follows: in, This represents the aggregation result of messages received by the i-th node from its neighboring nodes. Let j be the object features of the j-th node in the l-th iteration. Hyperparameters for controlling semantic injection strength Let be the learnable linear projection matrix of the first node domain. Let be the learnable linear projection matrix of the second node domain. The set of all nodes. for The elements corresponding to node i and node j in the middle. This is the fully connected interaction weight matrix. for The elements corresponding to node i and node j in the middle. This is the semantic prior path weight matrix.

[0010] Furthermore, the formulas for calculating the object features of a node and the relational features of its edges are as follows: in, Let i be the object features of node i in the (l+1)th round. It is the ReLU activation function. This represents the aggregation result of messages received by the i-th node from its neighboring nodes. For nodes i Aggregate the message aggregation results of the edges connected to it. Let be the feature vector of the i-th node in the l-th iteration. Let m be the relational feature of edge m in the (l+1)th round. This represents the aggregation result of other edge messages received by the m-th edge, where the m-th edge is the edge connecting node i and node j. Aggregate the message aggregation results of edge m and its connected nodes. The relational characteristics of edge m in the l-th round.

[0011] Furthermore, message passing between objects, between relationships, and from objects to relationships all utilizes a fully connected interaction weight matrix to aggregate messages.

[0012] Furthermore, the joint objective function The calculation formula is: in, The classification loss is the cross-entropy between object features and their true class labels. The cross-entropy loss is calculated based on the cosine similarity between relational features and truth prototypes. For instance-level contrast loss, For orthogonal loss, This represents the loss between prototypes.

[0013] Furthermore, in, The relationship features between node i and node j For positive sample predicate prototypes, To conduct Normalization For negative sample predicate prototypes, 2 indicates calculation Norm, To obtain the maximum value, The first threshold, Indicates calculation Norm, For prototype matrix, for The transpose of the matrix, To obtain the minimum value, For the m-th prototype, For the nth prototype, This is the second threshold.

[0014] Furthermore, in step S2, Using the pre-trained object detector HOD-Net, objects in remote sensing images are located and their categories are predicted. The object pair proposal generation module removes obvious non-interactive background pairs to generate a sparse set of adjacent object pairs. Visual features are extracted using RoIAlign, semantic features are extracted using the GloVe word vector model based on the predicted object category labels, and spatial features are obtained by encoding the normalized object spatial bounding box. Visual features, semantic features, and spatial features are concatenated and projected onto a unified-dimensional latent space through a multilayer perceptron to obtain the initial object features of the nodes. The initial relationship features of the edges are obtained by extracting the visual features of the joint bounding box of the two objects and fusing them with the paired spatial features of the two objects through element-level addition.

[0015] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: This invention addresses the problem of error cascading propagation caused by target detection errors in a two-stage framework. It innovatively constructs parallel global contextual pathways and semantic prior pathways, and generates dynamic message passing weights through an additive fusion strategy with controllable semantic injection strength. This decouples visual perception from semantic reasoning, effectively preventing the accumulation and diffusion of detection noise into the relation prediction stage, and overcoming the information blocking defects caused by existing multiplicative gating mechanisms when object labels are incorrect. Furthermore, addressing the issues of long-tailed distribution, high inter-class similarity, and high intra-class dissimilarity in remote sensing data, this invention introduces a joint prototype learning mechanism that includes prototype matching, intra-class compactness, inter-class orthogonality, and push-away constraints. This mechanism forces noisy instance features to align to stable semantic prototypes, effectively solving the problem of difficult rare relation identification.

[0016] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the method provided by the present invention.

[0019] Figure 2 This is a visualization of the experimental results provided by the present invention.

[0020] Figure 3 This is a comparison chart of the grouping predicate recall gain results provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.

[0022] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0023] The following is combined with Figures 1 to 3 The present invention will be further described in detail below, which describes a method for generating scene graphs from remote sensing images based on semantic topology correction and prototype learning: In this embodiment, as Figure 1 As shown, a method for generating scene graphs from remote sensing images based on semantic topology correction and prototype learning is provided, including the following steps: S1: Acquire remote sensing images.

[0024] S2: Perform target detection and feature initialization on the remote sensing image to obtain an initial graph, where the nodes of the graph represent objects and the edges represent the relationships between objects.

[0025] The specific process is as follows: For the input remote sensing image, the pre-trained object detector HOD-Net is used to locate objects and predict their categories.

[0026] To address the characteristics of large image size and dense object density in remote sensing images, and to reduce computational complexity, a Pair Proposal Generation (PPG) module is used to remove obvious non-interactive background pairs and generate a sparse set of adjacent object pairs.

[0027] For each retained node in the set of neighboring object pairs, its initial object features are constructed. Specifically, the region visual features extracted from the feature map of the object detector by RoIAlign, the semantic features extracted by the GloVe word vector model based on the predicted object category label, and the spatial features obtained by encoding the normalized object space bounding box are concatenated and projected onto a latent space of a unified dimension through a multilayer perceptron (MLP) to obtain the initial object features of the node.

[0028] For the edges connecting pairs of objects, the initial relational features are obtained by extracting the visual features of the joint bounding box of the two objects and fusing them with the paired spatial features of the two objects through element-level addition.

[0029] At this point, the object pair proposal generation is complete, and the initialization graph is obtained.

[0030] S3: Through message passing methods of object-to-object, relation-to-relationship, and object-relationship interactions, based on the initial graph, message passing is carried out between nodes and edges. After iterative message aggregation and feature updates, a fine feature map is obtained. Among them, the object-to-object interaction adopts an additive fusion strategy that combines global context and image-level semantic priors.

[0031] This embodiment employs a graph neural network to perform multi-round iterative message passing between objects (nodes) and relationships (edges). To fully capture the contextual dependencies in the scene, this process includes three parallel message passing paths: 1. Object-to-object: propagating contextual features between nodes to capture dependencies between objects; 2. Relation-to-relation: directly exchanging structured information between edges; 3. Object-relation interaction (including object-to-relation and relation-to-object): performing bidirectional feature aggregation between nodes and their connected edges.

[0032] 1. For object-to-object message passing, unlike traditional methods that use prior information as a multiplication gate, this embodiment employs an additive fusion strategy to decouple semantic reasoning from perceptual errors. This embodiment designs a dynamic dual-path topology correction mechanism to generate message passing weights, comprising two parallel paths: (1) Global context path: Based on the nodes and edges of the initial graph, construct a fully connected interaction weight matrix that is initialized to all 1s and excludes self-connections. This is used to maintain global information flow among all objects in an image, ensuring that long-distance dependencies are not lost. For any relation pair ( i , j ): in, for The elements corresponding to node i and node j in the middle. This is the fully connected interaction weight matrix.

[0033] (2) Semantic prior path: based on the object category predicted by object detection. and In the semantic prior matrix The semantic prior path weight matrix is ​​obtained by looking up the table. It is used to introduce logical constraint error correction signals between categories.

[0034] in, for The elements corresponding to node i and node j in the middle. This is the semantic prior path weight matrix. For the corresponding object category in the semantic prior matrix and The element value, This is the semantic prior matrix. Let i be the object category of the i-th node. Let j be the object category of the j-th node.

[0035] Construction of the semantic prior matrix: To mitigate noise interference in visual features, this method explicitly models the intrinsic semantic dependencies between visually irrelevant object categories for subsequent correction of visual perception errors. A global semantic relevance matrix is ​​constructed by statistically analyzing the co-occurrence frequency of all object categories in remote sensing image datasets (e.g., the STAR dataset): the semantic prior matrix. , ,in, The total number of object categories. Let be the real number field. For any two object categories... and Elements in the matrix Indicates that when the known subject is In the case of the object Conditional probability of co-occurrence : in, For remote sensing image datasets that simultaneously contain and The number of images, For remote sensing image datasets containing The total number of images. This calculation method is based on image-level co-occurrence statistics, avoiding statistical bias caused by the repeated occurrence of certain high-frequency objects within a single image.

[0036] The message passing process involves alternating feature aggregation and updating of nodes and edges. During the aggregation phase, this embodiment employs different strategies to decouple semantic reasoning from perceived errors.

[0037] For noise-sensitive object-to-object information flows, an additive fusion strategy based on dual-path topology correction is adopted, which explicitly incorporates semantic priors as correction terms into the visual context to alleviate visual ambiguity.

[0038] The process by which a node receives message aggregation from its neighboring nodes can be represented as follows: in, This represents the aggregation result of messages received by the i-th node from its neighboring nodes. Let j be the object features of the j-th node in the l-th iteration. Hyperparameters for controlling semantic injection strength Let be the learnable linear projection matrix of the first node domain. Let be the learnable linear projection matrix of the second node domain. It is the set of all nodes.

[0039] This mechanism allows the model to use semantic priors as correction signals when visual features are unreliable, i.e., the object features of the j-th node. Semantic pathways in the presence of noise It can dynamically adjust aggregation weights and proactively correct topological connections based on category co-occurrence logic, dynamically guide information flow, suppress erroneous connections, and strengthen reasonable connections.

[0040] 2. To maintain stable connections within the structure, only use [specific methods] for relationship aggregation paths. ,side m (Connecting node i and node j) Aggregates data from other edges n Message: in, This is the aggregation result of other edge messages received by the m-th edge. for The elements corresponding to edge m and edge n are given, where the m-th edge is the edge connecting node i and node j. Let n be the relational feature of the nth edge in the l-th iteration. It is a linear projection matrix. Let be the set of all edges.

[0041] 3. For object-relationship interaction information flow, including message passing from objects to relationships and message passing from relationships to objects, both utilize... To ensure robust overall topological connectivity. Represents a node i Aggregate the edges that connect the subject ( ) and the edge that connects as an object ( ), Representing an edge m Aggregate its subject node i and object node j The information is as follows: in, For nodes i Aggregate the message aggregation results of the edges connected to it. For nodes i Edges connected as subjects for The elements corresponding to node i and edge n in the middle. The linear projection matrix of the first subject character. For nodes i As an edge connecting objects The linear projection matrix of the first object role. Aggregate the message aggregation results of edge m and its connected nodes. for The elements corresponding to node i on edge m. The linear projection matrix for the second subject role. Let be the feature vector of the i-th node in the l-th iteration. for The elements corresponding to edge m and node j. is the linear projection matrix of the second object role.

[0042] Feature updates for nodes and edges are performed using aggregated messages. Nodes receive messages from other nodes and edges, while edges receive messages from other edges and nodes. The message passing weights are normalized based on the number of objects from which the aggregated messages originate, and then added to their own features via residual connections. Specifically, in each iteration, node message passing is performed first, followed by edge message passing; that is, the features are calculated first... and ,renew Then calculate and ,renew .

[0043] and The calculation formulas are as follows: in, It is the ReLU activation function. Let i be the object features of node i in the (l+1)th round. Let m be the relational feature of edge m in the (l+1)th round. The relational characteristics of edge m in the l-th round.

[0044] After L iterations, the final refined object features and relationship features are obtained. L is the total number of iterations.

[0045] For each predicate category, initialize it with GloVe word vectors as a learnable prototype vector in the shared feature space. To align noisy instance-level features with stable prototype-level semantic representations, this invention designs a joint objective function that includes multiple constraints. The calculation formula is: .

[0046] The classification loss is the cross-entropy between object features and their true class labels.

[0047] The cross-entropy loss is calculated based on the cosine similarity between relational features and truth prototypes: in, The relationship features between node i and node j For positive sample predicate prototypes, For the learnable prototype vector of the predicate category, To conduct Normalization This represents the inner product operation, where K is the number of predicate prototypes, and is a hyperparameter. This is the temperature coefficient.

[0048] To address intra-class variability, instance-level contrastive loss is used. By minimizing features to the predicate prototype of positive samples The Euclidean distance, while pushing away the negative sample predicate prototypes. And ensure that the distance between positive and negative samples is greater than the first threshold. To ensure that instance features are tightly clustered around the prototype, the calculation formula is: in, 2 indicates calculation Norm, To obtain the maximum value.

[0049] To address inter-class similarity, orthogonal loss is used. pass Norm-constrained prototype matrix With the corresponding transpose matrix The orthogonality of these elements makes prototypes of different categories as distinguishable as possible in the semantic space. The calculation formula is as follows: in, Indicates calculation Norm, For prototype matrix, for The transpose of .

[0050] Force the distance between different prototypes to be greater than the second threshold Ensure clear classification boundaries: in, To obtain the minimum value, For the m-th prototype, This is the nth prototype.

[0051] By optimizing these loss terms, the model can mitigate long-tail bias and visual blur in remote sensing scene maps.

[0052] In the relational reasoning stage, the cosine similarity between refined relational features and all learnable prototype vectors is calculated, and a temperature coefficient is used. The prototype index k with the highest similarity is selected as the final predicted predicate category, generating structured semantic triples.

[0053] in, For the final predicted predicate category, The value of k to maximize the expression within the parentheses.

[0054] S4: Obtain the scene map of the remote sensing image based on the fine feature map.

[0055] This embodiment demonstrates significant technical advantages in remote sensing scene map generation tasks through the design of the STORM framework, specifically in the following aspects: This method significantly improves the accuracy of relational reasoning and effectively mitigates the propagation of accumulated errors. Addressing target detection noise caused by blurred object features in remote sensing images, this embodiment constructs a semantic prior path independent of visual noise, introducing additional semantic logic verification information beyond visual feature extraction. This embodiment conducts comparative experiments on three sub-tasks of scene graph generation: 1. Predicate Classification (PredCls): Predicting predicate labels given ground truth bounding boxes and object categories; 2. Scene Graph Classification (SGCls): Predicting object categories and predicates given ground truth bounding boxes; 3. Scene Graph Detection (SGDet): Directly detecting objects from the original image and predicting the relationships between them. Furthermore, based on the bounding box format used in object detection, the scene graph generation task is further divided into two categories: HBB (Horizontal Bounding Box) and OBB (Rotated Bounding Box). To comprehensively evaluate performance, especially for multi-class long-tail distributions, this embodiment uses multi-label recall (MR@K), mean multi-label recall (mMR@K), and harmonic mean recall (HMR@K). In this embodiment, K is set to 1500 and 2000.

[0056] As shown in Tables 1 and 2, experiments on the STAR dataset demonstrate that this embodiment performs competitively in the PredCls predicate classification task compared to existing suboptimal methods, and achieves significant advantages in the SGCls scene graph classification task, which best reflects relational reasoning capabilities. Specifically, in the SGCls task, this embodiment improves the mMR@2000 metric by 4.08% with the horizontal bounding box (HBB) setting and by 4.52% with the rotated bounding box (OBB) setting. These significant performance improvements quantitatively verify that this embodiment can effectively prevent the accumulation and propagation of errors from the detection stage to the inference stage, achieving effective decoupling between visual perception and semantic reasoning.

[0057] Table 1 Comparative Experiment Results of Predicate Classification Task

[0058] Table 2 Comparison Experiment Results of Scene Graph Classification Tasks

[0059] Table 3 Comparative Experiment Results of Scene Graph Detection Task

[0060] This method possesses strong noise robustness and latent relationship discovery capabilities, making it suitable for real-world and complex remote sensing application environments. As shown in Table 3, this embodiment maintains superior performance and achieves significant gains even in the most challenging scene graph detection SGDet task. This indicates that even if the features output by the front-end detector contain significant noise or positional bias, the dynamic topology correction mechanism of this embodiment can still actively recover damaged information through semantic signals, demonstrating strong robustness in real-world and complex remote sensing scenarios. Figure 2 The visualization experiment results further show that in complex scenes with dense objects (such as ports), this embodiment is significantly better than RPCM. It can not only accurately identify basic relationships, but also discover potential reasonable relationships that are not manually labeled but are logically consistent through semantic reasoning, such as "within safe distance of". This proves that this embodiment has extremely strong reasoning robustness and information completion ability in real complex scenes.

[0061] This method effectively solves the challenge of identifying rare relationships caused by long-tail distribution, achieving feature space equalization. Addressing the prevalent long-tail distribution problem in remote sensing data, characterized by dominant head relationships and rare tail relationships, this embodiment combines a prototype learning mechanism with intra-class compactness and inter-class orthogonality constraints. This aligns noisy instance features to a stable semantic prototype, clearly defining the discrimination boundary for rare categories and significantly improving the recognition ability of tail categories. In the scene graph classification task of OBB (Optical Object Bypass) rotation target detection, predicates are divided into three groups—head, middle, and tail—based on the cumulative predicate count, at proportions of 65%, 25%, and 10%. Figure 3 The vertical axis represents the cumulative recall of each predicate within each group, and the average cumulative recall of all predicates is the metric mMR@1500. Experimental data show that compared to the baseline method RPCM, this embodiment achieves the largest improvement in cumulative recall for the tail predicate category, increasing by 1.91, demonstrating the effectiveness of this method in handling data imbalance and significantly improving the model's ability to capture rare relations.

[0062] This method establishes a dynamic balance between visual perception and semantic reasoning, exhibiting good interpretability. This embodiment verifies the balancing effect of semantic injection intensity α on the global contextual pathway and the semantic prior pathway through ablation experiments. As shown in Table 4, the experiments demonstrate that the model performance reaches its peak when α=2.0. This result quantitatively reveals the balance between visual features and semantic logic; excessively low semantic priors are insufficient to correct visual errors, while excessively high semantic priors can obscure the visual information itself. The additive fusion strategy provided in this embodiment successfully finds this optimal trade-off point, ensuring the accuracy and robustness of the model's reasoning.

[0063] Table 4. Results of Semantic Injection Intensity Ablation Experiment

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating scene graphs from remotely sensed images based on semantic topology correction and prototype learning, characterized in that, include: S1: Acquire remote sensing images; S2: Perform target detection and feature initialization on the remote sensing image to obtain an initial graph, where the nodes of the graph are objects and the edges of the graph are the relationships between objects. S3: Message passing between nodes and edges is achieved through object-to-object, relation-to-relationship, and object-relationship interactions. Through iterative message aggregation and feature updates, a refined feature map is obtained. Specifically, object-to-object interactions employ an additive fusion strategy combining global context and image-level semantic priors. Based on nodes and edges, construct a fully connected interaction weight matrix initialized to all 1s and excluding self-connections. The fully connected interaction weight matrix is ​​used to maintain global information flow among all objects in the image. Based on the object category and semantic prior matrix of the node, the semantic prior path weight matrix is ​​obtained. The semantic prior path weight matrix is ​​used to introduce logical constraint error correction signals between categories. Based on the fully connected interaction weight matrix and the semantic prior path weight matrix, the aggregation result of the neighbor node messages received by the node is calculated. In step S3, in each iteration, message passing is first performed between objects and between relationships, updating the object features of the nodes. Then, message passing is calculated between relationships and between objects and between objects, updating the relationship features of the edges. Among these, object-relationship interactions include message passing from objects to relationships and message passing between relationships and objects. S4: Obtain the scene map of the remote sensing image based on the fine feature map.

2. The remote sensing image scene graph generation method based on semantic topology correction and prototype learning as described in claim 1, characterized in that, The semantic prior matrix is ​​constructed by statistically analyzing the co-occurrence frequency of all object categories in the remote sensing image dataset.

3. The remote sensing image scene graph generation method based on semantic topology correction and prototype learning as described in claim 1, characterized in that, The process by which a node receives message aggregation from its neighboring nodes can be represented as follows: in, This represents the aggregation result of messages received by the i-th node from its neighboring nodes. Let j be the object features of the j-th node in the l-th iteration. Hyperparameters for controlling semantic injection strength Let be the learnable linear projection matrix of the first node domain. Let be the learnable linear projection matrix of the second node domain. The set of all nodes. for The elements corresponding to node i and node j in the middle. This is the fully connected interaction weight matrix. for The elements corresponding to node i and node j in the middle. This is the semantic prior path weight matrix.

4. The remote sensing image scene graph generation method based on semantic topology correction and prototype learning as described in claim 1, characterized in that, The formulas for calculating the object features of a node and the relationship features of its edges are as follows: in, Let i be the object features of node i in the (l+1)th round. It is the ReLU activation function. This is the aggregation result of the neighbor node messages received by the i-th node. For nodes i Aggregate the message aggregation results of the edges connected to it. Let be the feature vector of the i-th node in the l-th iteration. Let m be the relational feature of edge m in the (l+1)th round. This represents the aggregation result of other edge messages received by the m-th edge, where the m-th edge is the edge connecting node i and node j. Aggregate the message aggregation results of edge m and its connected nodes. The relational characteristics of edge m in the l-th round.

5. The remote sensing image scene graph generation method based on semantic topology correction and prototype learning as described in claim 1, characterized in that, Message passing between objects, between relationships, and from objects to relationships all utilizes a fully connected interaction weight matrix to aggregate messages.

6. The remote sensing image scene graph generation method based on semantic topology correction and prototype learning as described in claim 1, characterized in that, Joint objective function The calculation formula is: in, The classification loss is the cross-entropy between object features and their true class labels. The cross-entropy loss is calculated based on the cosine similarity between relational features and truth prototypes. For instance-level contrast loss, For orthogonal loss, This represents the loss between prototypes.

7. The remote sensing image scene graph generation method based on semantic topology correction and prototype learning as described in claim 6, characterized in that, in, Let i be the relationship feature between node i and node j. For positive sample predicate prototypes, To conduct Normalization For negative sample predicate prototypes, 2 indicates calculation Norm, To obtain the maximum value, The first threshold, Indicates calculation Norm, For prototype matrix, for The transpose of the matrix, To obtain the minimum value, For the m-th prototype, For the nth prototype, This is the second threshold.

8. The remote sensing image scene graph generation method based on semantic topology correction and prototype learning as described in claim 1, characterized in that, In step S2, Using the pre-trained object detector HOD-Net, objects in remote sensing images are located and their categories are predicted. The object pair proposal generation module removes obvious non-interactive background pairs to generate a sparse set of adjacent object pairs. Visual features are extracted using RoIAlign, semantic features are extracted using the GloVe word vector model based on the predicted object category labels, and spatial features are obtained by encoding the normalized object spatial bounding box. Visual features, semantic features, and spatial features are concatenated and projected onto a unified-dimensional latent space through a multilayer perceptron to obtain the initial object features of the nodes. The initial relationship features of the edges are obtained by extracting the visual features of the joint bounding box of the two objects and fusing them with the paired spatial features of the two objects through element-level addition.

Images