A remote sensing image scene graph generation method based on spatial anisotropic reasoning

Abstract

The application discloses a remote sensing image scene graph generation method based on spatial anisotropic reasoning, comprising the steps of obtaining a remote sensing image and extracting object features, rotating a target frame and an initial class score by using a directional target detector; generating a relationship candidate edge and a corresponding directional joint frame and a relationship feature based on a candidate object pair; fusing object visual features, spatial embedding and semantic embedding through an object enhancer; constructing an anisotropic elliptical influence propagation module in the relationship coding stage, mapping each rotating target frame into an adaptive Gaussian elliptical influence area, and generating a sparse symmetric gating matrix according to the elliptical metric to modulate the message aggregation of the relationship edge to the object node in the bidirectional message passing; constructing a relative spatial configuration attention module in the relationship decoding stage, injecting the relative geometric configuration of the subject, object and their joint area into the relationship feature and performing predicate classification, and finally outputting a remote sensing image scene graph composed of a subject object, a relationship predicate and an object object.

Description

Technical Field

[0001] This invention relates to the field of computer vision and intelligent interpretation of remote sensing images, and in particular to a method for generating scene graphs of remote sensing images based on spatial anisotropy reasoning. Background Technology

[0002] Intelligent interpretation of remote sensing images has evolved from single entity recognition tasks such as target detection, semantic segmentation, and change detection to a stage of deep understanding of objects, regions, and their interrelationships in complex scenes. Scene graphs, using objects as nodes and semantic relationships between objects as edges, can organize scattered target categories, spatial locations, and relational predicates in remote sensing images into structured graph representations, providing crucial support for scene understanding, visual question answering, and spatial reasoning in multimodal large models.

[0003] Compared to natural images, remote sensing images typically have characteristics such as large coverage areas, a large number of imaged objects, significant differences in target scale, and complex scene composition. A single remote sensing image may simultaneously contain multiple functional areas such as ports, airports, roads, docks, and residential areas. Target objects within different functional areas exhibit significant differences in spatial distribution, semantic attributes, and relationship types. For target objects with similar functional attributes or spatial correlation characteristics, their effective relationships usually occur concentrated within local functional areas. If unconstrained global relationship interactions are performed on all target objects within the entire large-format remote sensing image, redundant correlations and noise information between distant, weakly correlated targets can easily be introduced, thereby weakening the expressive power of the local effective relationship context and affecting the accuracy of the generated scene map from the remote sensing image.

[0004] Furthermore, target objects in remote sensing images often exhibit characteristics such as insufficient appearance and texture details, lack of depth information, and relatively fixed imaging perspectives. The discrimination of relationship predicates between targets often relies heavily on the spatial configuration relationships between the subject and object objects, such as geometric structure, relative direction, target distance, overlap, parallelism, intersection, same-direction, or opposite-direction relationships. Existing scene graph generation methods are mostly designed for natural images, typically employing global attention mechanisms or uniform message passing to model inter-object relationships. These methods fail to fully integrate the local influence range of the target center and the relative spatial configuration characteristics of the subject and object within the relationship in the remote sensing scene, resulting in room for improvement in the accuracy and robustness of relationship predicate prediction in complex, large-format remote sensing images. Therefore, a remote sensing image scene graph generation method based on spatial anisotropy reasoning is needed to address these issues. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a remote sensing image scene graph generation method based on spatial anisotropy reasoning, which can alleviate the problems of large global interactive noise, insufficient local functional context awareness of the object center, and insufficient characterization of the relative spatial configuration of the subject and object within the relation by relational predicates in complex and large-format remote sensing images.

[0006] To achieve the above technical solution, the technical solution adopted by the present invention is as follows: A method for generating scene graphs from remotely sensed images based on spatial anisotropy reasoning includes the following steps: S1, Remote sensing image target detection: Acquiring the remote sensing image to be processed. The remote sensing image is input into a pre-trained directional target detector to obtain object features. Rotate the target box of the object and initial object category score ; S2, Candidate Relation Extraction: The detected objects are combined with relation pairs and pruned to form a candidate edge set. For the candidate edge set Each of the items in the main object and object The candidate relationships are constructed, generating a directed union box that encloses the subject object and the object object. Relationship features are obtained from the feature map of the entire image by pooling based on the directed joint bounding box. Thus, the set of relational features is obtained. and relation-oriented union frame set ; S3, Object Feature Enhancement: Enhances object features Rotate the target box from the object The obtained spatial embedding and the initial object category score The obtained semantic embedding input object enhancer obtains enhanced object features. and updated object category scores ; S4, Spatial Influence Relation Encoding: Object features, enhanced object features, semantic embeddings generated from updated object category scores, and relation features are input into the relation encoder to obtain encoded object features. Relationship characteristics ; S5, Decoding relative spatial configuration relationships: Decoding the encoded object features Relationship characteristics Input the relative spatial configuration attention module to obtain the spatial configuration-enhanced relation representation; S6, Scene Graph Generation: The features of the subject object, the features of the object, and the spatial configuration-enhanced relation representation are fused together to output the predicate category scores of the candidate relations. The object category scores are then combined to generate a remote sensing image scene graph containing the subject object, relation predicates, and object objects.

[0007] Preferably, S1 specifically includes: Acquire remote sensing images to be processed The remote sensing image is input into a pre-trained directional target detector, which then locates, classifies, and extracts features from the target objects in the remote sensing image. The rotated bounding boxes output by the directional target detector represent target objects distributed in arbitrary directions within the remote sensing image. Each rotated target box in the diagram is represented as: ; in, For the first Rotate the target bounding box of an object; and The first The x and y coordinates of the center point of the object's rotation target bounding box in the image coordinate system; and The first Rotate the width and height of the target bounding box of an object; For the first The orientation angle of the object's rotating target bounding box relative to the image coordinate system.

[0008] The detection and feature extraction process of the directional target detector is as follows: ; in, This represents the set of all object visual features extracted by the directional target detector; This represents the set of category scores obtained by the directional target detector for the initial prediction of each object category.

[0009] Preferably, S2 specifically includes: Relationship pairs are constructed for detected objects in remote sensing images, and any two objects are grouped into candidate relationship pairs according to the form of subject and object. Subsequently, candidate relationship pruning is performed on the candidate relationship pairs to remove relationship pairs with low probability of existence, and retain the set of candidate edges used for relational predicate reasoning. .

[0010] For the candidate edge set Each candidate relationship in ,in Represents the main object, Represents the object, rotating the target box according to the subject object. Rotate the target box of the object Construct a directional union box that can enclose both simultaneously. and based on the directional joint frame Region feature pooling is performed on the feature map of the remote sensing image to obtain the relation features corresponding to the candidate relations. .

[0011] The candidate relation extraction process satisfies: ; in, A set of directed union boxes for candidate relations; This is the set of relation features for candidate relations.

[0012] Preferably, in S3, the specific implementation of the object feature enhancement is as follows: Rotate the target box set of the object Perform spatial embedding and assign initial object category scores Perform semantic embedding to incorporate object features The input object enhancers of both are merged and updated to satisfy the following: ; in, This represents the set of enhanced object features after being updated by the object enhancer; This represents the final object category score after being updated by the object enhancer; As a spatial embedding function, the normalized width, normalized height, normalized center coordinates, aspect ratio, orientation angle, scale ratio relative to the image diagonal, scale ratio relative to the image area, and center horizontal and vertical coordinate ratio of the object's rotating target box are encoded and mapped through a multilayer perceptron to obtain spatial geometric features. The semantic embedding function normalizes the initial object category scores into category probabilities, and then performs a weighted combination of the category semantic embedding matrix initialized by GloVe pre-trained word vectors according to the category probabilities to obtain the object category semantic features.

[0013] Preferably, in step S4, the relation encoder first captures global interactions between objects through hybrid attention, then constructs the spatial influence region of the object center through the anisotropic elliptical influence propagation module, and modulates the message aggregation from relation edges to object nodes in bidirectional message passing using a sparse symmetric gating matrix generated by the geometric embedding of the object rotation target box and the semantic embedding of the object, thereby obtaining the encoded object features. Relationship characteristics The specific methods for spatial influence relationship coding are as follows: The anisotropic ellipse influence propagation module performs bidirectional message passing between object nodes and relation edges; in the... In this iteration, the relation edges are updated by the subject object node and the object object node, satisfying: ; in, It is a non-linear activation function. and These are the learnable projection matrices for the subject and object objects, respectively.

[0014] The object node is updated with weighted relation edges associated with it when it is both the subject and the object, satisfying the following: ; in, It is a sparse symmetric gated matrix. and Representing objects respectively When acting as a subject and an object, it represents the set of adjacent objects. and These represent the relational message projection matrices when the objects are the subject and object, respectively.

[0015] Preferably, the construction of the sparse symmetric gating matrix G includes modeling the anisotropic elliptical influence region, specifically as follows: Map the rotation bounding box of each object to a two-dimensional Gaussian distribution. ,in Construct a diagonal covariance matrix in the local coordinate system of the target bounding box based on the object's width, height, and effective expansion factor, satisfying: ; in, and Representing objects respectively Effective expansion factor along the width and height directions; and Representing objects respectively The width and height of the rotated target bounding box; It is a numerically stable term.

[0016] Furthermore, based on the object's orientation angle Rotating the local covariance to the global image coordinate system satisfies: ; ; in, Indicates by direction angle A defined rotation matrix; Representation Object The covariance matrix in the global image coordinate system.

[0017] Preferably, the sparse symmetric gating matrix G is generated based on the elliptic metric distance between the subject object and the object object in the candidate relation, specifically: For candidate relations The covariance matrices of the two objects are added together to obtain the joint covariance matrix, and the elliptic metric distance between the centers of the two objects is calculated, satisfying: ; ; in, Denotes the joint covariance matrix. Indicates candidate relationship The corresponding elliptic metric distance; This represents the distance scale adjustment factor; and These represent the coordinates of the center point of the subject and the object, respectively.

[0018] Furthermore, the gating weights are obtained based on the elliptic metric distance and the distance decay intensity, satisfying: ; ; ; in, Indicates candidate relationship Gating weights; Fill in the corresponding positions of the candidate edges, perform symmetry processing, and set the diagonal elements to zero to obtain a sparse symmetric gated matrix. . This represents the candidate relation energy term determined by both the symmetric distance attenuation intensity and the elliptic metric distance. This indicates the upper limit of the smooth truncation.

[0019] Preferably, in S5, the relation features of each candidate relation are used as the query, the object features of the corresponding subject object and object object are used as the key and value, and the relative spatial configuration between the subject object, object object and their directed union box is embedded in the attention calculation process to obtain the relation representation with enhanced spatial configuration; the specific implementation of the relative spatial configuration attention is as follows: Spatial embedding is performed on the rotating bounding boxes of the subject object, the rotating bounding boxes of the object object, and the orientation union box to obtain the subject geometric embedding, object geometric embedding, and relation geometric embedding. Based on the differences between the relation geometric embedding and the subject and object geometric embeddings, relative geometric representations of the subject and object are constructed. The relative spatial configuration attention module generates queries using candidate relation features, generates keys and values ​​using corresponding subject and object features, and incorporates the relative geometric representations into the attention matching process. The geometric modulation attention weights satisfy the following: ; in, Indicates candidate relationship The geometric modulation attention weights corresponding to object n in the middle. ; This represents a query vector generated from relational features; Represents a key vector generated from the features of a subject object or an object object; Representation Object Relative geometric representation of the directed union box.

[0020] Preferably, in S5, the decoding of the relative spatial configuration relationship is specifically performed as follows: The relation features are updated through fully connected layers and residual connections; when stacking multiple layers of relative space to configure attention modules, the relation features output by each layer are averaged to obtain the final relation representation, which is used for subsequent relation predicate classification.

[0021] Preferably, the object classification loss and the relation predicate classification loss are jointly optimized during the training phase, as follows: The object classification loss is used to constrain the updated object category scores, and the relation predicate classification loss is used to constrain the predicate category scores of candidate relations. Together, they are used to optimize the remote sensing image scene graph generation results.

[0022] Furthermore, a remote sensing image scene graph generation model architecture based on spatial anisotropic reasoning is provided to execute the aforementioned remote sensing image scene graph generation method based on spatial anisotropic reasoning. The model architecture includes a directional target detector, a candidate relation extraction module, an object enhancer, a relation encoder, a relation decoder, and a scene graph output module. The relation encoder includes a hybrid attention module and an anisotropic elliptical influence propagation module; the relation decoder includes relative spatial configuration relation decoding; and the scene graph output module is used to generate a remote sensing image scene graph composed of subject objects, relation predicates, and object objects based on object category scores and relation predicate category scores.

[0023] The beneficial effects of this invention are as follows: 1. Enhanced context awareness of local spatial influence at the object center: The soft spatial scope of each object is described by the anisotropic elliptical influence region, enabling the object to selectively aggregate the spatially and semantically related relational context around it.

[0024] 2. Weakly correlated noise interaction is suppressed: The aggregation of relation messages to object nodes is modulated by a sparse symmetric gating matrix based on elliptic metric distance, thereby optimizing the bidirectional information interaction between object nodes and relation edges and reducing the interference of weakly correlated objects located outside the local space of the object on relation reasoning.

[0025] 3. Enhanced spatial configuration-related predicate reasoning ability: By introducing the relative geometric configuration of the subject object, object object and its directed union box into relation decoding through relative spatial configuration attention, predicate prediction can comprehensively utilize spatial clues such as the position, direction, scale and layout differences of the object relative to the directed union box.

[0026] 4. Enhanced adaptability to complex remote sensing scenes: This method forms a progressive inference chain consisting of global context modeling, object-centered local influence propagation, and relational internal spatial configuration decoding. It is suitable for remote sensing image scene graph generation tasks with large coverage, dense target distribution, and significant differences in target scale and orientation. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the steps of a remote sensing image scene graph generation method based on spatial anisotropy reasoning according to the present invention. Figure 2 This is a schematic diagram illustrating the construction of the anisotropic Gaussian ellipse influence region in this invention; Figure 3 This is a schematic diagram of the relative spatial configuration attention module in this invention. Detailed Implementation

[0028] Example 1: like Figure 1 As shown, the method employs a two-stage scene graph generation framework: the first stage utilizes a pre-trained directional object detector to extract object features, object rotation bounding boxes, and object category scores from the remote sensing image; the second stage sequentially uses an object enhancer, a relation encoder, and a relation decoder to infer the relationships between candidate objects, outputting a remote sensing image scene graph composed of the subject object, relation predicates, and object objects. The steps are as follows: S1, Remote sensing image target detection: Acquiring the remote sensing image to be processed. The remote sensing image is input into a pre-trained directional target detector to obtain object features. Rotate the target box of the object and initial object category score ; S2, Candidate Relation Extraction: The detected objects are combined with relation pairs and pruned to form a candidate edge set. For the candidate edge set Each of the items in the main object and object The candidate relationships are constructed, and a directed union box is generated that surrounds the subject object and the object object. Relationship features are obtained from the feature map of the entire image by pooling based on the directional joint bounding box. Thus, the set of relational features is obtained. and relation-oriented union frame set ; S3, Object Feature Enhancement: Enhances object features Rotate the target box from the object The obtained spatial embedding and the initial object category score The obtained semantic embedding input object enhancer yields enhanced object features O~ and updated object class scores. ; S4, Spatial Influence Relationship Encoding: Object features, enhanced object features, semantic embeddings generated from updated object category scores, and relation features are input into the relation encoder. The relation encoder first captures global interactions between objects through hybrid attention, then constructs the spatial influence region of the object center through the anisotropic elliptical influence propagation module. It modulates message aggregation from relation edges to object nodes in bidirectional message passing using a sparse symmetric gating matrix generated from the geometric embedding of the object's rotated bounding box and the semantic embedding of the object, thus obtaining the encoded object features. Relationship characteristics ; S5, Decoding relative spatial configuration relationships: Decoding the encoded object features Relationship characteristics Input the relative spatial configuration attention module, use the relation features of each candidate relation as the query, use the object features of the corresponding subject object and object object as the key and value, and embed the relative spatial configuration between the subject object, object object and their directed union box into the attention calculation process to obtain the relation representation after spatial configuration enhancement; S6, Scene Graph Generation: The features of the subject object, the features of the object, and the spatial configuration-enhanced relation representation are fused together to output the predicate category scores of the candidate relations. The object category scores are then combined to generate a remote sensing image scene graph containing the subject object, relation predicates, and object objects.

[0029] The steps described above are explained in detail below.

[0030] The method first performs directional target detection and candidate relation extraction on the remote sensing image, and then uses a spatial anisotropic inference network to encode, decode, and classify object features and relation features, and finally generates a scene map of the remote sensing image.

[0031] S1, Remote sensing image target detection. Acquiring the remote sensing image to be processed. The remote sensing image Input the pre-trained directional target detector to obtain the object feature set. Object rotation target box set and the initial set of object category scores The set of object rotation target boxes. Each rotated target box in the diagram is represented as ; in and Indicates the first The x and y coordinates of the center point of each object in the image coordinate system and Indicates the first Rotate the width and height of the target bounding box of an object. Indicates the first The orientation angle of the object's rotating target bounding box relative to the image coordinate system.

[0032] The directional object detector uses a pre-trained HOD-Net. When training the spatial anisotropic inference network, the network parameters of the directional object detector are fixed, and subsequent relation inference is performed only using its output object visual features, object rotation bounding boxes, and object category scores. The process is as follows: ; in, Represents a set of object characteristics for N objects. Dimensions representing object characteristics and relational characteristics; Represents the set of target bounding boxes for object rotation; Represents the initial set of object category scores. Indicates the number of object categories. By using rotated bounding boxes, target objects such as ships, aircraft, vehicles, and road facilities distributed in arbitrary directions in remote sensing images can be more accurately represented.

[0033] S2, Candidate Relation Extraction. Relation pairs are constructed for detected objects in the remote sensing image, grouping any two objects into candidate relation pairs based on subject and object. Subsequently, candidate relation pruning is performed on these pairs, removing those with a low probability of existence and retaining the set of candidate edges for relational predicate reasoning. .

[0034] For the candidate edge set Each candidate relationship in ,in Represents the main object, Represents the object, rotating the target box according to the subject object. Rotate the target box of the object Construct a directional union box that can enclose both simultaneously. and based on the directional joint frame Region feature pooling is performed on the feature map of the remote sensing image to obtain the relation features corresponding to the candidate relations. The process is as follows: ; in, This represents the set of relation features for M candidate relations. A set of directed union boxes representing candidate relationships. This represents the set of candidate edges. Candidate relation pruning can reduce the inference overhead for meaningless object pairs, but it may still retain weak correlations across distant functional regions. Therefore, the interaction strength of candidate relations is further adaptively modulated through the anisotropic elliptic influence propagation module.

[0035] S3, Object Feature Enhancement. The object enhancer consists of two layers of hybrid attention modules. It rotates the target bounding box set of the object. Spatial embedding is performed to obtain the spatial geometric features of the object; the initial object category scores are then processed. Semantic embedding is performed to obtain object category semantic features. The object visual features, object spatial geometric features, and object category semantic features are then input into the object augmenter to obtain an enhanced object feature set. and the updated set of object category scores .

[0036] ; in, As a spatial embedding function, it normalizes and encodes information such as the position, scale, shape, and orientation of the object's rotation target box, forming a nine-dimensional geometric description vector, which is then mapped through two fully connected layers to obtain the object's spatial geometric features; The semantic embedding function performs Softmax normalization on the initial object category scores and uses the resulting category probabilities to weight and combine the category semantic embedding matrix initialized by GloVe pre-trained word vectors to obtain object category semantic features. The object augmenter is used to update the object representation before relational inference, so that the object features simultaneously carry visual appearance, spatial location, and semantic category information.

[0037] S4, Spatial Influence Relationship Encoding. This involves encoding the original object features... Enhanced object features The data is then concatenated and combined with the updated object category semantic embeddings and relation features. Input the relation encoder to obtain the encoded object features. Relationship characteristics The process is as follows: ; in, This indicates feature splicing.

[0038] In this embodiment, the relation encoder consists of two identical coding blocks connected sequentially. Each coding block includes a hybrid attention module and an anisotropic elliptic influence propagation module. The hybrid attention module uses self-attention to model the global dependencies of the object's visual and semantic features respectively, and uses cross-attention to realize information interaction between the two types of features to obtain updated object features. The anisotropic elliptic influence propagation module constructs the object's central spatial influence region based on the object's rotated bounding box geometry, the object's predicted semantics, and the global context, and uses the corresponding sparse symmetric gating matrix to modulate the message aggregation from relation edges to object nodes in bidirectional message passing.

[0039] In the anisotropic elliptic influence propagation module, object nodes and relation edges perform bidirectional message passing, including message passing from object to relation and message passing from relation to object; 2 iterations are performed in each module. For the ... In the next iteration, the relation edge From the main object node and object node Update to meet the following requirements: ; in, It is a non-linear activation function. and These are the learnable projection matrices for the subject and object, respectively. Through the propagation process from this object to the relation, candidate relation edges can receive visual, spatial, and semantic information from their corresponding subject and object objects.

[0040] Message passing between objects is controlled by a sparse symmetric gating matrix. Modulation is performed. For the object... When an object is a subject, its outgoing edge relationships and its incoming edge relationships when it is an object participate in the object node update, satisfying the following: ; in, It is a sparse symmetric gated matrix. and These represent the sets of adjacent objects when object i is the subject and object, respectively. and These represent the relational message projection matrices when the object is the subject and object, respectively. The gating matrix G controls the intensity of relational messages flowing into the object node, causing the object to preferentially aggregate relevant relational contexts within its local region.

[0041] Sparse symmetric gated matrix Anisotropic Gaussian elliptical influence regions based on object-rotated bounding boxes are constructed on the candidate edge set, and the ellipse expansion scale and distance attenuation intensity are adaptively adjusted through object prediction semantics and global context. For objects... The rotated target bounding box is mapped to a two-dimensional Gaussian distribution. The mean of them First, construct the diagonal covariance matrix in the local coordinate system of the target box: ; in, and These represent the effective expansion factors of object i along the width and height directions of the rotated target box, respectively. This is a numerically stable term. The effective expansion factor is used to control the extent to which the object's influence region expands relative to the original rotated target box, allowing objects of different sizes, semantic categories, and orientations to have different spatial influence ranges.

[0042] Furthermore, based on the object's orientation angle Construct rotation matrix And rotate the covariance matrix in the local coordinate system to the global image coordinate system: ; ; Each object's rotating bounding box is mapped to an anisotropic Gaussian ellipse influence region consistent with its orientation and scale, which can better adapt to the characteristics of arbitrary directional distribution of targets and unclear boundaries of local functional regions in remote sensing images.

[0043] In a preferred embodiment, the effective spread factor and the distance attenuation intensity factor are jointly determined by a preset reference value and an instance adaptive offset: ; ; ; in, , and As a preset baseline quantity, , and For object The corresponding instance adaptive offset, Representation Object The distance attenuation intensity factor. Indicates based on The bounded mapping function constructed by the function is used to restrict the effective expansion factor to a preset range of values; The function is used to perform a positive mapping with a lower bound on the distance attenuation intensity.

[0044] The instance adaptive offset is generated by object geometric embedding, object semantic features, self-attention layer, and fully connected layer. For objects... First, the rotated target box is encoded into a nine-dimensional geometric description vector, and then the object's geometric embedding is obtained through layer normalization and two fully connected nonlinear mapping layers. The updated object category scores are normalized, and the obtained category probabilities are used to weight and combine the category semantic embedding matrix initialized by GloVe pre-trained word vectors to obtain the object semantic features. The object's geometric embedding and semantic features are concatenated and projected into a unified hidden space. Then, through a self-attention layer and a fully connected layer, the output is the instance adaptive offset corresponding to the width expansion factor, height expansion factor, and distance decay intensity factor. The process is as follows: ; ; ; in, This represents the category semantic embedding matrix initialized by GloVe pre-trained word vectors. Indicates the self-attention layer. This represents the fully connected layer used to generate adaptive offsets for three instances. Through the self-attention layer, each object can perceive the overall scale, object density, and contextual distribution within the image, thereby enabling its spatial influence region to adapt to differences in imaging scale and scene density across different remote sensing images.

[0045] For candidate relations The covariance matrices of the two objects are added together to obtain the joint covariance matrix, and the Mahalanobis distance between the centers of the two objects in the elliptic metric space is calculated: ; ; in, This represents the distance scale adjustment factor. This represents the joint covariance matrix.

[0046] Furthermore, through objects and object The distance attenuation intensity factor is used to calculate the symmetric attenuation term: ; according to and Obtain energy term and through Smooth truncation of functions: ; in, The truncation hyperparameter is set to 40 to improve numerical stability. Further... Mapped to gate weights: ; Gating weights The gating weight decreases as the elliptic metric distance increases. A larger gating weight corresponds to two objects located within each other's spatial influence regions and having a strong potential correlation in orientation, scale, and semantics; a smaller gating weight corresponds to two objects located far apart or in different functional regions. The gating weight on candidate edges... Fill the matrix The corresponding positions are determined, and the matrix is ​​symmetric and its diagonal elements are set to zero, thus obtaining the sparse symmetric gated matrix. .

[0047] It should be noted that the candidate relationship Maintaining the directionality from subject to object, candidate relationship It can correspond to different relational predicates; the sparse symmetric gating matrix is ​​only used to represent the spatial influence strength between object pairs, satisfying And modulate the directed relation edges respectively. and The message is propagated to the object node without changing the directionality of the relation predicate.

[0048] S5, decoding of relative spatial configuration relationships. For example... Figure 3 As shown, the object features output by the relation encoder Relationship characteristics Input relative space to configure the attention module. For candidate relations Rotate the target bounding box of the main object respectively. Rotating target box of object and candidate relationship-oriented joint frame Spatial embedding is performed to obtain the main geometric embedding. Object geometric embedding Relational geometric embedding .

[0049] ; ; ; Based on the differences between relational geometric embedding and subject geometric embedding and object geometric embedding, subject relative geometric representation and object relative geometric representation are constructed: ; ; in, This represents the relative geometric representation of the subject object with respect to the directed union box. This represents the relative geometric representation of an object with respect to an oriented union box. This relative geometric representation can describe the differences in position, orientation, and scale between the subject and object objects within the same candidate relation region.

[0050] Relative spatial configuration attention module with candidate relation features Generate query vectors to correspond to the features of the main object. and characteristics of the object Generate key vectors and value vectors: ; ; ; in, , and Let represent the learnable projection matrices for query, key, and value, respectively. Incorporate relative geometric representations into the attention matching process and calculate geometrically modulated attention weights: ; in, Indicates candidate relationship medium object The corresponding geometric modulation attention weights. Compared to attention calculations that rely solely on visual feature similarity, these attention weights simultaneously consider relation features, object features, and the geometric differences between objects and the relation joint regions, enabling the model to better distinguish relation predicates governed by spatial configuration.

[0051] Spatial augmentation messages of the subject and object objects are aggregated based on geometric modulation attention weights: ; in, This indicates element-wise multiplication.

[0052] Further updates to relational features are achieved through fully connected layers and residual connections: ; In one alternative implementation, the following settings are provided: A layer-wise spatial configuration attention module is used, where each layer receives the same initial relation features and generates spatial configuration-enhanced relation features. The outputs of each layer are averaged to obtain the final relation representation; preferably... .

[0053] ; By averaging the attention modules with multi-layer relative spatial configurations, we can stably utilize the spatially enhanced relation representations formed by different decoding layers, thereby reducing the impact of single-layer attention fluctuations on relation predicate prediction results.

[0054] S6, Scene Graph Generation. After obtaining the final relation representation, the subject object features and object object features are concatenated to obtain object pair features. These object pair features are then projected and fused element-wise with the final relation representation, and input into a classifier to obtain candidate relations. Relational predicate category score: ; Based on object category score Rotate the target box of the object Relational predicate category score The output includes a set of relational results containing the subject object and its rotated bounding box, relational predicate, object object and its rotated bounding box, and corresponding confidence scores. This set is structured as a remote sensing image scene graph in the form of triples: <subject object, relational predicate, object object>. The remote sensing image scene graph describes the spatial and semantic relationships between objects in a structured form of <subject object, relational predicate, object object>.

[0055] During the training phase, the method jointly optimizes the object classification loss and the relation predicate classification loss, and the overall loss function satisfies: ; Among them, object classification loss Used to constrain the updated object category scores, relational predicate classification loss. The predicate category score is used to constrain candidate relations. Both can be lost using cross-entropy loss, satisfying the following conditions respectively: ; ; in, Representation Object The true category label, Indicates candidate relationship The model obtains true predicate labels. Through joint training, the model can simultaneously optimize object category recognition and relational predicate inference, enabling the output remote sensing image scene graph to have good predictive consistency at both the object node and relational edge levels.

[0056] Example 2: like Figure 2 As shown, to verify the effectiveness of the method of the present invention, this embodiment conducts experiments on the STAR remote sensing image scene map generation dataset, and further verifies it on the AUG unmanned aerial vehicle remote sensing image scene map generation dataset.

[0057] (1) Dataset and task settings: The STAR dataset contains over 1200 large-scale remote sensing scene images, ranging in size from 512×768 to 27860×31096, covering scenes such as transportation, energy, and daily activities. The dataset includes 48 object categories, 58 relation categories, over 210,000 object annotations, and 400,000 relation triplet annotations. Following the publicly available data partitioning method, the STAR dataset is divided into training, validation, and test sets, with proportions of 60%, 20%, and 20%, respectively.

[0058] To verify the adaptability of this invention to different remote sensing image scenes, further verification was conducted on the AUG dataset. The AUG dataset contains 400 UAV top-down images with a resolution of 6000×4000, including 25594 object annotations and 16970 relation triplet annotations; of which 360 images were used for training and 38 images were used for testing.

[0059] The validation was performed on two datasets for predicate classification, scene graph classification, and scene graph detection tasks, respectively. Specifically, the predicate classification task predicts the relational predicate between object pairs given a rotated bounding box of a real object and the object category; the scene graph classification task predicts both the object category and the relational predicate given a rotated bounding box of a real object; and the scene graph detection task uses only remote sensing images as input, first performing oriented object detection, and then predicting the object category and the relational predicate.

[0060] (2) Experimental environment and parameter settings: HOD-Net was used as a pre-trained directed object detector on the STAR dataset, and a candidate relation pruning module was used to filter object pairs during the inference phase. During training, the parameters of the pre-trained directed object detector were fixed, and only the object augmenter, relation encoder, relation decoder, and relation classifier were trained.

[0061] For the predicate classification and scene graph classification tasks in the STAR dataset, the batch size was set to 4, the initial learning rate was set to 0.001, and the number of training iterations was set to 10,000. For the scene graph detection task, the batch size was set to 2, the initial learning rate was set to 0.001, and the number of training iterations was set to 5,000. The optimizer used was a stochastic gradient descent optimizer with a momentum of 0.9, and a warm-up strategy was used for the first 500 iterations.

[0062] Distance scale adjustment factor in the anisotropic elliptic influence propagation module Set to 0.5 for smooth truncation parameters. Set to 40, numerical stability term The value is set to 0.00000001. The baseline spread factor in both the width and height directions is set to 2.0, and the baseline distance attenuation strength is set to 1.5. The relation encoder consists of two sequentially connected coding blocks, each including a hybrid attention module and an anisotropic elliptic influence propagation module; each anisotropic elliptic influence propagation module performs two bidirectional message passing operations. The relation decoder consists of four layers of relative spatial configuration attention modules, each receiving the same initial relation features, and averaging the outputs of each layer to obtain the final relation representation.

[0063] On the AUG dataset, Faster R-CNN with a VGG16 backbone network was used as the pre-trained object detector. Since the object detector outputs horizontal bounding boxes, each horizontal bounding box was converted into a rotated bounding box representation with an orientation angle of 0 before being input into a spatial anisotropic inference network for object relationship inference. The training iterations for the predicate classification task, scene graph classification task, and scene graph detection task were all set to 40,000, the batch size was set to 2, and the initial learning rate was set to 0.005. The optimizer used was a stochastic gradient descent optimizer with a momentum of 0.9, and a warm-up strategy was used for the first 500 iterations.

[0064] (3) Evaluation indicators: For the STAR dataset, since there may be multiple semantically consistent relational predicates between the same subject and object in remote sensing images, the multi-label recall rate MR@K and the average multi-label recall rate mMR@K are used as evaluation metrics, where K is 1500 and 2000 respectively.

[0065] MR@K is used to measure the proportion of correctly retrieved object relations among the top K prediction results; for the same object pair, a prediction predicate is considered correct if it matches any of the ground truth labeled predicates. Since the sample size of different relation predicates exhibits a long-tail distribution, using only MR@K is easily affected by high-frequency predicates. Therefore, this embodiment also uses mMR@K, which is the average of the multi-label recall rates calculated for each relation predicate category, to evaluate the model's balanced recognition capability across different relation categories.

[0066] For the AUG dataset, recall R@K and mean recall mR@K are used as evaluation metrics, where K is set to 50 and 150, respectively. R@K measures the proportion of correct relations among the top K predicted relations to all true relations; mR@K, by averaging the recall of each relation predicate category, weakens the dominant role of high-frequency relation predicates in the overall evaluation results and is used to evaluate the balanced recognition capability of this invention for different relation categories in UAV top-down image scenes.

[0067] (4) Overall experimental results: As shown in Table 1, the model of this invention (SARNet) achieved high relation prediction performance in predicate classification, scene graph classification, and scene graph detection tasks on the STAR dataset. In the predicate classification task, the MR@1500 and MR@2000 of the model of this invention were 69.06% and 70.77%, respectively; in the scene graph classification task, they were 54.41% and 55.65%, respectively. In the scene graph detection task, the MR@1500, MR@2000, mMR@1500, and mMR@2000 of the model of this invention were 33.52%, 34.56%, 14.05%, and 14.61%, respectively. After semantic filtering, the above indicators of the model of this invention in the scene graph detection task further reached 35.73%, 36.64%, 14.88%, and 15.37%. The results show that the model of this invention can perform object relation reasoning in remote sensing scenes with dense targets and large spatial range.

[0068] As shown in Table 2, the model of this invention achieved good overall performance in the scene graph classification task of the AUG dataset, and achieved 22.20% R@150 and 8.02% mR@150 in the scene graph detection task, indicating that the method of this invention has a certain adaptability to UAV top-view image scenes.

[0069] Table 1: Performance comparison and evaluation on the STAR dataset;

[0070] Table 2: Performance comparison and evaluation on the AUG dataset;

[0071] (5) Ablation experiment: As shown in Table 3, the model's relation prediction performance on all three tasks was improved after introducing either the Anisotropic Elliptical Influence Propagation Module (AEIP) or the Relative Spatial Configuration Attention Module (RSCA). Specifically, when only AEIP was introduced, the mMR@1500 and mMR@2000 for the predicate classification task increased from 33.25% and 34.92% to 39.55% and 40.58%, respectively; when only RSCA was introduced, the mMR@1500 and mMR@2000 for the scene graph classification task increased from 24.68% and 25.53% to 27.24% and 28.20%, respectively.

[0072] Table 3: Comparative Evaluation of Ablation Experiment Performance;

[0073] With the introduction of both AEIP and RSCA, the complete SARNet achieved high overall performance in predicate classification and scene graph classification tasks. In the scene graph detection task, MR@1500, MR@2000, mMR@1500, and mMR@2000 achieved 35.73%, 36.64%, 14.88%, and 15.37%, respectively. These results validate the effectiveness and complementarity of the AEIP and RSCA modules.

Claims

1. A method for generating scene graphs from remotely sensed images based on spatial anisotropy reasoning, characterized in that, Includes the following steps: S1, Remote sensing image target detection: Acquiring the remote sensing image to be processed. The remote sensing image is input into a pre-trained directional target detector to obtain object features. Rotate the target box of the object and initial object category score ; S2, Candidate Relation Extraction: The detected objects are combined with relation pairs and pruned to form a candidate edge set. For the candidate edge set Each of the items in the main object and object The candidate relationships are constructed, generating a directed union box that encloses the subject object and the object object. Relationship features are obtained from the feature map of the entire image by pooling based on the directed joint bounding box. Thus, the set of relational features is obtained. and relation-oriented union frame set ; S3, Object Feature Enhancement: Enhances object features Rotate the target box from the object The obtained spatial embedding and the initial object category score The obtained semantic embedding input object enhancer obtains enhanced object features. and updated object category scores ; S4, Spatial Influence Relation Encoding: Object features, enhanced object features, semantic embeddings generated from updated object category scores, and relation features are input into the relation encoder to obtain encoded object features. Relationship characteristics ; S5, Decoding relative spatial configuration relationships: Decoding the encoded object features Relationship characteristics Input the relative spatial configuration attention module to obtain the spatial configuration-enhanced relation representation; S6, Scene Graph Generation: The features of the subject object, the features of the object, and the spatial configuration-enhanced relation representation are fused together to output the predicate category scores of the candidate relations. The object category scores are then combined to generate a remote sensing image scene graph containing the subject object, relation predicates, and object objects.

2. The method for generating scene graphs from remotely sensed images based on spatial anisotropy reasoning according to claim 1, characterized in that, S1 specifically includes: Acquire remote sensing images to be processed The remote sensing image is input into a pre-trained directional target detector, which then locates, classifies, and extracts features from the target objects in the remote sensing image. The rotated bounding boxes output by the directional target detector represent target objects distributed in arbitrary directions within the remote sensing image. Each rotated target box in the diagram is represented as: ； in, For the first Rotate the target bounding box of an object; and The first The x and y coordinates of the center point of the object's rotation target bounding box in the image coordinate system; and The first Rotate the width and height of the target bounding box of an object; For the first The orientation angle of the object's rotating target bounding box relative to the image coordinate system. The detection and feature extraction process of the directional target detector is as follows: ； in, This represents the set of all object visual features extracted by the directional target detector; This represents the set of category scores obtained by the directional target detector for the initial prediction of each object category.

3. The method for generating scene graphs from remotely sensed images based on spatial anisotropy reasoning according to claim 1, characterized in that, S2 specifically includes: Relationship pairs are constructed for detected objects in remote sensing images, and any two objects are grouped into candidate relationship pairs according to the form of subject and object. Subsequently, candidate relationship pruning is performed on the candidate relationship pairs to remove relationship pairs with low probability of existence, and retain the set of candidate edges used for relational predicate reasoning. . For the candidate edge set Each candidate relationship in ,in Represents the main object, Represents the object, rotating the target box according to the subject object. Rotate the target box of the object Construct a directional union box that can enclose both simultaneously. and based on the directional joint frame Region feature pooling is performed on the feature map of the remote sensing image to obtain the relation features corresponding to the candidate relations. . The candidate relation extraction process satisfies: ； in, A set of directed union boxes for candidate relations; This is the set of relation features for candidate relations.

4. The method for generating scene graphs from remotely sensed images based on spatial anisotropy reasoning according to claim 1, characterized in that, In S3, the specific methods for enhancing object features are as follows: Rotate the target box set of the object Perform spatial embedding and assign initial object category scores Perform semantic embedding to incorporate object features The input object enhancers of both are merged and updated to satisfy the following: ； in, This represents the set of enhanced object features after being updated by the object enhancer; This represents the final object category score after being updated by the object enhancer; As a spatial embedding function, the normalized width, normalized height, normalized center coordinates, aspect ratio, orientation angle, scale ratio relative to the image diagonal, scale ratio relative to the image area, and center horizontal and vertical coordinate ratio of the object's rotating target box are encoded and mapped through a multilayer perceptron to obtain spatial geometric features. The semantic embedding function normalizes the initial object category scores into category probabilities, and then performs a weighted combination of the category semantic embedding matrix initialized by GloVe pre-trained word vectors according to the category probabilities to obtain the object category semantic features.

5. The method for generating scene graphs from remotely sensed images based on spatial anisotropy reasoning according to claim 1, characterized in that, In step S4, the relation encoder first captures global interactions between objects through hybrid attention, then constructs the spatial influence region of the object center through the anisotropic elliptical influence propagation module, and modulates the message aggregation from relation edges to object nodes in bidirectional message passing using a sparse symmetric gating matrix generated by the geometric embedding of the object rotation target box and the semantic embedding of the object, thus obtaining the encoded object features. Relationship characteristics The specific methods for spatial influence relationship coding are as follows: The anisotropic ellipse influence propagation module performs bidirectional message passing between object nodes and relation edges; in the... In this iteration, the relation edges are updated by the subject object node and the object object node, satisfying: ； in, It is a non-linear activation function. and These are the learnable projection matrices for the subject and object objects, respectively. The object node is updated with weighted relation edges associated with it when it is both the subject and the object, satisfying the following: ； in, It is a sparse symmetric gated matrix. and Representing objects respectively When acting as a subject and an object, it represents the set of adjacent objects. and These represent the relational message projection matrices when the objects are the subject and object, respectively.

6. The method for generating scene graphs from remotely sensed images based on spatial anisotropy reasoning according to claim 5, characterized in that, The construction of the sparse symmetric gating matrix G includes modeling the anisotropic elliptical influence region, and the specific steps are as follows: Map the rotation bounding box of each object to a two-dimensional Gaussian distribution. ,in Construct a diagonal covariance matrix in the local coordinate system of the target bounding box based on the object's width, height, and effective expansion factor, satisfying: ； in, and Representing objects respectively Effective expansion factor along the width and height directions; and Representing objects respectively The width and height of the rotated target bounding box; It is a numerically stable term.

7. The method for generating scene graphs from remotely sensed images based on spatial anisotropy reasoning according to claim 5, characterized in that, The sparse symmetric gate matrix G is generated based on the elliptic metric distance between the subject and object objects in the candidate relation, specifically: For candidate relations The covariance matrices of the two objects are added together to obtain the joint covariance matrix, and the elliptic metric distance between the centers of the two objects is calculated, satisfying: ； ； in, Denotes the joint covariance matrix. Indicates candidate relationship The corresponding elliptic metric distance; This represents the distance scale adjustment factor; and These represent the coordinates of the center point of the subject and the object, respectively.

8. The method for generating scene graphs from remotely sensed images based on spatial anisotropy reasoning according to claim 1, characterized in that, In S5, the relation features of each candidate relation are used as the query, and the object features of the corresponding subject and object objects are used as the key and value. The relative spatial configuration between the subject object, object object, and their directed union boxes is embedded in the attention calculation process to obtain the relation representation with enhanced spatial configuration. The specific implementation of the relative spatial configuration attention is as follows: Spatial embedding is performed on the rotating target box of the subject object, the rotating target box of the object object, and the orientation union box to obtain subject geometric embedding, object geometric embedding, and relation geometric embedding; Based on the differences between the relational geometric embedding and the subject geometric embedding and the object geometric embedding, a relative geometric representation of the subject and an object are constructed respectively. The relative space configuration attention module generates a query using candidate relation features, generates keys and values ​​using corresponding subject and object features, and incorporates the relative geometric representation into the attention matching process. The geometric modulation attention weights satisfy the following: ； in, Indicates candidate relationship The geometric modulation attention weights corresponding to object n in the middle. ; This represents a query vector generated from relational features; Represents a key vector generated from the features of a subject object or an object object; Representation Object Relative geometric representation of the directed union box.

9. The method for generating scene graphs from remotely sensed images based on spatial anisotropy reasoning according to claim 1, characterized in that, In S5, the specific method for decoding the relative spatial configuration relationship is as follows: The relation features are updated through fully connected layers and residual connections; when stacking multiple layers of relative space to configure attention modules, the relation features output by each layer are averaged to obtain the final relation representation, which is used for subsequent relation predicate classification.

10. The method for generating scene graphs from remotely sensed images based on spatial anisotropy reasoning according to claim 1, characterized in that, During the training phase, the object classification loss and the relation predicate classification loss are jointly optimized, as follows: The object classification loss is used to constrain the updated object category scores, and the relation predicate classification loss is used to constrain the predicate category scores of candidate relations. Together, they are used to optimize the remote sensing image scene graph generation results.

Images