A remote sensing image spatial relationship identification method, medium and device

Abstract

The application discloses a remote sensing image space relation recognition method, medium and equipment, relates to the technical field of space relation recognition of computer vision; the remote sensing image space relation recognition method mainly comprises the following steps: labeling remote sensing objects by using space relations, preprocessing the labeled data set, obtaining the characteristics and representation of the remote sensing objects by using a target detection network, training a linear neural network to obtain a trained linear neural network, and obtaining a space relation prediction result by using the target detection network and the trained linear neural network. The remote sensing image space relation recognition method, medium and equipment provided by the application can simultaneously learn three kinds of space relations, more comprehensively and accurately recognize complex space relations, and meet the multi-azimuth semantic understanding requirement of space relations.

Description

Technical Field

[0001] This invention relates to the field of spatial relationship recognition technology in computer vision, and more specifically, to a method, medium, and device for spatial relationship recognition in remote sensing images. Background Technology

[0002] Spatial relationships are crucial for understanding remote sensing images, as much of the information they provide depends on the spatial interactions between geographic objects. With the development of deep learning, significant progress has been made in many semantic understanding tasks, such as scene classification, object detection, semantic segmentation, and image caption generation. However, research on spatial relationships between objects has not received sufficient attention. Modeling spatial relationships in remote sensing images can not only improve the understanding of hidden information within the images but also help achieve better performance in the aforementioned tasks.

[0003] In Geographic Information Systems (GIS) and spatial analysis, spatial relationships are used to analyze and simulate the interactions between different features or entities in space. Most classifications of spatial relationships categorize them into distance relationships, directional relationships, and topological relationships. Among these, topological relationships are the most fundamental spatial relationships in the semantic understanding of remote sensing images, and they do not change with rotation or scale; they are qualitative relationships. Direction relationships also primarily appear qualitatively, described using directional terms such as east, south, west, north, southeast, northeast, southwest, and northwest. Distance relationships can be quantitative, involving the calculation of Euclidean distance between surfaces, or qualitative, described by proximity.

[0004] In computational vision, visual relationships primarily encompass action and spatial relationships, and specialized visual tasks exist to study object relationships. These tasks mainly output object relationship triples (subject, predicate, object), such as visual relationship detection and scene graph generation. In remote sensing imagery, relationships between objects are primarily manifested as spatial relationships. Existing research on spatial relationships in remote sensing images has been applied to various tasks, such as object detection, change detection, image description generation, and knowledge graph generation, mainly aiming to improve model performance by learning spatial relationships. Some studies focusing on spatial relationship recognition express spatial relationships using object relationship triples, but these studies, when describing the spatial relationships between a group of objects, are limited to only one type of relationship, ignoring the diversity of spatial relationships, the differences between different types of spatial relationships, and their inherent connections. Limiting a group of objects to only one spatial relationship makes it difficult to see the full picture of their actual spatial interactions. Three spatial relationships with progressively stronger constraints, when combined, can enrich semantic information and improve the level of remote sensing image understanding. For example, when describing the relationship between farmland and lakes, simply using the direction "north" provides very limited information. Adding topological terms like "distancing" and "close" distance can provide much more information about the image.

[0005] Existing spatial relationship models cannot learn complex spatial relationships. They suffer from the problem of using a single spatial relationship to describe complex spatial interactions. In other words, they cannot learn the diversity of spatial relationships and the connections between different spatial relationships, making it difficult for the models to be applied in practice to solve real-world problems and limiting the development of downstream tasks. Summary of the Invention

[0006] The purpose of this invention is to provide a method, medium, and device for identifying spatial relationships in remote sensing images, enabling more comprehensive and accurate identification of complex spatial relationships.

[0007] This invention provides a method for spatial relationship recognition in remote sensing images, comprising the following steps: S1: acquiring labeled remote sensing images and their remote sensing objects, and labeling the remote sensing objects using spatial relationships to obtain a labeled dataset; spatial relationships include topological spatial relationships, azimuth spatial relationships, and distance spatial relationships; S2: preprocessing the labeled dataset to obtain a training set and a test set; S3: using the training set and an object detection network to obtain the features of the remote sensing objects; S4: performing relative position calculation and stitching operations based on the features of the remote sensing objects to obtain the representation of the remote sensing objects; S5: constructing a linear neural network, and training the linear neural network using the representation of the remote sensing objects to obtain a trained linear neural network; S6: using the test set, an object detection network, and the trained linear neural network to obtain spatial relationship prediction results.

[0008] Furthermore, the topological spatial relationships of the aforementioned remote sensing image spatial relationship recognition method include: disjoint, connected, contained, and intersecting; the directional spatial relationships include: east, west, south, north, southeast, northeast, southwest, northwest, and center; and the distance spatial relationships include: very close, near, and far.

[0009] Furthermore, step S2 of the aforementioned remote sensing image spatial relationship recognition method specifically includes: performing mean subtraction, variance division, and scaling on each image in the labeled dataset, and dividing it according to a 7:3 ratio to obtain a training set and a test set.

[0010] Furthermore, the target detection network of the aforementioned remote sensing image spatial relationship recognition method is a pre-trained target detection network Faster-RCNN. The pre-trained target detection network Faster-RCNN includes an input layer, a pooling layer, a convolutional layer, and an output layer. The input layer is used to receive the original image data. The pooling layer is used to reduce the spatial resolution of the feature map while keeping the key features unchanged. The convolutional layer is used to perform convolution operations on the input image and extract local features at different scales and directions. The output layer is used to output the class probability distribution and bounding box coordinates.

[0011] Furthermore, the features of the remote sensing objects in the aforementioned remote sensing image spatial relationship recognition method include object classification probability, object feature map, and object bounding box parameters; the object bounding box parameters are used to represent the coordinates and size of the center point of the bounding box of the remote sensing object.

[0012] Furthermore, step S4 of the aforementioned remote sensing image spatial relationship recognition method specifically includes: S41: grouping remote sensing objects into pairs to obtain spatial relationship object groups, where each spatial relationship object group includes a subject object and an object object; obtaining relative position parameters based on the characteristics of the remote sensing objects, as shown in the formula:

[0013]

[0014]

[0015] Among them, bbox s The relative position parameter of the subject, x s x-coordinate of the center point of the subject's bounding box o w is the x-coordinate of the center point of the bounding box of the object. o y is the width of the bounding box of the object. s The y-coordinate of the center point of the subject's bounding box. o h is the ordinate of the center point of the bounding box of the object. o bbox is the height of the bounding box of the object. oS42: Based on the characteristics and relative position parameters of the remote sensing object, the representation of the remote sensing object is obtained. The representation of the remote sensing object includes the representation of the subject object and the representation of the object object, as shown in the formula:

[0016] rep s =concat(cls) s bbox s ,v s )

[0017] rep o =concat(cls) o bbox o ,v o )

[0018] Among them, rep s The representation of the subject object, concat() is the concatenation function, cls s Classification probability of the subject object, bbox s The relative position parameter of the subject, v s The object feature map of the subject object; rep o cls represents the object of the verb. o The classification probability of the object, bbox o v is the relative position parameter of the object. o This is the object feature map of the object.

[0019] Further, step S5 of the aforementioned remote sensing image spatial relationship recognition method includes: S51: Constructing a linear neural network, the linear neural network including a topology relationship compression network, a direction relationship compression network, a distance relationship compression network, a fully connected layer for object representation topology projection transformation, a fully connected layer for subject representation topology projection transformation, a fully connected layer for object representation direction projection transformation, a fully connected layer for subject representation direction projection transformation, a fully connected layer for object representation distance projection transformation, and a fully connected layer for subject representation distance projection transformation; the topology relationship compression network is used to compress the dimension of topology relationships, the direction relationship compression network is used to compress the dimension of direction relationships, the distance relationship compression network is used to compress the dimension of distance relationships, and the object relationship compression network is used to compress the dimension of distance relationships. The image representation topological projection transformation fully connected layer is used to transform the representation of the object to the topological feature space. The subject representation topological projection transformation fully connected layer is used to transform the representation of the subject to the topological feature space. The object representation direction projection transformation fully connected layer is used to transform the representation of the object to the direction feature space. The subject representation direction projection transformation fully connected layer is used to transform the representation of the subject to the direction feature space. The object representation distance projection transformation fully connected layer is used to transform the representation of the object to the distance feature space. The subject representation distance projection transformation fully connected layer is used to transform the representation of the subject to the distance feature space. S52: Based on the representation of the remote sensing object, a linear neural network is used to obtain the predicted spatial relationship, as shown in the formula:

[0020]

[0021]

[0022]

[0023] Where, r tpe For the predicted topological relationships, W o-tp The projection transformation parameters, rep, are used to transform the representation of the object to the topological feature space. o W represents the object of the verb. s-tp The projection transformation parameters, rep, are used to transform the representation of the subject object to the topological feature space. s The representation of the subject object, r dire For the predicted directional relationship, W o-dir W is the transformation parameter for projecting the representation of the object to the directional feature space. s-dir The transformation parameters r are projection transformation parameters used to transform the representation of the subject object into the directional feature space. dise For the predicted distance relationship, W o-dis W is the transformation parameter for projecting the representation of the object to the distance feature space. s-disThe transformation parameters are used to convert the representation of the subject object to the projection transformation space; S53: Using a linear neural network, the predicted spatial relationship is compressed, a loss function is constructed, and the linear neural network is trained using the loss function to obtain a trained linear neural network, as shown in the formula:

[0024] Compress tp =Linear(d1,N tp )

[0025] Compress dir =Linear(d2,N dir )

[0026] Compress dis =Linear(d3,N dis )

[0027]

[0028]

[0029]

[0030] Among them, Compress tp For topological compression networks, "Linear" indicates that the network has a linear structure, and "Compress" indicates that the network has a linear structure. dir Compress the network for directional relationships. dis For distance relationship compression networks, d1 is the dimension of the predicted topological relationships, d2 is the dimension of the predicted directional relationships, d3 is the dimension of the predicted distance relationships, and N is the dimension of the predicted distance relationships. tp N represents the number of categories of topological relationships. dir N represents the number of categories of directional relationships. dis represents the number of categories of distance relationships, log represents the logarithm, and softmax() is the normalization function. For the predicted topological relationships,

[0031] For the predicted directional relationship, For the predicted distance relationship, Calculate the loss function for topological spatial relationships. The loss function calculated for spatial relationships of directions. The loss function R is calculated for distance relationships. tp The training set contains all object topological relation triples, where s represents the subject object and o represents the object object. R represents the true value of the topological distribution. dir To provide all object orientation relation triples existing in the training set. R represents the true value of the directional relationship distribution. dis To provide all object distance triples in the training set, This represents the true value of the distance relationship distribution.

[0032] Further, step S6 of the above-mentioned remote sensing image spatial relationship recognition method includes: S61: obtaining the first feature of the remote sensing object using a target detection network based on the test set; S62: obtaining the first representation of the remote sensing object based on the first feature of the remote sensing object; and obtaining the first predicted spatial relationship based on the first representation of the remote sensing object using a trained linear neural network, as shown in the formula:

[0033]

[0034] Where A represents the first predicted spatial relationship. Let x be the topological relation of the first predicted spatial relationship, and z be the object feature map. e For object classification and object bounding box prediction, The first predicted spatial relationship is the directional relationship. The distance relationship is the first predicted spatial relationship; S63: Set the object feature map in the first feature of the remote sensing object to the average value of the feature maps of all images in the test set input at one time to obtain the second feature of the remote sensing object; Based on the second feature of the remote sensing object, obtain the second representation of the remote sensing object; Based on the second representation of the remote sensing object, use the trained linear neural network to obtain the second predicted spatial relationship, as shown in the formula:

[0035]

[0036] Where B represents the second predicted spatial relationship. This refers to the topological relationship of the second predicted spatial relationship. z is the average of the feature maps of all images in the test set as a single input. e For object classification and object bounding box prediction, The second predicted spatial relationship is the directional relationship. S64: Based on the first and second predicted spatial relationships, obtain the probability distribution of each spatial relationship, as shown in the formula:

[0037]

[0038]

[0039]

[0040]

[0041] Where, r tp + r is the difference in topological relationship between the first predicted spatial relationship and the second predicted spatial relationship. dir + r is the difference between the directional relationships of the first predicted spatial relationship and the second predicted spatial relationship. dis + Let A be the difference between the distances between the first predicted spatial relationship and the second predicted spatial relationship, and B be the distance between them. Let x be the topological relation of the first predicted spatial relationship, and z be the object feature map. e For object classification and object bounding box prediction, The first predicted spatial relationship is the directional relationship. The distance relationship is the first predicted spatial relationship. This refers to the topological relationship of the second predicted spatial relationship. z is the average of the feature maps of all images in the test set as a single input. e For object classification and object bounding box prediction, The second prediction is the directional relationship of spatial relationships. The distance relationship is the second predicted spatial relationship; Let be the probability distribution of the topological relationship. Let be the probability distribution of the directional relationship. The distance relationship is the probability distribution, softmax() is the normalization function, and compress() is the normalization function. tp For topological compression networks, "Linear" indicates that the network has a linear structure, and "Compress" indicates that the network has a linear structure. dir Compress the network for directional relationships. dis S65: Select the relationship category with the highest probability distribution for each spatial relationship as the final spatial relationship prediction result.

[0042] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described remote sensing image spatial relationship recognition method.

[0043] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the remote sensing image spatial relationship recognition method described above.

[0044] The remote sensing image spatial relationship recognition method, medium, and device provided by this invention have the following beneficial effects:

[0045] This method combines topological, directional, and distance spatial relationships to annotate spatial relationships in remote sensing images using the format "topology, direction, distance." It constructs object representations comprising object feature maps, object classifications, and object bounding boxes, projecting these representations onto three different low-dimensional spaces. The three spatial relationships are modeled as vector subtraction between object representations in the three spaces, and the computational process is optimized by leveraging the inherent connections and constraints between these spatial relationships. During the inference phase, the differences between normal inputs and inputs with fixed object feature maps in different spaces are calculated to obtain the three spatial relationships, which are then combined to express complex spatial interactions between objects. This method possesses the ability to learn three spatial relationships simultaneously and utilizes the inherent connections between them to achieve more comprehensive and accurate identification of complex spatial relationships, meeting the needs of multi-faceted semantic understanding of spatial relationships and demonstrating promising application prospects. Attached Figure Description

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0047] Figure 1 A flowchart of the remote sensing image spatial relationship recognition method provided by the present invention;

[0048] Figure 2 A schematic diagram of the framework of the remote sensing image spatial relationship recognition method based on deep learning provided by the present invention;

[0049] Figure 3 This invention provides a schematic diagram of object projection into relation space, using topological space as an example.

[0050] Figure 4 This invention provides a schematic diagram of the reasoning stage analysis.

[0051] Figure 5 A flowchart for remote sensing image spatial relationship recognition based on deep learning provided by this invention;

[0052] Figure 6 This is a diagram illustrating the composition of a computer device provided by the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and technical effects of the present invention clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0054] In this embodiment, the flowchart of the remote sensing image spatial relationship recognition method is as follows: Figure 1As shown, the remote sensing image spatial relationship recognition method specifically includes the following steps: S1: Obtain labeled remote sensing images and their remote sensing objects, and label the remote sensing objects using spatial relationships to obtain a labeled dataset; spatial relationships include topological spatial relationships, azimuth spatial relationships, and distance spatial relationships; S2: Preprocess the labeled dataset to obtain a training set and a test set; S3: Based on the training set, use an object detection network to obtain the features of the remote sensing objects;

[0055] S4: Based on the characteristics of the remote sensing object, perform relative position calculation and stitching operations to obtain the representation of the remote sensing object; S5: Construct a linear neural network, use the representation of the remote sensing object to train the linear neural network, and obtain a trained linear neural network; S6: Based on the test set, use the target detection network and the trained linear neural network to obtain the spatial relationship prediction results.

[0056] Specifically, the topological spatial relationships of the aforementioned remote sensing image spatial relationship recognition method include: disjoint, connected, contained, and intersecting; the directional spatial relationships include: east, west, south, north, southeast, northeast, southwest, northwest, and center; and the distance spatial relationships include: very close, near, and far.

[0057] Specifically, step S2 of the above-mentioned remote sensing image spatial relationship recognition method includes: performing mean subtraction, variance division, and scaling on each image in the labeled dataset, and dividing it into a training set and a test set according to a 7:3 ratio.

[0058] Specifically, the target detection network of the aforementioned remote sensing image spatial relationship recognition method is a pre-trained target detection network Faster-RCNN. The pre-trained target detection network Faster-RCNN includes an input layer, a pooling layer, a convolutional layer, and an output layer. The input layer is used to receive the original image data. The pooling layer is used to reduce the spatial resolution of the feature map while keeping the key features unchanged. The convolutional layer is used to perform convolution operations on the input image and extract local features at different scales and directions. The output layer is used to output the class probability distribution and bounding box coordinates.

[0059] Specifically, the features of the remote sensing objects in the aforementioned remote sensing image spatial relationship recognition method include object classification probability, object feature map, and object bounding box parameters; the object bounding box parameters are used to represent the center point coordinates and size of the bounding box of the remote sensing object.

[0060] Specifically, step S4 of the above-mentioned method for spatial relationship recognition in remote sensing images includes: S41: Grouping remote sensing objects into pairs to obtain spatial relationship object groups, which include subject objects and object objects; obtaining relative position parameters based on the characteristics of the remote sensing objects, as shown in the formula:

[0061]

[0062]

[0063] Among them, bbox s The relative position parameter of the subject, x s x-coordinate of the center point of the subject's bounding box o w is the x-coordinate of the center point of the bounding box of the object. o y is the width of the bounding box of the object. s The y-coordinate of the center point of the subject's bounding box. o h is the ordinate of the center point of the bounding box of the object. o bbox is the height of the bounding box of the object. o S42: Based on the characteristics and relative position parameters of the remote sensing object, the representation of the remote sensing object is obtained. The representation of the remote sensing object includes the representation of the subject object and the representation of the object object, as shown in the formula:

[0064] rep s =concat(cls) s bbox s ,v s )

[0065] rep o =concat(cls) o bbox o ,v o )

[0066] Among them, rep s The representation of the subject object, concat() is the concatenation function, cls s Classification probability of the subject object, bbox s The relative position parameter of the subject, v s The object feature map of the subject object; rep o cls represents the object of the verb. o The classification probability of the object, bbox o v is the relative position parameter of the object. o This is the object feature map of the object.

[0067] Specifically, step S5 of the aforementioned remote sensing image spatial relationship recognition method includes: S51: Constructing a linear neural network, which includes a topology relationship compression network, a direction relationship compression network, a distance relationship compression network, a fully connected layer for object representation topology projection transformation, a fully connected layer for subject representation topology projection transformation, a fully connected layer for object representation direction projection transformation, a fully connected layer for subject representation direction projection transformation, a fully connected layer for object representation distance projection transformation, and a fully connected layer for subject representation distance projection transformation; the topology relationship compression network is used to compress the dimension of topology relationships; the direction relationship compression network is used to compress the dimension of direction relationships; the distance relationship compression network is used to compress the dimension of distance relationships; and the object representation... A fully connected layer representing topological projection transformation is used to transform the representation of the object to the topological feature space. A fully connected layer representing topological projection transformation of the subject is used to transform the representation of the subject to the topological feature space. A fully connected layer representing orientation projection transformation of the object is used to transform the representation of the object to the orientation feature space. A fully connected layer representing orientation projection transformation of the subject is used to transform the representation of the subject to the orientation feature space. A fully connected layer representing distance projection transformation of the object is used to transform the representation of the object to the distance feature space. A fully connected layer representing distance projection transformation of the subject is used to transform the representation of the subject to the distance feature space. S52: Based on the representation of the remote sensing object, a linear neural network is used to obtain the predicted spatial relationship, as shown in the formula:

[0068]

[0069]

[0070]

[0071] Where, r tpe For the predicted topological relationships, W o-tp The projection transformation parameters, rep, are used to transform the representation of the object to the topological feature space. o W represents the object of the verb. s-tp The projection transformation parameters, rep, are used to transform the representation of the subject object to the topological feature space. s The representation of the subject object, r dire For the predicted directional relationship, W o-dir W is the transformation parameter for projecting the representation of the object to the directional feature space. s-dir The transformation parameters r are projection transformation parameters used to transform the representation of the subject object into the directional feature space. dise For the predicted distance relationship, W o-dis W is the transformation parameter for projecting the representation of the object to the distance feature space. s-disThe transformation parameters are used to convert the representation of the subject object to the projection transformation space; S53: Using a linear neural network, the predicted spatial relationship is compressed, a loss function is constructed, and the linear neural network is trained using the loss function to obtain a trained linear neural network, as shown in the formula:

[0072] Compress tp =Linear(d1,N tp )

[0073] Compress dir =Linear(d2,N dir )

[0074] Compress dis =Linear(d3,N dis )

[0075]

[0076]

[0077]

[0078] Among them, Compress tp For topological compression networks, "Linear" indicates that the network has a linear structure, and "Compress" indicates that the network has a linear structure. dir Compress the network for directional relationships. dis For distance relationship compression networks, d1 is the dimension of the predicted topological relationships, d2 is the dimension of the predicted directional relationships, d3 is the dimension of the predicted distance relationships, and N is the dimension of the predicted distance relationships. tp N represents the number of categories of topological relationships. dir N represents the number of categories of directional relationships. dis represents the number of categories of distance relationships, log represents the logarithm, and softmax() is the normalization function. For the predicted topological relationships,

[0079] For the predicted directional relationship, For the predicted distance relationship, Calculation of topological spatial relationships

[0080] loss function The loss function calculated for spatial relationships of directions. The loss function R is calculated for distance relationships. tp The training set contains all object topological relation triples, where s represents the subject object and o represents the object object. R represents the true value of the topological distribution. dirTo provide all object orientation relation triples existing in the training set. R represents the true value of the directional relationship distribution. dis To provide all object distance triples in the training set, This represents the true value of the distance relationship distribution.

[0081] Specifically, step S6 of the aforementioned remote sensing image spatial relationship recognition method includes: S61: obtaining the first feature of the remote sensing object using a target detection network based on a test set; S62: obtaining the first representation of the remote sensing object based on the first feature; and obtaining the first predicted spatial relationship based on the first representation of the remote sensing object using a trained linear neural network, as shown in the formula:

[0082]

[0083] Where A represents the first predicted spatial relationship. Let x be the topological relation of the first predicted spatial relationship, and z be the object feature map. e For object classification and object bounding box prediction, The first predicted spatial relationship is the directional relationship. The distance relationship is the first predicted spatial relationship; S63: Set the object feature map in the first feature of the remote sensing object to the average value of the feature maps of all images in the test set input at one time to obtain the second feature of the remote sensing object; Based on the second feature of the remote sensing object, obtain the second representation of the remote sensing object; Based on the second representation of the remote sensing object, use the trained linear neural network to obtain the second predicted spatial relationship, as shown in the formula:

[0084]

[0085] Where B represents the second predicted spatial relationship. This refers to the topological relationship of the second predicted spatial relationship. z is the average of the feature maps of all images in the test set as a single input. e For object classification and object bounding box prediction, The second predicted spatial relationship is the directional relationship. S64: Based on the first and second predicted spatial relationships, obtain the probability distribution of each spatial relationship, as shown in the formula:

[0086]

[0087]

[0088]

[0089]

[0090] Where, r tp + r is the difference in topological relationship between the first predicted spatial relationship and the second predicted spatial relationship. dir + r is the difference between the directional relationships of the first predicted spatial relationship and the second predicted spatial relationship. dis + Let A be the difference between the distances between the first predicted spatial relationship and the second predicted spatial relationship, and B be the distance between them. Let x be the topological relation of the first predicted spatial relationship, and z be the object feature map. e For object classification and object bounding box prediction, The first predicted spatial relationship is the directional relationship. The distance relationship is the first predicted spatial relationship. This refers to the topological relationship of the second predicted spatial relationship. z is the average of the feature maps of all images in the test set as a single input. e For object classification and object bounding box prediction, The second predicted spatial relationship is the directional relationship. The distance relationship is the second predicted spatial relationship; Let be the probability distribution of the topological relationship. Let be the probability distribution of the directional relationship. The distance relationship is the probability distribution, softmax() is the normalization function, and compress() is the normalization function. tp For topological compression networks, "Linear" indicates that the network has a linear structure, and "Compress" indicates that the network has a linear structure. dir Compress the network for directional relationships. dis S65: Select the relationship category with the highest probability distribution for each spatial relationship as the final spatial relationship prediction result.

[0091] In some embodiments, the above-described method for identifying spatial relationships in remote sensing images can also be implemented in the following ways.

[0092] In this embodiment, a deep learning-based remote sensing image spatial relationship recognition method jointly labels the spatial interactions between objects in remote sensing images in the form of "topology, orientation, and distance," projecting object representations onto three different low-dimensional spaces. The topological, orientation, and distance relationships are modeled as vector subtraction between object representations in the three spatial relationship spaces. The inherent connections and constraints between spatial relationships are utilized to optimize the computation process. During the inference stage, the difference between the normal input and the input with fixed object feature maps in different spaces is calculated to obtain the three spatial relationships, which are then combined to express the complex spatial interactions between objects. This invention effectively solves the problem of existing technologies being unable to simultaneously detect multiple spatial relationships, significantly improving the current level of semantic understanding of remote sensing images. The model is simple and easy to train.

[0093] In this embodiment, the remote sensing image spatial relationship recognition method based on deep learning is as follows: Figure 2 As shown, the specific steps include: S1, the dataset uses joint topology, orientation, and distance spatial relationships to annotate the spatial interactions between objects in the remote sensing image in the form of "topology, orientation, distance". The complete annotation format is as follows: "baseball field, distant, northeast, near, playground":

[0094] 1) Topological spatial relationships: disjoint, contiguous, contained, intersecting;

[0095] 2) Spatial orientation: East, West, South, North, Southeast, Northeast, Southwest, Northwest, Center;

[0096] 3) Spatial relationship: very close, near, far;

[0097] The possible combinations of relationships are shown in Table 1, with a total of 42 different spatial relationships:

[0098] Table 1: Possible combinations of relationships

[0099]

[0100] S2. The data in the dataset has been preprocessed and divided into training and test sets in a 7:3 ratio:

[0101] Iterate through a batch of remote sensing images input at once, perform mean subtraction and variance division on each image, scale the image to a specified size range, and add corresponding scaling annotations to the bounding box information in the dataset.

[0102] S3. During the training phase, the training set images are input into the object detection network to extract features of the remote sensing objects, including feature maps, classification probabilities, and bounding boxes:

[0103] The image is input to a pre-trained object detection network, Faster-RCNN, for feature extraction and object bounding box regression. The Faster-RCNN network structure includes an input layer, pooling layer, convolutional layer, and output layer. This yields the classification probability cls of identifiable remote sensing objects in the image, the object feature map v, and the object bounding box parameters (x, y, w, h). The object bounding box represents the object's location; it is a rectangular box that perfectly encloses the object, where (x, y) represents the coordinates of the bounding box's center point, and (w, h) represents the width and height of the bounding box.

[0104] S4. Group the remote sensing object features obtained above into pairs, calculate the relative parameters of each pair relative to the other based on their respective bounding boxes, and concatenate them with the feature map and classification probability to form the representation of the remote sensing object:

[0105] The object classification probabilities cls obtained from a set of objects processed by an object detection network, the object feature maps v, and the object bounding box parameters (x, y, w, h) are input into the feature construction module. These objects are referred to as the subject and object objects of the spatial relationship. Object representation construction mainly includes relative parameter calculation and feature concatenation.

[0106] The relative parameter calculation section calculates the relative positional parameter bbox between the subject and object objects in spatial relationships. x ,t y ,t w ,t h Among the position parameters (t) x ,t y (t) indicates a translation of a group of objects with their dimensions unchanged relative to each other. w ,t h This indicates the width and height changes of a group of objects in the logarithmic space relative to each other. The relative position parameters bbox of the subject and object objects are denoted as bbox respectively. s bbox o The calculation is as follows:

[0107]

[0108]

[0109] Among them, (x s ,y s ,w s ,h s ) and (x o ,y o ,w o ,h o ) are the bounding box parameters of the subject and object objects obtained in step 3), respectively.

[0110] The concatenation part connects the object classification probability cls, the object feature map v, and the relative position parameter bbox using the concat function to construct the representation rep of the subject and object objects respectively. s and rep o This is used to centrally represent the semantic features, location information, and visual features of the subject and object, and the dimension of the representation is denoted as M:

[0111] rep s =concat(cls) s bbox s ,v s )

[0112] rep o =concat(cls) o bbox o ,v o )

[0113] S5. The representation of a set of remote sensing objects is converted into three sets of low-dimensional representations. The difference between the low-dimensional representations of each set of objects is calculated. The three differences are used as the predicted topological, orientation, and distance spatial relationship labels, respectively. The transformation parameters are adjusted according to the differences between the predicted three spatial relationships and the actual three spatial relationships, so that the predicted values ​​are as close as possible to the actual values. The specific process is as follows:

[0114] The subject and object representations are projected onto three different dimensional spaces, d1, d2, and d3, through transformation parameters, to identify different spatial relationships in different feature spaces. The actual spatial relationship between the subject and object can be expressed by the following formula:

[0115] r tp ≈W o-tp rep o -W s-tp rep s

[0116] r dir ≈W o-dir rep o -W s-dir rep s

[0117] r dis ≈W o-dis rep o -W s-dis rep s

[0118] Among them, W s-tp W o-tp These respectively represent the subject object as rep sand object representation rep o The projection transformation parameters, W, are used to transform the object to the topological feature space. s-dir W o-dir These respectively represent the subject object as rep s and object representation rep o The projection transformation parameters, W, are transformed to the orientation feature space. s-dis W o-dis These respectively represent the subject object as rep s and object representation rep o Projection transformation parameters for conversion to distance feature space.

[0119] Based on the above formulas, the predicted spatial relationships are as follows:

[0120]

[0121]

[0122]

[0123] 2) Since the predicted spatial relationships are obtained through 1) by connecting object features, there is a dimensionality inconsistency with the true values ​​of the spatial relationships. The predicted spatial relationships are input into the following three linear networks: Compress tp Compress dir Compress dis Compress the dimension to be the same as the dimension of the spatial relationship:

[0124] Compress tp =Linear(d1,N tp )

[0125] Compress dir =Linear(d2,N dir )

[0126] Compress dis =Linear(d3,N dis )

[0127] Where d1, d2, and d3 represent the predicted spatial relationships, respectively. The dimension of N. tp N dir N dis These represent the number of categories for topological, directional, and distance relationships, respectively.

[0128] The compression network (Compress) is dynamically updated using the following loss function. tpCompress dir Compress dis and projection conversion W s-tp W o-tp W s-dir W o-dir W s-dis W o-dis Parameters:

[0129]

[0130]

[0131]

[0132] Where R tp R dir R dis Let represent all object topological relation triples, object direction relation triples, and object distance relation triples existing in the training set, respectively, and let s and o represent the subject object and object object, respectively. This represents the true value of the spatial relationship distribution.

[0133] S6. In the inference phase, test set images are input into the target detection network, as in steps 3)-4), to obtain a representation of the remote sensing object. Based on the transformation parameters learned during the training phase, the three predicted spatial relationships are calculated. In step 4), the value of the object feature map is set to the average value of all image feature maps input at once during the inference phase. Based on this, a new remote sensing object representation is formed, and the three spatial relationships are then calculated. The spatial relationship obtained from the object representation without feature map processing is called A, and the spatial relationship obtained from the object representation with the feature map set to the average value of all image feature maps input at once is called B. Calculate A and B as the inference results of the three spatial relationships. Refer to the flowchart for details. Figure 3 The specific process is as follows:

[0134] 1) Input test set images into the object detection network to obtain object feature maps, object classification predictions, and object bounding box predictions, and calculate the relative position parameters of the subject and object respectively;

[0135] 2) Connect the subject and object feature maps, bounding box predictions, and relative position parameters to construct subject and object representations. Multiply the object representations by the transformation parameters determined during the training phase, and subtract the product to obtain the spatial relationship prediction:

[0136]

[0137] Where x represents the object feature map, z e This represents object classification and object bounding box prediction. These are the topological relationships, directional relationships, and distance relationships obtained from the object feature map, object classification, and object bounding box, respectively.

[0138] The feature maps of the subject and object are manually set to the average of the feature maps of all input images at once. This average feature map, bounding box prediction, and relative position parameters are then concatenated to construct another set of subject and object representations. This set of object representations is multiplied by the transformation parameters determined during the training phase, and the resulting product is subtracted to obtain the spatial relationship prediction:

[0139]

[0140] in This represents the average value of all image feature maps input at once. These represent the predicted topological relationships, directional relationships, and distance relationships obtained from object feature maps, object classifications, and object bounding boxes, which are set to fixed values ​​by humans, respectively.

[0141] 3) Calculate the difference between the two predicted values:

[0142]

[0143]

[0144] r tp + ,r dir + ,r dis + via Compress tp ,Compress dir ,Compress dis The data is compressed and converted to the same dimension as the spatial relation type. A softmax operation is then performed on the result to obtain the probability distribution of each spatial relation type.

[0145]

[0146]

[0147]

[0148] Overall situation as follows Figure 4 As shown.

[0149] The relationship category with the highest probability for each spatial relationship distribution is selected as the final spatial relationship prediction result. The overall process is as follows: Figure 5 As shown. This approach strengthens the influence of object features on the reasoning process, which is conducive to deriving spatial relationships that are more consistent with reality.

[0150] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the remote sensing image spatial relationship recognition method described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0151] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the aforementioned remote sensing image spatial relationship recognition method. Figure 6 As shown, the computer device may include: at least one processor 151, such as a CPU (Central Processing Unit), at least one communication interface 153, memory 154, and at least one communication bus 152. The communication bus 152 is used to enable communication between these components. The communication interface 153 may include a display screen and a keyboard; optionally, the communication interface 153 may also include a standard wired interface or a wireless interface. The memory 154 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk drive. Optionally, the memory 154 may also be at least one storage device located remotely from the processor 151. The memory 154 stores application programs, and the processor 151 calls the program code stored in the memory 154 to execute any of the aforementioned method steps. The communication bus 152 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 152 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6The term 154 is represented by a single line, but this does not imply a single bus or a single type of bus. The memory 154 may include volatile memory, such as random-access memory (RAM); it may also include non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it may include a combination of the above types of memory. The processor 151 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. The processor 151 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. Optionally, the memory 154 is also used to store program instructions. The processor 151 can call the program instructions to implement the remote sensing image spatial relationship recognition method as described in this embodiment.

[0152] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for recognizing spatial relationships in remote sensing images, characterized in that, Includes the following steps: S1: Obtain labeled remote sensing images and their remote sensing objects, and label the remote sensing objects using spatial relationships to obtain a labeled dataset; the spatial relationships include topological spatial relationships, azimuth spatial relationships, and distance spatial relationships; S2: Preprocess the labeled dataset to obtain the training set and the test set; S3: Based on the training set, use the target detection network to obtain the features of the remote sensing object; S4: Based on the characteristics of the remote sensing object, perform relative position calculation and stitching operations to obtain the representation of the remote sensing object; S5: Construct a linear neural network, and train the linear neural network using the representation of the remote sensing object to obtain a trained linear neural network; S6: Based on the test set, use the target detection network and the trained linear neural network to obtain the spatial relationship prediction results; The target detection network is Faster-RCNN, which includes an input layer, a pooling layer, a convolutional layer, and an output layer. The input layer receives the original image data. The pooling layer reduces the spatial resolution of the feature map while preserving key features. The convolutional layer performs convolution operations on the input image to extract local features at different scales and directions. The output layer outputs the class probability distribution and bounding box coordinates. The features of the remote sensing object include object classification probability, object feature map, and object bounding box parameters; the object bounding box parameters are used to represent the center point coordinates and size of the bounding box of the remote sensing object. Step S6 includes: S61: Based on the test set, use a target detection network to obtain the first feature of the remote sensing object; S62: Based on the first feature of the remote sensing object, obtain the first representation of the remote sensing object; based on the first representation of the remote sensing object, use the trained linear neural network to obtain the first predicted spatial relationship, as shown in the formula: Where A represents the first predicted spatial relationship. Let x be the topological relation of the first predicted spatial relationship, and z be the object feature map. e For object classification and object bounding box prediction, The first predicted spatial relationship is the directional relationship. The distance relationship is the first predicted spatial relationship; S63: Set the object feature map in the first feature of the remote sensing object to the average value of the feature maps of all images in the test set input at one time to obtain the second feature of the remote sensing object; obtain the second representation of the remote sensing object based on the second feature of the remote sensing object; obtain the second predicted spatial relationship based on the second representation of the remote sensing object using the trained linear neural network, as shown in the formula: Where B represents the second predicted spatial relationship. The average of the feature maps of all images in the test set as a single input; S64: Based on the first and second predicted spatial relationships, the probability distribution of each spatial relationship is obtained, as shown in the formula: Where, r tp + r is the difference in topological relationship between the first predicted spatial relationship and the second predicted spatial relationship. dir + r is the difference between the directional relationships of the first predicted spatial relationship and the second predicted spatial relationship. dis + This represents the difference in distance between the first predicted spatial relationship and the second predicted spatial relationship. Let be the probability distribution of the topological relationship. Let be the probability distribution of the directional relationship. The probability distribution of distance relationships is given by `softmax()`, which is the normalization function. tp For topological compression networks, "Linear" indicates that the network has a linear structure, and "Compress" indicates that the network is a linear structure. dir Compress the network for directional relationships. dis Compress the network based on distance relationships; S65: Select the relationship category with the highest probability distribution for each spatial relationship as the final spatial relationship prediction result.

2. The method for spatial relationship recognition in remote sensing images according to claim 1, characterized in that, The topological spatial relationships include: disjoint, connected, contained, and intersecting; the directional spatial relationships include: east, west, south, north, southeast, northeast, southwest, northwest, and center.

3. The method for spatial relationship recognition in remote sensing images according to claim 2, characterized in that, Step S2 specifically includes: performing mean subtraction, variance division, and scaling on each image in the labeled dataset, and dividing it into a training set and a test set according to a 7:3 ratio.

4. The method for spatial relationship recognition in remote sensing images according to claim 1, characterized in that, Step S4 specifically includes: S41: Group the remote sensing objects into pairs to obtain spatial relationship object groups, which include subject objects and object objects; based on the characteristics of the remote sensing objects, obtain the relative position parameters, as shown in the formula: Among them, bbox s The relative position parameter of the subject, x s x-coordinate of the center point of the subject's bounding box o w is the x-coordinate of the center point of the bounding box of the object. o y is the width of the bounding box of the object. s The y-coordinate of the center point of the subject's bounding box. o h is the ordinate of the center point of the bounding box of the object. o bbox is the height of the bounding box of the object. o The relative position parameter of the object; S42: Based on the characteristics of the remote sensing object and the relative position parameters, a representation of the remote sensing object is obtained. The representation of the remote sensing object includes a representation of the subject object and a representation of the object, as shown in the formula: rep s =concat(cls s ,bbox s ,v s ) rep o =concat(cls o ,bbox o ,v o ) Among them, rep s The representation of the subject object, concat() is the concatenation function, cls s Classification probability of the subject object, bbox s The relative position parameter of the subject, v s The object feature map of the subject object; rep o cls represents the object of the verb. o The classification probability of the object, bbox o v is the relative position parameter of the object. o This is the object feature map of the object.

5. The method for identifying spatial relationships in remote sensing images according to claim 4, characterized in that, Step S5 includes: S51: Construct a linear neural network, which includes a topology compression network, a direction relationship compression network, a distance relationship compression network, a fully connected layer for topological projection transformation of object representation, a fully connected layer for topological projection transformation of subject representation, a fully connected layer for directional projection transformation of object representation, a fully connected layer for directional projection transformation of subject representation, a fully connected layer for distance projection transformation of object representation, and a fully connected layer for distance projection transformation of subject representation. The topology compression network is used to compress the dimension of topology relationships, the direction relationship compression network is used to compress the dimension of directional relationships, and the distance relationship compression network is used to compress the dimension of distance relationships. The object representation topological projection transformation fully connected layer is used to transform the representation of the object to the topological feature space. The subject representation topological projection transformation fully connected layer is used to transform the representation of the subject to the topological feature space. The object representation direction projection transformation fully connected layer is used to transform the representation of the object to the direction feature space. The subject representation direction projection transformation fully connected layer is used to transform the representation of the subject to the direction feature space. The object representation distance projection transformation fully connected layer is used to transform the representation of the object to the distance feature space. The subject representation distance projection transformation fully connected layer is used to transform the representation of the subject to the distance feature space. S52: Based on the representation of the remote sensing object, the predicted spatial relationship is obtained using the linear neural network, as shown in the formula: Where, r tpe For the predicted topological relationships, W o-tp The projection transformation parameters, rep, are used to transform the representation of the object to the topological feature space. o W represents the object of the verb. s-tp The projection transformation parameters, rep, are used to transform the representation of the subject object to the topological feature space. s The representation of the subject object, r dire For the predicted directional relationship, W o-dir W is the transformation parameter for projecting the representation of the object to the directional feature space. s-dir The transformation parameters r are projection transformation parameters used to transform the representation of the subject object into the directional feature space. dise For the predicted distance relationship, W o-dis W is the transformation parameter for projecting the representation of the object to the distance feature space. s-dis The transformation parameters are used to convert the representation of the subject object to the projection transformation space. S53: Using the linear neural network, the predicted spatial relationship is compressed, a loss function is constructed, and the linear neural network is trained using the loss function to obtain a trained linear neural network, as shown in the formula: Compress tp =Linear(d1,N tp ) Compress dir =Linear(d2,N dir ) Compress dis =Linear(d3,N dis ) Where d1 is the dimension of the predicted topological relationship, d2 is the dimension of the predicted directional relationship, d3 is the dimension of the predicted distance relationship, and N is the dimension of the predicted distance relationship. tp N represents the number of categories of topological relationships. dir N represents the number of categories of directional relationships. dis Let be the number of categories of distance relationships, and log be the logarithm. Calculate the loss function for topological spatial relationships. The loss function calculated for spatial relationships of directions. The loss function R is calculated for distance relationships. tp For all object topological relation triples existing in the training set, s represents the subject object and o represents the object object. R represents the true value of the topological distribution. dir To provide all object orientation relation triples existing in the training set. R represents the true value of the directional relationship distribution. dis To provide all object distance triples in the training set, This represents the true value of the distance relationship distribution.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the remote sensing image spatial relationship recognition method as described in any one of claims 1-5.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the remote sensing image spatial relationship recognition method as described in any one of claims 1-5.

Images