Scene graph generation method and system supporting historical and cultural block scenes

Abstract

The present invention discloses a scene graph generation method and system supporting the scene of historical and cultural blocks, including: analyzing historical and cultural geographical elements and related relationships, constructing a scene graph generation data set for historical and cultural blocks; training a landscape semantic recognition model, extracting visual features, semantic features, and spatial features of target objects; using the co-occurrence frequency of target relationships as prior knowledge, and realizing the prediction of target relationships based on an attention mechanism and a feature fusion method; using a street view image that is continuous in time and space as input, and based on a landscape semantic recognition model and a target relationship prediction model, generating a scene graph corresponding to each street view image, and using the scene graph fusion model as input to realize the fusion of the scene graph. The advantages of the present invention are: it can generate a scene graph with the characteristics of a historical and cultural block, effectively express the attributes of landscape elements and the characteristics of the spatial distribution relationship of elements in the historical and cultural block, assist in the optimization of resources and policies, improve urban efficiency, and promote sustainable development.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence image understanding technology, and in particular to a scene graph generation method and system that supports accurate and comprehensive information mining and expression in historical and cultural block scenes. Background Art

[0002] Historical and cultural blocks have unique aesthetic, cultural, social and economic values. However, how to mine more accurate and comprehensive element information and express more intuitive information at a fine scale in blocks to assist urban managers in decision-making, optimize resource allocation and promote the sustainable development of historical and cultural blocks still requires in-depth research and exploration.

[0003] With the promotion and deepening of big data technology and the vigorous development of artificial intelligence industrial applications, feature extraction, target detection, semantic segmentation and image classification are often used to obtain information from street view images. Although these methods can provide image features, segmentation results or category labels, they lack deep semantic understanding of images. At the same time, understanding the results generated by these methods usually requires users to have professional knowledge and skills, and it is relatively difficult for non-professional users to interact with computer systems. In addition, the current methods such as feature extraction, target detection, image segmentation and image classification are difficult to expand on their semantic understanding, which is not conducive to accurate and comprehensive information mining at a fine scale. There are limitations in the expression of the attributes, spatial distribution relationships and correlation relationships of landscape elements in historical and cultural blocks, and it is difficult to effectively improve the information utilization efficiency of urban managers.

[0004] The scene graph can represent the entire target image as a topological structure to abstractly express the high-level information in the image. However, how to use the thinking of the visual scene graph to effectively express the attributes of landscape elements and the spatial distribution relationship characteristics of elements in historical and cultural blocks is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] In view of the defects of the prior art, the present invention provides a method and system for generating a scene graph supporting a historical and cultural block scene.

[0006] In order to achieve the above invention object, the technical solution adopted by the present invention is as follows:

[0007] A method for generating a scene graph supporting a historical and cultural block scene comprises the following steps:

[0008] S1. Refer to existing datasets, such as the ImageNet dataset and the coco dataset, collect street view images of relevant historical and cultural blocks, take into account the data characteristics of street view images of historical and cultural blocks, and use image annotation tools to construct scene graphs of historical and cultural blocks to generate datasets.

[0009] S2. Use the historical and cultural block scene graph dataset as training data to train the landscape semantic recognition model and extract the visual features, semantic features, and spatial features of the target object;

[0010] S3, relationship prediction model design, taking the co-occurrence frequency of target relationships as prior knowledge, integrating it into the feature level to enhance the feature representation effect of target relationships, and using attention mechanism and feature fusion method to predict target relationships;

[0011] S4. Taking continuous spatiotemporal street view images as input, the trained landscape semantic recognition model and target relationship prediction model are used to process the spatiotemporal continuous street view images in the target area to obtain a spatiotemporal continuous scene graph corresponding to each street view image.

[0012] S5. Fusing the generated spatiotemporal continuous scene graphs based on a scene graph fusion model. The scene graph fusion model includes: a predicate alignment module, an embedding learning module, an entity alignment module, and a scene graph fusion module.

[0013] The sub-steps of S5 are as follows:

[0014] S51, predicate alignment module:

[0015] Use the trained model in S4 to process street view images, ensuring that the generated series of scene graphs have the same naming scheme, find the same predicates and provide a unified vector space for relation embedding;

[0016] Adjacent scene graphs are fused into one, and embeddings of relation triplets and attribute triplets are learned.

[0017] S52, embedded learning module:

[0018] The TransE model is used to learn structural embeddings for entity alignment between scene graphs, minimizing the objective function.

[0019] The model focuses on the target predicate relationship through the weighting parameters.

[0020] The corrupted triplets are used as negative samples by replacing the head entity with a random entity or replacing its feature attributes with random feature attributes.

[0021] Jointly learn structure embedding and attribute feature embedding, transform them into the same vector space, obtain the similarity between entities based on cosine similarity, and use the similarity threshold to remove entity pairs that cannot be aligned.

[0022] S53, entity alignment module:

[0023] The learned embedding features are used to align entities in adjacent scene graphs and calculate the similarity between entities.

[0024] Use the similarity threshold to remove dissimilar entity pairs and obtain the expected alignable entity pairs.

[0025] S54, Update large scene graph

[0026] Initialize an empty scene graph, traverse the entity pairs that were previously matched and determined to be the same entity, and merge the entity pairs into one entity.

[0027] The relationship connection in the new scene graph is determined according to the connection relationship between the entities in the old scene graph, and the execution is repeated until all matching entity pairs are completed.

[0028] Furthermore, the sub-steps of S1 are as follows:

[0029] S11. Collection and organization of street view images:

[0030] Historical and cultural blocks were selected as research objects, street view images were collected in a spatiotemporal continuous manner, and the image data were sorted.

[0031] S12. Use of image annotation tools:

[0032] Use professional image annotation tools to annotate targets in street view images in the form of target frames, marking the specific location and shape of the targets.

[0033] S13, target relation triple annotation:

[0034] Through a series of <subject, relationship, object> triplets, the target relationship in street view images is described in the form of a graph structure.

[0035] Add target box annotations to the target objects and annotate the relationships between targets in the form of triplets.

[0036] S14. Target selection and labeling:

[0037] In terms of targets, select target objects with historical and cultural characteristics, including: historical and cultural signs, traditional specialty store signs, promotional posters, special decorations, couplets, iconic stone tablets, and special landmarks.

[0038] At the same time, common features also need to be marked, including: road traffic signs, drainage outlets, roadblocks, debris, tricycles, and street lights.

[0039] S15. Scene graph construction:

[0040] According to the annotated target boxes and relationship triplet sets, the composition is instantiated to form a street view image scene graph.

[0041] In the scene graph, graph nodes represent the ground objects in the image, edges represent the relationship between objects, and there is a one-to-one correspondence between nodes and target box annotations.

[0042] The relationships in the scene graph include topology, direction, distance, spatial relationships, and semantic relationships.

[0043] Furthermore, the sub-steps of S2 are as follows:

[0044] S21. Select encoder and decoder:

[0045] ResNet-r50 is selected as the encoder to extract high-dimensional features of street view images.

[0046] A semantic segmentation decoder is used to decode the features extracted by the encoder to distinguish the semantic differences between different landscape elements and accurately extract various landscape elements in the historical core area.

[0047] S22. Introducing polarized attention mechanism:

[0048] Based on the existing encoder-decoder framework, a polarized attention mechanism is introduced to enhance the model's ability to distinguish different landscape elements.

[0049] S23. Transfer learning:

[0050] The general visual information and prior information about street scenes in the ImageNet dataset and the coco dataset are transferred to the encoder so that the network can be trained to learn the specific features of street scenes in historical and cultural blocks based on general street scene knowledge.

[0051] The encoder’s transfer learning process involves pre-training on the ImageNet dataset and fine-tuning on the Cityscapes dataset and the Historic Core Street Scene dataset.

[0052] The parameters of all models are optimized using stochastic gradient descent optimizer.

[0053] S24, feature extraction:

[0054] The deep feature map of street view imagery is extracted using CNN pre-trained with the landscape semantic recognition model.

[0055] A region alignment algorithm is used to extract visual features that characterize object pairs and their relationships.

[0056] The absolute and relative spatial relationships between objects are considered, and the position coordinates, overlap, and distance of the object boxes are used to quantify spatial features.

[0057] The Word2vec model is used to convert target category words into high-dimensional vectors as semantic features.

[0058] Furthermore, the sub-steps of S3 are as follows:

[0059] S31. Integration of prior knowledge

[0060] The co-occurrence frequency between target objects in the historical and cultural block scene graph generation dataset is counted.

[0061] The prior co-occurrence frequency of target pair relations is calculated and incorporated into model learning as prior knowledge.

[0062] S32, attention mechanism fusion features

[0063] The attention mechanism is introduced to learn the attention weights of spatial features, semantic features, visual features, and prior information in an adaptive manner.

[0064] According to the attention weight of the feature, the softmax function is used to calculate the specific weight, and various features are fused by weighted merging to form a fused feature representation.

[0065] The classifier is used to predict the relationship category of the target pair by using the fusion attention weighted multi-features and prior co-occurrence frequency. The output of the classifier is used to obtain the relationship probability through the softmax activation function.

[0066] In the model training stage, the relationship prediction task is regarded as a multi-classification task, and the multi-classification cross entropy is used as the loss function. The gradient of the loss is back-propagated to continuously learn and optimize the model parameters.

[0067] In the testing phase, forward calculation is performed based on the model parameters to predict the relationship category between the subject and object in the triple.

[0068] Furthermore, the transfer learning process of the encoder in S2 is expressed as:

[0069]

[0070] Among them, P fe Represents the encoder parameters trained using the ImageNet dataset and the coco dataset, which contains general visual image information D img , and used for traditional image classification T f Task. Through transfer learning, in D img Based on the study of the characteristics of the landscape elements in the historical core area hcsv , and change the parameter task to feature extraction T f ′. So far, the encoder parameters P with general visual information and core area landscape element information are obtained. fe ′.

[0071] The transfer learning process of the semantic segmentation decoder is expressed as:

[0072]

[0073] Where P and P′ represent the semantic segmentation decoder parameters before and after migration, respectively, where P is trained using the Cityscapes dataset, which has street view information of European cities. fe And used for street object recognition task Seg cs In P fe Based on the training network learning historical core area street view information P fe ′, and change the model task to identify landscape features, thus obtaining the transferred semantic segmentation decoder parameters P′.

[0074] Furthermore, the visual feature in S24 is expressed as:

[0075] V ij =f vis (I, X i , X j , X ij )

[0076] Where V ij is the target pair relationship (o i , o j ), f vis is a function used to extract visual features, I is the input street view image, X i and X i Based on the target box b i and b j The extracted visual features, X ij The visual features of the corresponding relationship representation area.

[0077] The semantic features are expressed as:

[0078] S ij =f scm (p i , p j , I ij ,U ij ,D ij )

[0079] In the formula, S ij is the target pair relationship (o i , o j ), f scm is a function used to extract semantic features, p i and p i is the position coordinate of the corresponding target frame, U ij is the intersection-over-union ratio of the corresponding target box, D ij is the distance between the corresponding target boxes.

[0080] The spatial feature is expressed as:

[0081] SP ij =f spa (V i , V j )

[0082] Where, SP ij To represent the semantic features of the target pair, V i and V j is the target category O i and O j Word embedding features extracted through language model.

[0083] Furthermore, the prior co-occurrence frequency of the target pair relationship in S31 is expressed as:

[0084]

[0085] In the formula, P(r ij |O i ,O j ) are two given features O i and O j The relationship ij The prior co-occurrence frequency, N ij Given O on the training data set i and O j The relationship ij N is the total number of target pair relations in the training dataset.

[0086] Furthermore, in S32, the relationship category of the target pair is predicted by a classifier using the fused attention weighted multi-features and the prior co-occurrence frequency. The output of the classifier is used to obtain the relationship probability through the softmax activation function. The calculation process is expressed as:

[0087] R ij =σ(W r F ij +b r )

[0088] Among them, R ij For the target pair (O i , O j ), σ(·) the activation function of the classifier, W r and b r are the weight and bias parameters of the classifier, respectively, and F ij The fused features.

[0089] The loss function is defined as follows:

[0090]

[0091] Among them, Ω represents the parameters that need to be trained for the entire relationship prediction model, and y ij (i) Represents the binary label of the relationship between subject i and object j in the triplet in category c, R ij (c) The probability that the relationship category predicted by the representative model for a triple is c, the subject is i, the object is j, and C is the total number of relationship categories.

[0092] Furthermore, in S5, in order to learn the structural embedding, the following objective function is minimized:

[0093]

[0094]

[0095] Among them, T r is a set of valid relation triples, T r ′ is the set of damaged triples, count(r) is the number of times relation r appears, and |T| is the merged triple G 1，2 The total number of triplets in . Adding the weight parameter α helps the model focus more on the target relationship.

[0096] To learn attribute feature embedding, the following objective function is minimized:

[0097]

[0098] T a ={<h,r,a> ∈G}; f(t a )=‖h+rf a (a)‖

[0099] T a ′={<h′,r,a> |h′∈E}∪{ <h,r,a′<|a′∈A}

[0100] Where T a The set of valid attribute triplets from the training dataset, T′ a is a set of damaged triples, A is a set of attributes in G, and the damaged triples are used as negative samples by replacing the head entity with a random entity or replacing its feature attributes with random feature attributes. a ) is based on the embedding of the head entity h, the embedding of the relation r, and the representation vector of the attribute (using the combination function f a (t a )Calculate the credibility score obtained.

[0101] Joint learning of structural embedding and attribute feature embedding to minimize the objective function J SIM , embed the attribute features into h ceWith h se The structural embedding is transformed into the same vector space as follows:

[0102]

[0103] In the formula, cos(h se ,h ce ) is the vector h ce and h se The cosine similarity of .

[0104] The overall objective function of joint learning of structure embedding and attribute feature embedding is:

[0105] J=J SE +J CE +J SIM

[0106] The entity alignment module calculates the similarity of triples to perform entity alignment, and the formula is as follows:

[0107]

[0108] Where, entity h1∈G1, calculates the similarity between all entities h2∈G2 in the scene graph G2. <h2,h map > are the expected alignable triplets, and the similarity threshold β is used to remove the triplets that are dissimilar and cannot be aligned.

[0109] In another embodiment of the present invention, a scene graph generation system for supporting historical and cultural block scenes is provided. The system can be used to implement the above-mentioned scene graph generation method for supporting historical and cultural block scenes, specifically including:

[0110] Dataset construction module: Refer to the ImageNet dataset and the coco dataset, collect street view images of relevant historical and cultural blocks, take into account the data characteristics of street view images of historical and cultural blocks, and construct a scene graph generation dataset of historical and cultural blocks.

[0111] Landscape semantic recognition module:

[0112] Extract the visual, semantic and spatial features of the target object.

[0113] Relationship prediction module:

[0114] Incorporate target relationship co-occurrence frequency as prior knowledge.

[0115] Use attention mechanism and feature fusion method to predict target relations.

[0116] Scene graph generation module:

[0117] Take continuous spatiotemporal street view images as input.

[0118] The street view images are processed using the landscape semantic recognition module and the relationship prediction module to generate a spatiotemporal continuous scene graph corresponding to each street view image.

[0119] The scene graph fusion module includes the following sub-modules:

[0120] Predicate alignment module: fuses adjacent scene graphs into one and performs embedding learning of relation triplets and attribute triplets.

[0121] Embedding learning module: learns structural embeddings, controls the model to focus on aligned triplets, and performs entity alignment.

[0122] Entity alignment module: Use the learned embedding features to align entities in adjacent scene graphs and calculate the similarity between entities.

[0123] Update large scene graph module: Determine the relationship connection in the new scene graph based on the connection relationship between the entities in the old scene graph until all matching entity pairs are merged.

[0124] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for generating a scene graph supporting a historical and cultural block scene is implemented.

[0125] The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for generating a scene graph supporting a historical and cultural block scene is implemented.

[0126] Compared with the prior art, the advantages of the present invention are:

[0127] 1. Comprehensive dataset construction: By using diverse datasets to construct historical and cultural block scene graphs to generate datasets, the system can better understand the diversity and complexity of historical and cultural blocks.

[0128] 2. Landscape semantic recognition model: The training landscape semantic recognition model can effectively identify various landscape elements in historical and cultural blocks and extract their visual features, semantic features and spatial features, thus laying the foundation for scene graph generation.

[0129] 3. Relationship prediction model: Designing a relationship prediction model and incorporating the co-occurrence frequency of target relationships as prior knowledge can improve the accuracy of the prediction of target relationships and help to better understand the associations between various elements within historical and cultural blocks.

[0130] 4. Generation of spatiotemporal continuous scene graphs: Using continuous spatiotemporal street view images as input, a spatiotemporal continuous scene graph corresponding to each street view image is generated, which can better reflect the changes and characteristics of historical and cultural blocks in the spatiotemporal dimensions.

[0131] 5. Scene graph fusion module: Through the predicate alignment module, embedding learning module, entity alignment module and scene graph fusion module, the generated spatiotemporal continuous scene graph can be effectively fused to improve the integrity and effectiveness of the scene graph, and better express the attributes of landscape elements and the characteristics of spatial distribution relationships within historical and cultural blocks.

[0132] 6. Multimodal feature fusion: The system adopts multiple feature fusion methods, including visual features, semantic features, spatial features, and the fusion of prior knowledge. It can comprehensively consider the contribution of different features to scene generation, thereby improving the accuracy and robustness of scene generation.

[0133] 7. Entity alignment and scene graph update: Through the entity alignment module and the scene graph update module, the entity alignment and scene graph update in adjacent scene graphs can be effectively handled, ensuring the coherence and accuracy of the generated scene graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0134] Figure 1 It is a flow chart of a method for generating a scene graph supporting a historical and cultural block scene according to an embodiment of the present invention. DETAILED DESCRIPTION

[0135] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0136] like Figure 1 As shown, the present invention provides a method for generating a scene graph supporting a historical and cultural block scene, comprising the following steps:

[0137] S1. Analyze historical and cultural geographical elements and related relationships, construct historical and cultural block scene maps to generate data sets;

[0138] S2. Using the historical and cultural block scene graph dataset as training data, train the landscape semantic recognition model, design and extract the visual features, semantic features, and spatial features of the target object;

[0139] S3, taking the co-occurrence frequency of target relations as prior knowledge, it is integrated into the relation prediction model at the feature level to strengthen the feature representation effect of the target relations, and finally the attention mechanism and feature fusion method are used to realize the prediction of the target relations;

[0140] S4, taking the spatiotemporally continuous street view images as input, generating a scene graph corresponding to each street view image based on the landscape semantic recognition model and the target relationship prediction model;

[0141] S5. Based on the scene graph fusion model, a series of spatiotemporal continuous scene graphs are fused. The fused scene graph can fully and effectively express the attributes of landscape elements and the spatial distribution relationship characteristics of elements in the historical and cultural blocks. This scene graph fusion model is divided into a predicate alignment module, an embedding learning module, an entity alignment module and a scene graph fusion module.

[0142] In this embodiment, S1 constructs a historical and cultural block scene graph to generate a data set. Taking into account the data characteristics of the street view images of the historical and cultural blocks, a professional image annotation tool is used to use a target frame to mark the specific position and shape of the target, and a series of <subject, relationship, object> triples are used to describe the target relationship in the remote sensing image in the form of a graph structure. When constructing the data set specifically, street view images are collected and data is sorted in a historical and cultural block in a spatiotemporal continuous manner. Volunteers with relevant professional knowledge backgrounds are organized to add target frame annotations to target objects in a human-computer interactive manner, and annotate the relationship between targets in the form of triples. In the end, each image can obtain several target frames and a set of target relationship triples. For the triple set, an instantiation composition can be performed to form a street view image scene graph. In the scene graph, the graph nodes represent the ground objects of the image, the edges represent the relationship between the targets, and the nodes and target frame annotations are in a one-to-one correspondence. In terms of targets, select targets with historical and cultural characteristics, including historical and cultural signs, traditional specialty store signs, promotional posters, characteristic decorations, couplets, iconic stone tablets, characteristic landmarks, etc. In addition, ordinary features can also be appropriately labeled, which is conducive to analyzing the attributes of landscape elements and the characteristics of the spatial distribution relationship of elements, and conducting comprehensive information mining and expression, including road traffic signs, drainage outlets, roadblocks, sundries, tricycles, street lights, etc. The relationships in the scene graph cover common spatial relationships such as topology, direction, distance, and some semantic relationships.

[0143] In this embodiment, S2 trains a landscape semantic recognition model;

[0144] In order to further adapt to the complex characteristics of landscape elements in street view images of historical core blocks, the landscape semantic recognition model introduces a polarized attention mechanism based on the existing encoder and decoder framework to enhance the model's ability to distinguish different landscape elements. In order to better utilize the collected training data, the transfer learning method is introduced to transfer the general visual information and prior information about street scenes in the ImageNet dataset and coco dataset to the encoder, and then train the network to learn the specific characteristics of street scenes in historical and cultural blocks based on general street view knowledge, maximize the training efficiency of the model, extract targeted street view features and improve the recognition accuracy of the final landscape elements. Among them, the encoder is selected as ResNet-r50 (ResNet network with 50 convolution layers). According to the high-dimensional features extracted by the encoder, the semantic segmentation decoder can distinguish the semantic differences of different landscape elements, and then accurately extract various landscape elements in the historical core area.

[0145] In the transfer learning part, the weight parameters of the encoder (pre-trained using ImageNet) are provided by PyTorch. In the decoder transfer learning process, the network is first iteratively trained 40,000 times on the Cityscapes dataset, and then trained 20,000 times on the proposed historical core street scene dataset. This paper uses the Stochastic Gradient Descent (SGD) optimizer to optimize the parameters of all models, with momentum set to 0.9, learning rate set to 0.01, and weight decay set to 0.0005.

[0146] The transfer learning process of the encoder can be expressed as:

[0147]

[0148] Among them, P fe Represents the encoder parameters trained using the ImageNet dataset and the coco dataset, which contains general visual image information D img , and used for traditional image classification T f Task. Through transfer learning, in D img Based on the study of the characteristics of the landscape elements in the historical core area hcsv , and change the parameter task to feature extraction T f ′. So far, the encoder parameters P with general visual information and core area landscape element information are obtained. fe ′.

[0149] Similar to the encoder, the transfer learning process of the semantic segmentation decoder can be expressed as:

[0150]

[0151] Where P and P′ represent the semantic segmentation decoder parameters before and after migration, respectively, where P is trained using the Cityscapes dataset, which has street view information of European cities. fe And used for street object recognition task Seg cs This article is in P fe Based on the training network learning historical core area street view information P fe ′, and change the model task to identify landscape features, thus obtaining the transferred semantic segmentation decoder parameters P′.

[0152] For visual features, the CNN pre-trained based on the landscape semantic recognition model in the previous part extracts the deep feature map of the street view image. According to the target frame range of the target pair, the ROI Align algorithm is used to extract the ROI features that represent the target pair and their relationship. For spatial features, considering the absolute and relative spatial relationship between targets, the position coordinates of the target frame, the overlap between target frames, and the distance between target frames are used to quantify the spatial features. This method uses the word embedding of the target category name as the semantic feature to represent the target relationship. The semantic feature uses the Word2vec model to convert the target category words into high-dimensional vectors.

[0153] The visual features can be expressed as:

[0154] V ij =f vis (I, X i , X j , X ij )

[0155] Where V ij is the target pair relationship (o i , o j ), f vis is a function used to extract visual features, I is the input street view image, X i and X i Based on the target box b i and b j Extracted ROI features, X ij is the ROI feature of the corresponding relationship representation area.

[0156] Semantic features can be expressed as:

[0157] S ij =f scm (p i , p j , I ij ,U ij ,D ij )

[0158] S ij is the target pair relationship (o i , o j ), f scm is a function used to extract semantic features, p i and p j is the position coordinate of the corresponding target frame, U ij is the intersection-over-union ratio of the corresponding target box, D ij is the distance between the corresponding target boxes.

[0159] The spatial features can be expressed as:

[0160] SP ij =f spa (V i , V j )

[0161] SP ij To represent the semantic features of the target pair, V i and V j is the target category O i and O j Word embedding features extracted through language model.

[0162] The co-occurrence frequency of target relations is used as prior knowledge and integrated into the relation prediction model at the feature level to enhance the feature representation effect of the target relations. Finally, the attention mechanism and feature fusion method are used to realize the prediction of the target relations.

[0163] In this embodiment, S3 includes:

[0164] S31. Count the co-occurrence frequencies between target objects in the existing historical and cultural block scene graph generation data set. The closeness of the relationship between them can be measured by calculating the number of co-occurrences or co-occurrence probabilities between each pair of target objects. Incorporate features into model learning to enhance the feature representation capability of target relationships. The prior co-occurrence frequency of target pair relationships can be expressed as:

[0165]

[0166] Among them, P(r ij |O i ,O j ) are two given features O i and O j The relationship ij The prior co-occurrence frequency, N ij Given O on the training data set i and O j The relationship ij The prior co-occurrence count of .

[0167] N is the total number of target pair relations in the training dataset.

[0168] S32. The attention mechanism allows the neural network to selectively focus on specific parts of the input. During the information processing process, it can automatically learn and focus on a subset of input features. The feature attention module adaptively learns the attention weights of features (spatial features, semantic features, visual features, prior information). The attention mechanism can be achieved by mapping the input information to the corresponding attention weights and then performing a weighted operation between the input information and the attention weights. v 、e s 、e c 、e r They represent the attention of visual features, spatial features, semantic features, and prior information respectively.

[0169] Then the specific weight is calculated through the softmax function. Taking visual features as an example, the attention weight is:

[0170]

[0171] Various features are fused by weighted merging, and the fused features can be expressed as:

[0172] F ij =[α v V ij ,α s S ij ,α c C ij ,α r r ij ]

[0173] For the target pair, the multi-features weighted by fusion attention and the prior co-occurrence frequency can be used to predict their relationship categories through the classifier, and its calculation process can be expressed as:

[0174] R ij =σ(W r F ij +b r )

[0175] Among them, R ij For the target pair (O i , O j ) is the probability output of the relationship, σ(·) is the activation function of the classifier, which can be softmax, W r and b r are the weight and bias parameters of the classifier respectively.

[0176] In the model training stage, the relationship prediction task can be regarded as a multi-classification task, and the multi-classification cross entropy is used as the loss function to train the relationship prediction model. The loss function can be defined as follows:

[0177]

[0178] Among them, Ω represents the parameters that need to be trained for the entire relationship prediction model, and y ij (i) Represents the binary label of the relationship between subject i and object j in the triplet in category c, and C is the total number of relationship categories. Through back propagation, the model parameter Ω can be continuously learned and optimized according to the gradient of the loss. In the test phase, the subject and object targets in the triplet can be forward calculated according to the model parameters to predict the corresponding relationship category.

[0179] In this embodiment, S5 can align entities in the generated series of spatiotemporal continuous scene graphs based on the embedding model. The scene graph fusion task is performed based on the feature similarity of entities in adjacent scene graphs. This scene graph fusion model is divided into a predicate alignment module, an embedding learning module, an entity alignment module, and a scene graph fusion module.

[0180] In the predicate alignment module, adjacent scene graphs are trained by the same model and have the same naming scheme, which can find the same predicates and provide a unified vector space for relation embedding. The adjacent scene graphs G1 and G2 are fused into G 1，2 , the fused G 1，2 Decompose into relation triples T r With attribute triple T a Embedding learning is performed (based on the TransE model). Adjacent scene graphs have similar entities and relations, so the relation triple T r can fall into the same vector space. Similarly, the attribute triple T a They can also fall into the same vector space. In order to enable entity embedding to capture the similarities between entities from different scene graphs, the structural embedding of the entity is transferred to the same vector space.

[0181] S52 embedding learning module. For structural embedding, TransE is used to learn structural embeddings for entity alignment between scene graphs. The model controls the embedding learning to pay more attention to aligned triplets by adding a weight. The same predicate appears in two scene graphs, allowing the model to learn more information from aligned triplets.

[0182] To learn structural embeddings, the following objective function is minimized:

[0183]

[0184]

[0185] Among them, T r is a set of valid relation triples, T r ′ is the set of damaged triples, count(r) is the number of times relation r appears, and |T| is the merged triple G 1，2 The total number of triplets in . Since aligned predicates appear in adjacent scene graphs at the same time, the aligned predicates appear more frequently than the unaligned predicates, which allows the model to learn more information from aligned triplets. Adding the weight parameter α helps the model focus more on the target relationship.

[0186] To learn attribute feature embedding, the following objective function is minimized:

[0187]

[0188] T a ={<h,r,a> ∈G}; f(t a )=‖h+rf a (a)‖

[0189] T′ a ={<h′,r,a> |h′∈E}∪{<h,r,a′> |a′∈A}

[0190] Where T a The set of valid attribute triplets from the training dataset, T′ a is a set of damaged triples, (A is a set of attributes in G), and the damaged triples are used as negative samples by replacing the head entity with a random entity or replacing its feature attributes with random feature attributes. a ) is based on the embedding of the head entity h, the embedding of the relation r, and the representation vector of the attribute (using the combination function f a (t a )Calculate the credibility score obtained.

[0191] S53 Joint learning of structural embedding and attribute feature embedding. Minimize the objective function J SIM , embed the attribute features into h ce With h se The structural embedding is transformed into the same vector space.

[0192]

[0193] Among them, cos(h se ,h ce ) is the vector h ce and h seThe cosine similarity of . Structural embedding obtains the similarity between entities in two scene graphs based on entity relationships, while attribute feature embedding obtains the similarity between entities based on feature values. The overall objective function of joint learning of structural embedding and attribute feature embedding is:

[0194] J=J SE +J CE +J SIM

[0195] S54 Entity Alignment. Joint learning of structure embedding and attribute feature embedding allows similar nodes from adjacent scene graphs to have similar embeddings. Therefore, entity alignment is performed through the obtained embeddings;

[0196]

[0197] Among them, there is an entity h1∈G1, and the similarity between h1 and all entities h2∈G2 is calculated. map > is the expected alignable entity pair. A similarity threshold β is used to remove entity pairs that are not similar enough to be aligned.

[0198] S55 updates the large scene graph. First, initialize an empty scene graph. Traverse the entity pairs (E ij , E (i+1)j ), each entity pair (E ij , E (i+1)j ), merge them into one entity E merged , and add it to the empty scene graph. For each pair of entities E merged and E merged′ , determine the relationship connection based on the connection relationship between their old scene graphs. Repeat the above steps until all matching entity pairs are traversed.

[0199] In another embodiment of the present invention, a scene graph generation system for supporting historical and cultural block scenes is provided. The system can be used to implement the above-mentioned scene graph generation method for supporting historical and cultural block scenes, specifically including:

[0200] Dataset construction module: Refer to the ImageNet dataset and the coco dataset, collect street view images of relevant historical and cultural blocks, take into account the data characteristics of street view images of historical and cultural blocks, and construct a scene graph generation dataset of historical and cultural blocks.

[0201] Landscape semantic recognition module:

[0202] Extract the visual, semantic and spatial features of the target object.

[0203] Relationship prediction module:

[0204] ​Incorporate target relationship co-occurrence frequency as prior knowledge.

[0205] Use attention mechanism and feature fusion method to predict target relations.

[0206] Scene graph generation module:

[0207] Take continuous spatiotemporal street view images as input.

[0208] The street view images are processed using the landscape semantic recognition module and the relationship prediction module to generate a spatiotemporal continuous scene graph corresponding to each street view image.

[0209] The scene graph fusion module includes the following sub-modules:

[0210] Predicate alignment module: fuses adjacent scene graphs into one and performs embedding learning of relation triplets and attribute triplets.

[0211] Embedding learning module: learns structural embeddings, controls the model to focus on aligned triplets, and performs entity alignment.

[0212] Entity alignment module: Use the learned embedding features to align entities in adjacent scene graphs and calculate the similarity between entities.

[0213] Update large scene graph module: Determine the relationship connection in the new scene graph based on the connection relationship between the entities in the old scene graph until all matching entity pairs are merged.

[0214] In another embodiment of the present invention, a terminal device is provided, the terminal device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding functions; the processor described in the embodiment of the present invention can be used to support the operation of the scene graph generation method of the historical and cultural block scene, including the following steps:

[0215] S1. Refer to existing datasets, such as the ImageNet dataset and the coco dataset, collect street view images of relevant historical and cultural blocks, take into account the data characteristics of street view images of historical and cultural blocks, and use image annotation tools to construct scene graphs of historical and cultural blocks to generate datasets;

[0216] S2. Use the historical and cultural block scene graph dataset as training data to train the landscape semantic recognition model and extract the visual features, semantic features, and spatial features of the target object;

[0217] S3, relationship prediction model design, taking the co-occurrence frequency of target relationships as prior knowledge, integrating it into the feature level to enhance the feature representation effect of target relationships, and using attention mechanism and feature fusion method to predict target relationships;

[0218] S4. Taking continuous spatiotemporal street view images as input, the trained landscape semantic recognition model and target relationship prediction model are used to process the spatiotemporal continuous street view images in the target area to obtain a spatiotemporal continuous scene graph corresponding to each street view image.

[0219] S5. Fusing the generated spatiotemporal continuous scene graphs based on a scene graph fusion model. The scene graph fusion model includes: a predicate alignment module, an embedding learning module, an entity alignment module, and a scene graph fusion module.

[0220] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0221] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the scene graph generation method supporting the historical and cultural block scene in the above embodiment; the processor may load and execute the following steps of one or more instructions in the computer-readable storage medium:

[0222] S1. Refer to existing datasets, such as the ImageNet dataset and the coco dataset, collect street view images of relevant historical and cultural blocks, take into account the data characteristics of street view images of historical and cultural blocks, and use image annotation tools to construct scene graphs of historical and cultural blocks to generate datasets;

[0223] S2. Use the historical and cultural block scene graph dataset as training data to train the landscape semantic recognition model and extract the visual features, semantic features, and spatial features of the target object;

[0224] S3, relationship prediction model design, taking the co-occurrence frequency of target relationships as prior knowledge, integrating it into the feature level to enhance the feature representation effect of target relationships, and using attention mechanism and feature fusion method to predict target relationships;

[0225] S4. Taking continuous spatiotemporal street view images as input, the trained landscape semantic recognition model and target relationship prediction model are used to process the spatiotemporal continuous street view images in the target area to obtain a spatiotemporal continuous scene graph corresponding to each street view image.

[0226] S5. Fusing the generated spatiotemporal continuous scene graphs based on a scene graph fusion model. The scene graph fusion model includes: a predicate alignment module, an embedding learning module, an entity alignment module, and a scene graph fusion module.

[0227] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0228] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0229] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0230] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0231] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and should be understood that the protection scope of the present invention is not limited to such special statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for generating a scene graph supporting a historical and cultural block scene, characterized in that: The following steps are involved: S1. Refer to existing data sets, including ImageNet and Coco, collect street view images of relevant historical and cultural blocks, take into account the data characteristics of street view images of historical and cultural blocks, and use image annotation tools to construct scene graphs of historical and cultural blocks to generate data sets; S2. Use the historical and cultural block scene graph dataset as training data to train the landscape semantic recognition model and extract the visual features, semantic features, and spatial features of the target object; S3, relationship prediction model design, taking the co-occurrence frequency of target relationships as prior knowledge, integrating it into the feature level to enhance the feature representation effect of target relationships, and using attention mechanism and feature fusion method to predict target relationships; The sub-steps of S3 are as follows: S31. Integration of prior knowledge Count the co-occurrence frequencies between target objects in the historical and cultural block scene graph generation dataset; Calculate the prior co-occurrence frequency of target pair relationships and incorporate it into model learning as prior knowledge; S32, attention mechanism fusion features Introducing the attention mechanism to adaptively learn the attention weights of spatial features, semantic features, visual features, and prior information; According to the attention weight of the feature, the softmax function is used to calculate the specific weight, and various features are fused by weighted merging to form a fused feature representation; The classifier is used to predict the relationship category of the target pair by using the fusion attention-weighted multi-features and prior co-occurrence frequency; The output of the classifier is used to obtain the relationship probability through the softmax activation function; In the model training stage, the relationship prediction task is regarded as a multi-classification task, and the multi-classification cross entropy is used as the loss function. The gradient of the loss is back-propagated to continuously learn and optimize the model parameters. In the testing phase, forward calculation is performed based on the model parameters to predict the relationship category between the subject, object and target in the triple; S4, taking the continuous spatiotemporal street view images as input, using the trained landscape semantic recognition model and target relationship prediction model to process the spatiotemporal continuous street view images in the target area, and obtaining the spatiotemporal continuous scene graph corresponding to each street view image; S5, fusing the generated spatiotemporal continuous scene graph based on the scene graph fusion model; The design of the scene graph fusion model includes: predicate alignment module, embedding learning module, entity alignment module and scene graph fusion module; The sub-steps of S5 are as follows: S51, predicate alignment module: Use the trained model in S4 to process street view images, ensuring that the generated series of scene graphs have the same naming scheme, find the same predicates and provide a unified vector space for relation embedding; Fuse adjacent scene graphs into one and learn embeddings of relation triplets and attribute triplets; S52, embedded learning module: Use the TransE model to learn the structural embedding of entity alignment between scene graphs and minimize the objective function; The model focuses on the target predicate relation through the weighting parameters; The corrupted triplets are used as negative samples by replacing the head entity with a random entity or replacing its feature attributes with random feature attributes; Jointly learn structure embedding and attribute feature embedding, transform them into the same vector space, obtain the similarity between entities based on cosine similarity, and use the similarity threshold to remove entity pairs that cannot be aligned; S53, entity alignment module: Use the learned embedding features to align entities in adjacent scene graphs and calculate the similarity between entities; Use the similarity threshold to remove dissimilar entity pairs and obtain the expected alignable entity pairs; S54, Update large scene graph Initialize an empty scene graph, traverse the entity pairs that were previously matched and determined to be the same entity, and merge the entity pairs into one entity; The relationship connection in the new scene graph is determined according to the connection relationship between the entities in the old scene graph, and the execution is repeated until all matching entity pairs are completed.

2. The method for generating a scene graph supporting a historical and cultural block scene according to claim 1, characterized in that: The sub-steps of S1 are as follows: S11. Collection and organization of street view images: Select historical and cultural blocks as research objects, collect street view images in a spatiotemporal continuous manner, and organize the image data; S12. Use of image annotation tools: Use professional image annotation tools to annotate the targets in street view images in the form of target frames, marking the specific location and shape of the targets; S13, target relation triple annotation: Through a series of <subject, relationship, object> triplets, the target relationship in the street view image is described in the form of a graph structure; Add target box annotations to the target objects and annotate the relationship between targets in the form of triplets; S14. Target selection and labeling: In terms of targets, select target objects with historical and cultural characteristics, including: historical and cultural signs, traditional specialty store signs, promotional posters, special decorations, couplets, iconic stone tablets, and special landmarks; At the same time, common features also need to be marked, including: road traffic signs, drainage outlets, roadblocks, debris, tricycles, and street lights; S15. Scene graph construction: According to the annotated target boxes and relationship triplet sets, instantiate the composition to form a street view image scene graph; In the scene graph, the graph nodes represent the ground objects in the image, the edges represent the relationship between the objects, and there is a one-to-one correspondence between the nodes and the object box annotations; The relationships in the scene graph include topology, direction, distance, spatial relationships, and semantic relationships.

3. The method for generating a scene graph supporting a historical and cultural block scene according to claim 1, characterized in that: The sub-steps of S2 are as follows: S21. Select encoder and decoder: ResNet-r50 is selected as the encoder to extract high-dimensional features of street view images; The features extracted by the encoder are decoded using a semantic segmentation decoder to distinguish the semantic differences between different landscape elements and accurately extract various landscape elements in the historical core area; S22. Introducing polarized attention mechanism: Based on the existing encoder-decoder framework, a polarized attention mechanism is introduced to enhance the model's ability to distinguish different landscape elements. S23. Transfer learning: Migrate the general visual information and prior information about street scenes in the ImageNet dataset and the coco dataset to the encoder, so as to train the network to learn the specific features of street scenes in historical and cultural blocks based on general street scene knowledge; The encoder’s transfer learning process includes pre-training on the ImageNet dataset and fine-tuning on the Cityscapes dataset and the Historic Core Street Scene dataset; Optimize the parameters of all models using stochastic gradient descent optimizer; S24, feature extraction: The deep feature map of street view images is extracted using CNN pre-trained with landscape semantic recognition model; A region alignment algorithm is used to extract visual features that characterize object pairs and their relationships; Considering the absolute and relative spatial relationships between targets, the spatial features are quantified using the position coordinates, overlap, and distance of the target boxes. The Word2vec model is used to convert target category words into high-dimensional vectors as semantic features.

4. The method for generating a scene graph supporting a historical and cultural block scene according to claim 3, characterized in that: The transfer learning process of the encoder in S2 is expressed as: Among them, P fe Represents the encoder parameters trained using the ImageNet dataset and the coco dataset, which contains general visual image information D img , and used for traditional image classification T f Task; Through transfer learning, in D img Based on the study of the characteristics of the landscape elements in the historical core area hcsv , and change the parameter task to feature extraction T f ′; So far, the encoder parameters P with general visual information and core area landscape element information are obtained fe ′; The transfer learning process of the semantic segmentation decoder is expressed as: Where P and P′ represent the semantic segmentation decoder parameters before and after migration, respectively, where P is trained using the Cityscapes dataset, which has street view information of European cities. fe And used for street object recognition task Seg cs ; In P fe Based on the training network learning historical core area street view information P fe ′, and change the model task to identify landscape features, thus obtaining the transferred semantic segmentation decoder parameters P′.

5. The method for generating a scene graph supporting a historical and cultural block scene according to claim 1, characterized in that: The visual features in S24 are expressed as: V ij =f vis (I,X i ,X j ,X ij ) Where V ij is the target pair relationship (o i , o j ), f vis is a function used to extract visual features, I is the input street view image, X i and X i Based on the target box b i and b j The extracted visual features, X ij Visual features representing regions of corresponding relationships; The semantic features are expressed as: S ij =f scm (p i ,p j ,I ij ,U ij ,D ij ) In the formula, S ij is the target pair relationship (o i , o j ), f scm is a function used to extract semantic features, p i and p j is the position coordinate of the corresponding target frame, U ij is the intersection-over-union ratio of the corresponding target box, D ij is the distance between corresponding target boxes; The spatial feature is expressed as: SP ij =f spa (V i ,V j ) Where, SP ij To represent the semantic features of the target pair, V i and V j is the target category O i and O j Word embedding features extracted through language model.

6. The method for generating a scene graph supporting a historical and cultural block scene according to claim 5, characterized in that: The prior co-occurrence frequency of the target pair relationship in S31 is expressed as: In the formula, P(r ij |O i ,O j ) are two given features O i and O j The relationship ij The prior co-occurrence frequency, N ij Given O on the training data set i and O j The relationship ij The prior co-occurrence count of N is the total number of target pair relations in the training dataset.

7. The method for generating a scene graph supporting a historical and cultural block scene according to claim 6, characterized in that: In S32, the relationship category of the target pair is predicted by the classifier using the fused attention-weighted multi-features and the prior co-occurrence frequency; the output of the classifier is used to obtain the relationship probability through the softmax activation function; The calculation process is expressed as: R ij =σ(W r F ij +b r ) Among them, R ij For the target pair (O i , O j ), σ(·) the activation function of the classifier, W r and b r are the weight and bias parameters of the classifier, respectively, and F ij is the fused feature; The loss function is defined as follows: Among them, Ω represents the parameters that need to be trained for the entire relationship prediction model, and y ij (i) Represents the binary label of the relationship between subject i and object j in the triplet in category c, R ij (c) The probability that the relationship category predicted by the representative model for a triple is c, the subject is i, the object is j, and C is the total number of relationship categories.

8. The method for generating a scene graph supporting a historical and cultural block scene according to claim 7, characterized in that: In S5, in order to learn structural embedding, the following objective function is minimized: Among them, T r is a set of valid relation triples, T′ r is the set of damaged triples, count(r) is the number of times relation r appears, |T| is the merged triple G 1，2 The total number of triplets in the . Adding the weight parameter α helps the model focus more on the target relationship. To learn attribute feature embedding, the following objective function is minimized: T a ={<h,rr,a>∈G};f(t a )=||h+r-f a (a)|| T′ a ={<h′,r,a>|h′∈E}∪{<h,r,a′>|a′∈A} T a Valid attribute triples T′ from the training dataset a is a set of damaged triples, A is a set of attributes in G, and the damaged triples are used as negative samples by replacing the head entity with a random entity or replacing its feature attributes with random feature attributes. a ) is based on the embedding of the head entity h, the embedding of the relation r, and the representation vector of the attribute (using the combination function f a (t a ) calculates the credibility score obtained; Joint learning of structural embedding and attribute feature embedding to minimize the objective function J SIM , embed the attribute features into h ce With h se The structural embedding is transformed into the same vector space as follows: In the formula, cos(h se ,h ce ) is the vector h ce and h se The cosine similarity of The overall objective function of joint learning of structure embedding and attribute feature embedding is: J=J SE +J CE +J SIM The entity alignment module calculates the similarity of triples to perform entity alignment, and the formula is as follows: Where, entity h1∈G1, calculates the similarity between all entities h2∈G2 in the scene graph G2; <h2,h map > are the expected alignable triplets, and the similarity threshold β is used to remove the triplets that are dissimilar and cannot be aligned.

9. A scene graph generation system supporting historical and cultural block scenes, characterized by: The system can be used to implement the scene graph generation method supporting the historical and cultural block scene as described in any one of claims 1 to 8, specifically including: Dataset construction module: refer to the ImageNet dataset and the coco dataset to construct a historical and cultural block scene graph generation dataset; Landscape semantic recognition module: Extract visual features, semantic features, and spatial features of the target object; Relationship prediction module: Incorporate target relationship co-occurrence frequency as prior knowledge; Use attention mechanism and feature fusion method to predict target relationship; Scene graph generation module: Take continuous spatiotemporal street view images as input; The street view images are processed using the landscape semantic recognition module and the relationship prediction module to generate a spatiotemporal continuous scene graph corresponding to each street view image; The scene graph fusion module includes the following sub-modules: Predicate alignment module: fuses adjacent scene graphs into one and performs embedding learning of relation triplets and attribute triplets; Embedding learning module: learns structural embeddings, controls the model to focus on aligned triplets, and performs entity alignment; Entity alignment module: uses the learned embedding features to align entities in adjacent scene graphs and calculate the similarity between entities; Update large scene graph module: Determine the relationship connection in the new scene graph based on the connection relationship between the entities in the old scene graph until all matching entity pairs are merged.

Images