A scene graph semantic information extraction method and system for graph query

Abstract

The application provides a scene graph semantic information extraction method and system for a graph query. The method analyzes the entity, relationship and attribute structure in the Cypher language of a user query, and inputs the structure information as a prompt input of a scene graph generation model. The scene graph fuses the graph structure information of the query requirement to generate simple scene graph data, aligns the scene graph data with the original data model in the graph database, stores the scene graph, and finally realizes efficient query of semantic information in a picture by using a graph query language. The application is guided by the user query requirement, converts a picture into graph structure data by using a scene graph extraction model, and performs data alignment, solves the short board that a previous graph query language cannot query a picture, and innovatively and efficiently extracts scene semantic information according to the user query requirement, realizes efficient query, and improves the shortcomings of the previous scene graph extraction model, such as long extraction time and waste of storage space.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, graph data, big data technology, and database technology, and proposes a method and system for extracting semantic information from scene graphs for graph queries. Background Technology

[0002] With the rapid development of the internet and the advancement of information technology, graph data and big data technologies are being applied more and more widely in various fields. Graph data is a data structure that uses nodes and edges to represent entities and their relationships, and can be used to describe complex relationships and network structures. Images, as unstructured data, contain rich information about entities and relationships. However, due to the complexity and massive scale of image data, efficiently extracting useful semantic information from it according to user query needs has become a challenge.

[0003] In the field of graph databases, Cypher is a query language that uses graphs as its data model and is widely used. As a representative graph query language, Cypher provides a concise yet powerful way to query and manipulate graph data. Cypher's syntax is similar to natural language, making it easy to understand and learn. It uses ASCII characters to represent nodes and edges in the graph and describes query conditions through a series of pattern matches. However, the existing Cypher language cannot be directly applied to querying image data. Therefore, it is necessary to leverage graph-structured data to query images. This involves transforming unstructured images into graph-structured data and dynamically generating scene graphs within the images based on the user's query requirements. The data structure of these scene graphs is then aligned with the predefined graph data model in the graph database to achieve efficient and real-time querying and storage.

[0004] Traditional graph query methods primarily rely on graph-based query languages ​​such as SPARQL. However, these languages ​​cannot directly query semantic information within images, nor can they dynamically extract semantic information based on user query requirements. This invention, based on parsing the user's graph query language query requirements, utilizes a scene graph extraction model to extract entity, relation, and attribute information from images. It then employs data alignment technology to integrate the extracted semantic information from the scene graph with a predefined graph data model in a graph database. The aligned data is stored in the graph database, and the query results are returned to the user, thus enabling the querying of image semantic information using graph query languages. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for extracting scene graph semantic information for graph queries. Guided by user query needs, the method uses a scene graph extraction model to convert images into graph structure data and perform data alignment. This addresses the shortcomings of previous graph query languages ​​in that they could not query scene semantic information in images. Furthermore, it efficiently extracts scene semantic information according to user query needs to achieve efficient querying, thus improving upon the shortcomings of previous scene graph extraction models, such as long extraction time and wasted storage space.

[0006] The technical solution adopted by the present invention to achieve the above objectives is as follows: In a first aspect, the present invention provides a method for extracting semantic information from a scene graph for graph-oriented queries, the steps of which include: 1) Parse the graph structure information contained in the user query, which includes the entity, relation, and attribute structure in the Cypher language; 2) Construct a scene graph generation model, using the parsed graph structure information as the prompt input for the scene graph generation model to generate initial scene graph data; 3) Integrate the input prompts with the generated initial scene graph data to generate query-driven scene graph data; 4) Align the scene graph data with the original data model of the graph database, including aligning the nodes, relationships, and attributes of the scene graph with the nodes, relationships, and attributes of the original attribute graph of the graph database respectively; 5) After aligning the scene graph data with the original data model of the graph database, store and query the scene graph in the graph database, and realize the query of semantic information in the image using the graph query language.

[0007] Furthermore, step 1) involves parsing the graph structure information contained in the user query, which includes the following steps: Construct lexical analysis, syntactic analysis, and semantic analysis modules; Based on the syntax rules of the Cypher language, the nodes and relationships contained in the query are identified, their attribute information is extracted, and the attribute structure of the entity is constructed. Identify the relationships between entities in the query, and identify the associations between nodes and relationships based on the syntax rules and semantic information in the query.

[0008] Further, step 2) constructs a scene graph generation model to generate initial scene graph data, including the following steps: Based on image preprocessing, object detection and semantic segmentation models are used to detect and segment objects in the image; The visual features of the objects are extracted using a convolutional neural network model to obtain the feature vector representation of each object. The constructed scene graph is inferred and analyzed using a graph neural network model. Based on the extracted object entity features, each object entity is used as a node in the scene graph, and the relationships between object entities are used as edges in the scene graph. The parsed graph structure information is converted into graph data, which includes node features, edge features, and graph topology information. This data is then input into a graph neural network model to generate initial scene graph data.

[0009] Furthermore, the object detection and semantic segmentation models used are specifically the Mask R-CNN model, the convolutional neural network model used is specifically the ResNet50 model, and the graph neural network model used is specifically the GAT model.

[0010] Furthermore, step 3) integrates the prompted input with the generated initial scene graph data to generate query-driven scene graph data, including the following steps: The parsed graph structure information is fused with the results of object detection, segmentation, and feature extraction. The entities and relationships described in Cypher language are converted into nodes and edges in the scene graph, which correspond to the results of object detection and segmentation. The visual features of the object entities obtained from object detection and segmentation are integrated with the parsed graph structure information. The visual features are matched with the attributes and relationships described in the Cypher language, and corresponding feature information is added to each node and edge. Based on the results of data structure fusion and feature integration, the system outputs node features, edge features, and graph topology information, ultimately generating query-driven scene graph data.

[0011] Furthermore, step 4) aligns the nodes of the scene graph with the nodes of the original attribute graph in the graph database, including the following steps: Analyze the node types in the scene graph to determine the entity or concept represented by each node type; Define mapping rules, create a mapping table to map the node types in the scene graph to the node types in the graph database, and match the node identifiers in the scene graph with the node identifiers in the graph database; Graph matching algorithms are used to automatically align nodes. Node type mismatches, many-to-one or one-to-many alignment relationships, node type conflicts, identifier conflicts, and abnormal data that do not meet expectations are handled according to defined exception handling rules.

[0012] Furthermore, step 4) aligns the relationships in the scene graph with the relationships in the original attribute graph of the graph database, including the following steps: Determine the correspondence between entities and their relationships in the scene graph data and the data model of the graph database, and analyze the relationships and semantic meanings between entities; Create a relation mapping table to record the correspondence between entity relations in the scene graph and entity relations in the graph database. The mapping table includes the entity relation name, relation type, semantics in the scene graph, and the correspondence with the entity relations in the graph database. To handle exceptional cases of missing or conflicting relationships, default values ​​are defined to handle missing relationships and to handle situations where the same relationship has different semantics in the scene graph and the graph database.

[0013] Furthermore, when creating the relation mapping table, the rule template and knowledge base are combined for pre-definition and matching. At the same time, the name and description of the entity are input into the large language model. The large language model obtains the semantic representation of the entity. For multiple names or aliases of the same entity, semantic information and contextual information are compared to achieve entity unification and entity disambiguation, avoid creating unnecessary new relation types, and achieve automated mapping at runtime.

[0014] Furthermore, step 4) aligns the attributes of the scene graph with the attributes of the original attribute graph in the graph database, including the following steps: Identify the attributes of nodes in the scene graph and graph database, and analyze the data type and semantics of the attributes in the scene graph; Create an attribute mapping table to record the correspondence between attributes in the scene graph and attributes in the graph database. The attribute mapping table includes the scene graph attribute name, data type, semantics, and correspondence with the attributes in the graph database. The system handles exceptions such as missing or conflicting attributes, and finally verifies the attribute alignment results.

[0015] Secondly, the present invention also provides a scene graph semantic information extraction system for graph query, including a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the steps of the above method.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention, based on previous scene graph extraction models, adds a step of parsing user query requirements, enabling fine-tuning of the scene graph model's output, reducing scene graph extraction time, and saving storage space. Furthermore, data alignment technology is used to adjust the scene graph's data structure, ensuring the generated graph structure conforms to the original graph data model in the graph database, thus enabling the storage and management of semantic information in the scene graph. Based on this design, users can efficiently query semantic information from images using Cypher.

[0017] Compared to existing scene graph extraction models and graph query technologies, this invention fine-tunes the graph structure data generated from the scene graph based on user query requirements, making it more efficient and convenient, and ensuring that the semantic information of the generated scene graph better meets actual query needs. Furthermore, this invention addresses the shortcoming of existing graph query languages ​​that cannot directly query scene semantic information within images, achieving scene graph data storage and retrieval through data alignment. Attached Figure Description

[0018] Figure 1 This is a flowchart of a method for extracting semantic information from a graph-oriented query scenario according to the present invention. Detailed Implementation

[0019] To make the various technical features, advantages, or effects of the present invention more apparent and understandable, a detailed description is provided below in conjunction with the accompanying drawings.

[0020] This invention proposes a method for extracting semantic information from scene graphs based on graph queries. The process of this method is as follows: Figure 1 As shown, it includes the following steps: First, the graph structure information contained in the user's query needs is read and parsed. By constructing lexical analysis, syntax analysis and semantic analysis modules, the entity, attribute and relation information in the Cypher query is identified.

[0021] The process of parsing the graph structure information contained in a user query includes the following steps: ① Constructing lexical analysis, syntax analysis, and semantic analysis modules; ② Identifying the nodes and relationships contained in the query according to the syntax rules of the Cypher language, extracting their attribute information, and constructing the attribute structure of the entities; ③ Identifying the relationships between entities in the query, and identifying the associations between nodes and relationships according to the syntax rules and semantic information in the query.

[0022] Secondly, the parsed graph structure information is used as the prompt input for the scene graph generation model. The scene graph generation model is constructed by following these steps: using the Mask R-CNN model for object detection and segmentation, using the ResNet50 model for feature extraction and representation learning, using the GAT model for scene graph construction and inference, and designing the format of the input data.

[0023] The process involves: ① using the Mask R-CNN object detection and semantic segmentation model to detect and segment objects in images based on image preprocessing; ② using the ResNet50 convolutional neural network model to extract visual features of object entities, obtaining feature vector representations for each object entity; ③ using the GAT graph neural network model to reason and analyze the constructed scene graph, with each object entity serving as a node in the graph based on the extracted object entity features, and the relationships between object entities serving as edges in the graph; ④ converting the parsed graph structure information into graph data, which includes node features, edge features, and graph topology information, and inputting this data into the GAT graph neural network model to construct a scene graph generation model.

[0024] Furthermore, the scene graph fusion query requires graph structure information to generate concise scene graph data. It matches the entities and relationships obtained from parsing Cypher semantics with the results of object detection and segmentation, integrates the visual features obtained from object detection and segmentation, adds feature information to each node and edge, and outputs query-driven scene graph data based on the results of data structure fusion and feature integration.

[0025] Specifically, ① the parsed graph structure information is fused with the results of object detection, segmentation, and feature extraction, converting entities and relationships described in Cypher language into nodes and edges in the graph, corresponding to the results of object detection and segmentation; ② the visual features of object entities obtained from object detection and segmentation are integrated with the parsed graph structure information, matching the visual features with the attributes and relationships described in Cypher language, and adding corresponding feature information to each node and edge; ③ based on the results of data structure fusion and feature integration, node features, edge features, and graph topology information are output, ultimately generating query-driven scene graph data.

[0026] Next, the scene graph data is aligned with the original data model in the graph database, including three aspects: node alignment, relationship alignment, and attribute alignment. This requires analyzing the semantics and types of the nodes, relationships, and attributes in the scene graph, combining rule matching, knowledge base, and large language model to achieve entity unification and entity disambiguation, automatically creating mapping tables, running graph matching algorithms to match graph structures, and finally handling abnormal situations.

[0027] Aligning the nodes of the scene graph with the nodes of the original attribute graph in the graph database includes the following steps: ① Analyzing the node types in the scene graph to determine the entity or concept represented by each node type; ② Defining mapping rules and creating a mapping table to map the node types in the scene graph to the node types in the graph database, ensuring that the node identifiers in the scene graph match the node identifiers in the graph database; ③ Using a graph matching algorithm to automatically align the nodes, handling node type mismatches, many-to-one or one-to-many alignment relationships, node type conflicts, identifier conflicts, and unexpected abnormal data through defined exception handling rules.

[0028] Aligning the relationships in the scene graph with the existing attribute graph in the graph database involves the following steps: ① Determining the correspondence between entities and their relationships in the scene graph data and the data model of the graph database, and analyzing the associations and semantic meanings between entities; ② Creating a relationship mapping table to record the correspondence between entity relationships in the scene graph and entity relationships in the graph database. The mapping table includes the entity relationship name, relationship type, semantics in the scene graph, and the correspondence with the entity relationship in the graph database. When creating the relationship mapping table, pre-definition and matching are performed using rule templates and knowledge bases. At the same time, the entity name and description are input into the large language model. The large language model obtains the semantic representation of the entity. For multiple names or aliases of the same entity, semantic information and contextual information are compared to achieve entity unification and entity disambiguation, avoiding the creation of unnecessary new relationship types, and achieving automated mapping at runtime; ③ For handling abnormal cases of missing and conflicting relationships, default values ​​are defined to handle missing relationships and to handle cases where the same relationship has different semantics in the scene graph and the graph database.

[0029] Aligning the attributes of the scene graph with the attributes of the original attribute graph in the graph database includes the following steps: ① Identifying the attributes of nodes in the scene graph and the graph database, and analyzing the data type and semantics of the attributes in the scene graph; ② Creating an attribute mapping table to record the correspondence between the attributes in the scene graph and the attributes in the graph database. This attribute mapping table includes the attribute name, data type, semantics of the scene graph attribute, and the correspondence with the attribute in the graph database; ③ Handling abnormal situations such as missing attributes and attribute conflicts, and finally verifying the attribute alignment results.

[0030] Finally, by aligning the scene graph data with the original data model of the graph database, the scene graph can be stored and queried in the graph database, ultimately enabling efficient querying of semantic information in images using graph query language.

[0031] This invention also provides a scene graph semantic information extraction system for graph-oriented queries, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the above method.

[0032] The present invention proposes a method and system for extracting semantic information from scene graphs for graph-oriented queries. This method and system can be used in fields such as image search engine optimization and intelligent video surveillance systems. Specific examples are as follows: 1) Image Search Engine Optimization: This invention can be applied to image search engines, improving the accuracy and relevance of search results by efficiently querying the semantic information of images. When a user uploads or inputs a query image into the image search engine, the system first uses the technology involved in this invention to extract the semantic information from the image, identifying objects, scenes, and other relevant semantic information. The extracted image semantic information is used to construct a semantic index, which contains a description of the image's semantic information for subsequent image search and matching. The system uses the constructed semantic index to perform similar image retrieval. By comparing the semantic information of the user's query image with the semantic description in the index, the system can quickly find images similar to the query image. After retrieving similar images, the system sorts the results according to the degree of semantic matching, thereby improving the accuracy and relevance of search results.

[0033] 2) Intelligent Video Surveillance System: In an intelligent video surveillance system, this invention can help efficiently query and manage semantic information of scenes in surveillance videos, thereby helping security agencies or enterprises to more quickly locate specific scenes or events and analyze video content more accurately. The intelligent video surveillance system can realize intelligent video retrieval functions. Users can input keywords or upload query images to retrieve relevant surveillance video clips. The system will use semantic information for intelligent matching and retrieval, and can quickly locate relevant video clips or scenes. As user query needs change, the system can dynamically extract different semantic information to meet different query needs, better adapting to users' real-time query requirements through dynamic semantic information extraction. During real-time monitoring, the system continuously uses semantic information for scene analysis and monitoring. Once a specific scene or event is detected, such as a gathering of people or abnormal behavior, the system can immediately provide feedback.

[0034] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A method for extracting scene graph semantic information for graph query, characterized in that, Includes the following steps: 1) Parse the graph structure information contained in the user query, which includes the entity, relation, and attribute structure in the Cypher language; 2) Construct a scene graph generation model, using the graph structure information as the prompt input for the scene graph generation model to generate initial scene graph data; 3) Integrate the input prompts with the generated initial scene graph data to generate query-driven scene graph data, including the following steps: Perform data structure fusion between the parsed graph structure information and the results of object detection, segmentation, and feature extraction in the image; convert entities and relationships described in Cypher language into nodes and edges in the scene graph, corresponding to the results of object detection and segmentation; integrate the visual features of the object entities obtained from object detection and segmentation with the parsed graph structure information; match the visual features with the attributes and relationships described in Cypher language; add corresponding feature information to each node and edge; based on the results of data structure fusion and feature integration, output node features, edge features, and graph topology information, ultimately generating query-driven scene graph data. 4) Align the scene graph data with the original data model of the graph database, including aligning the nodes, relationships, and attributes of the scene graph with the nodes, relationships, and attributes of the original attribute graph of the graph database respectively; 5) After aligning the scene graph data with the original data model of the graph database, store and query the scene graph in the graph database, and realize the query of semantic information in the image using the graph query language.

2. The method of claim 1, wherein, Step 1) involves parsing the graph structure information contained in the user query, which includes the following steps: Construct lexical analysis, syntactic analysis, and semantic analysis modules; Based on the syntax rules of the Cypher language, the nodes and relationships contained in the query are identified, their attribute information is extracted, and the attribute structure of the entity is constructed. Identify the relationships between entities in the query, and identify the associations between nodes and relationships based on the syntax rules and semantic information in the query.

3. The method of claim 1, wherein, Step 2) Construct a scene graph generation model to generate initial scene graph data, including the following steps: Based on image preprocessing, object detection and semantic segmentation models are used to detect and segment objects in the image; The visual features of the objects are extracted using a convolutional neural network model to obtain the feature vector representation of each object. The graph neural network model is used to reason and analyze the constructed scene graph. Based on the extracted object entity features, each object entity is used as a node in the graph, and the relationship between object entities is used as an edge in the graph. The parsed graph structure information is converted into graph data, which includes node features, edge features, and graph topology information. This data is then input into a graph neural network model to generate initial scene graph data.

4. The method of claim 3, wherein, The object detection and semantic segmentation models used are specifically the Mask R-CNN model, the convolutional neural network model used is specifically the ResNet50 model, and the graph neural network model used is specifically the GAT model.

5. The method of claim 1, wherein, Step 4) aligns the nodes of the scene graph with the nodes of the original attribute graph in the graph database, including the following steps: Analyze the node types in the scene graph to determine the entity or concept represented by each node type; Define mapping rules, create a mapping table to map the node types in the scene graph to the node types in the graph database, and match the node identifiers in the scene graph with the node identifiers in the graph database; Graph matching algorithms are used to automatically align nodes. Node type mismatches, many-to-one or one-to-many alignment relationships, node type conflicts, identifier conflicts, and abnormal data that do not meet expectations are handled according to defined exception handling rules.

6. The method of claim 1, wherein, Step 4) aligns the relationships in the scene graph with the relationships in the original attribute graph of the graph database, including the following steps: Determine the correspondence between entities and their relationships in the scene graph data and the data model of the graph database, and analyze the relationships and semantic meanings between entities; Create a relation mapping table to record the correspondence between entity relations in the scene graph and entity relations in the graph database. The mapping table includes the entity relation name, relation type, semantics in the scene graph, and the correspondence with the entity relations in the graph database. To handle exceptional cases of missing or conflicting relationships, default values ​​are defined to handle missing relationships and to handle situations where the same relationship has different semantics in the scene graph and the graph database.

7. The method of claim 6, wherein, When creating a relation mapping table, pre-definition and matching are performed using rule templates and knowledge bases. At the same time, the names and descriptions of entities are input into the large language model, which obtains the semantic representation of the entities. For multiple names or aliases of the same entity, semantic and contextual information are compared to achieve entity unification and entity disambiguation, avoiding the creation of unnecessary new relation types and enabling automated mapping at runtime.

8. The method of claim 1, wherein, Step 4) aligns the attributes of the scene graph with the attributes of the original attribute graph in the graph database, including the following steps: Identify the attributes of nodes in the scene graph and graph database, and analyze the data type and semantics of the attributes in the scene graph; Create an attribute mapping table to record the correspondence between attributes in the scene graph and attributes in the graph database. The attribute mapping table includes the scene graph attribute name, data type, semantics, and correspondence with the attributes in the graph database. The system handles exceptions such as missing or conflicting attributes, and finally verifies the attribute alignment results.

9. A scene graph semantic information extraction system for graph-oriented queries, characterized in that, It includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the steps of the method according to any one of claims 1-8.

Images