A 3D Semantic Scene Graph Prediction Method Based on Multimodal Spatial Hierarchical Knowledge

Abstract

The present invention discloses a three-dimensional semantic scene graph prediction method based on multi-modal spatial hierarchical knowledge, including: S1: constructing a symbolic knowledge graph based on a spatial hierarchical structure; S2: visual context encoding under a symbolic knowledge guidance strategy; S3: multi-modal knowledge extraction based on text symbolic knowledge and visual context features; S4: object and relationship detection guided by multi-modal knowledge. By designing a graph inference network, symbolic knowledge from the text modality and scene visual context features from the visual modality are fused in the symbolic knowledge graph to achieve the extraction and learning of multi-modal spatial hierarchical knowledge, and using the object categories in the visual scene as indexes to extract corresponding multi-modal knowledge features, realizing object and relationship detection enhanced by multi-modal knowledge, and improving the algorithm performance and efficiency by introducing more expressive multi-modal knowledge, making the three-dimensional scene graph prediction method of the present invention have the advantages of high precision, high efficiency, strong robustness, etc.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and computer vision, and particularly relates to a three-dimensional scene graph detection and generation algorithm based on multi-modal spatial hierarchical knowledge. Background Art

[0002] Research on vision scene understanding centered on objects, such as semantic segmentation and object detection for three-dimensional point cloud scenes, is in a rapid development stage. However, the three-dimensional world not only contains objects but also a large number of relationships between objects. Most three-dimensional vision scene understanding algorithms completely ignore the implicit relationships between objects in the three-dimensional point cloud scene. Therefore, comprehensively utilizing the objects and relationships in the three-dimensional scene and systematically describing the semantic information of the three-dimensional point cloud scene has become one of the important topics in the field of computer vision.

[0003] The semantic scene graph describes the objects and relationships in a two-dimensional or three-dimensional vision scene in the form of a directed graph, and has the characteristics of being easy to understand and expand. Since its proposal, it has attracted much attention from researchers. In the field, the semantic scene graph for three-dimensional vision scenes is generally called a three-dimensional scene graph.

[0004] Existing three-dimensional scene graph prediction methods only rely on scene visual information to detect complex relationships between objects, ignoring the background information and prior knowledge related to three-dimensional vision scenes, and cannot meet the requirements of downstream tasks for algorithm performance and robustness. How to extract general background knowledge from complex and variable three-dimensional point cloud data and apply it to three-dimensional scene graph prediction methods plays a crucial role in promoting the development of unmanned operation technology industries such as intelligent operation robots, and has extremely high practical value and development prospects in fields such as industry and medicine.

[0005] Therefore, there is an urgent need for a new three-dimensional semantic scene graph prediction method. Summary of the Invention

[0006] The purpose of the present invention is: to overcome the problems of the existing technology, the present invention discloses a three-dimensional semantic scene graph prediction method based on multi-modal spatial hierarchical knowledge. By setting the method of the present invention, the three-dimensional scene graph prediction method in the present invention has high precision, high efficiency, and strong robustness.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A three-dimensional semantic scene graph prediction method based on multi-modal spatial hierarchical knowledge, the three-dimensional semantic scene graph prediction method includes the following steps:

[0009] S1: Construction of a symbolic knowledge graph based on a spatial hierarchical structure;

[0010] S2: Visual context encoding under a symbolic knowledge guidance strategy;

[0011] S3: Multimodal knowledge extraction based on text symbol knowledge and visual context features;

[0012] S4: Multimodal knowledge-guided object and relation detection.

[0013] According to a preferred embodiment, step S1 includes:

[0014] S11: Perform hierarchical classification processing on each object category in the object category set O in the given data set through a multi-layer perceptron;

[0015] S12: Based on the hierarchical classification information of object categories and the common sense knowledge graph ConceptNet, the common sense knowledge graph ConceptNet is screened and cleaned, and the filtered common sense knowledge graph ConceptNet is defined as Node Set Contains only common object categories in 3D visual scenes, a set of directed edges Represent relationships between common object categories;

[0016] S13: Based on the hierarchical classification of object categories, the GloVe word embedding vectors of the corresponding object categories and the embedding vectors of the corresponding levels are concatenated to represent the common sense knowledge graph. The embedding vector of each object node in the high-dimensional vector space;

[0017] When the knowledge graph The hierarchical classification results of any two object nodes in are cross-layer, and the knowledge graphs of the two nodes are If there is at least one path with 2 hops, a supporting edge is added between the two nodes to represent the physical support relationship between the two object categories, and the corresponding word embedding vector in GloVe is used to represent Characteristics of directed edges in ;

[0018] Finally, the updated common sense knowledge graph ConceptNet knowledge graph As a symbolic knowledge graph

[0019] According to a preferred embodiment, step S11 further includes: dividing each object category into three levels: bottom level, middle level, or top level;

[0020] The bottom layer only contains one type of object, the ground. The ground provides physical support for the middle-layer object categories, and the middle-layer object categories provide physical support for the object categories in the top layer.

[0021] According to a preferred embodiment, the data screening and cleaning of the common sense knowledge graph ConceptNet in step S11 includes:

[0022] Extract common object categories in the 3D visual scene from the SUNRGBD, Scannet, and 3RScan datasets, and filter out the relation triples in the commonsense knowledge graph ConceptNet that only contain these object categories.

[0023] According to a preferred embodiment, step S2 includes:

[0024] S21: Based on the 3D point cloud scene, under the guidance of the symbolic knowledge graph that depicts the spatial hierarchical structure knowledge of the 3D point cloud scene construct the hierarchical visual graph of the input 3D visual scene

[0025] S22: Use VoteNet as the object detection network to generate the bounding box information of several candidate object instances and the candidate object set corresponding to the corresponding object categories. Input the bounding box parameters of each object, and map the bounding box parameters to a high-dimensional vector space through a multi-layer perceptron to obtain the object spatial features;

[0026] According to the bounding box parameters v of the object i extract the 3D point set corresponding to the object instance, and use Point CloudTransformer as the visual feature extraction network to extract the visual features of each object in the input scene ;

[0027] Based on the object category corresponding to each candidate object instance, obtain the word embedding vector of the object category through GloVe as the semantic feature of the object;

[0028] Cascade the visual features, spatial features, and semantic features of the object to obtain the initial features of each node in the node set of the hierarchical visual graph ;

[0029] Moreover, when there is at least one support edge between the nodes of any two object nodes in the hierarchical visual graph in the symbolic knowledge graph , add a directed edge between the two object nodes in the hierarchical visual graph and add it to the directed edge set representing the physical support relationship between the two object instances in the input 3D point cloud scene. Through cascading the features of the two participating nodes and calculating through a multi-layer perceptron to obtain the initial features of each edge in ;

[0030] S23: Given the hierarchical visual graph adopt a graph neural network with region perception ability to iteratively update the hidden features of the nodes and directed edges in the hierarchical visual graph ; The local area in

[0031] At time t, node features are enhanced according to each node and the local area where it is located; and the features of the directed edges in each local area are enhanced;

[0032] After feature enhancement, the region-aware graph neural network performs information transmission for each node and directed edge in the hierarchical visual graph After iteration, the hidden features of the object nodes in the graph neural network are taken as the visual context features of the object nodes The hidden features of the directed edges are used as the visual context features of the directed edges

[0033] According to a preferred embodiment, in step S23, at time t, node features are enhanced according to each node and the local area where it is located:

[0034]

[0035] Where represents the hidden feature of object node i at time t, and N r (i) represents other object nodes in the same local area as i, represents the enhanced hidden feature of the node, and ψ represents the forward propagation network;

[0036] The feature enhancement of the directed edges in each local area is:

[0037]

[0038] Where represents the hidden feature of the directed edge between nodes i and j at time t, represents the enhanced hidden feature of the directed edge.

[0039] According to a preferred embodiment, in step S23, after feature enhancement, the region-aware graph neural network performs information transmission for each node and directed edge in the hierarchical visual graph :

[0040]

[0041]

[0042] Where and respectively represent the information aggregated at time t for updating node i and the directed edge between nodes i and j, which is calculated by the following formula:

[0043]

[0044] where N v (i) represents the set of nodes adjacent to node i in the hierarchical visual graph and and respectively represent two different non - linear vector transformations;

[0045] Similarly, it can be calculated by the following formula:

[0046]

[0047] where and respectively represent two three - layer multi - layer perceptron networks.

[0048] According to a preferred embodiment, step S3 includes:

[0049] Based on the symbolic knowledge graph and the visual context features and using a graph reasoning network to integrate text knowledge and visual knowledge, and iteratively update the knowledge features of each node and edge in the symbolic knowledge graph in

[0050] After iteration, take the multi - modal knowledge features of each node and directed edge output by the graph reasoning network as the multi - modal knowledge of the object category and the multi - modal knowledge of the directed edge

[0051] According to a preferred embodiment, in step S3, the update process of the graph reasoning network is defined as follows:

[0052]

[0053]

[0054] where and respectively represent the multi - modal knowledge features of nodes and edges in the symbolic knowledge graph at time t in and respectively represent the information used to update node i and the directed edge between node i and j at time t, and its calculation process is defined as:

[0055]

[0056]

[0057] N k (i) represents the set of nodes adjacent to node i in the symbolic knowledge graph and and represent non - linear transformations.

[0058] According to a preferred embodiment, step S4 includes:

[0059] S41: Based on the visual context features in the hierarchical visual graph and and and the corresponding multimodal knowledge and and the visual context of the object node preliminarily predict the object categories corresponding to the objects in the input scene;

[0060] And according to the preliminary prediction results, screen out the node multimodal knowledge corresponding to the top three confidence object categories For the directed edges on the hierarchical visual graph According to the object category prediction results, select the top three confidence object categories for the two object nodes connected by the directed edge respectively, and select the corresponding directed - edge multimodal knowledge as the index

[0061] S42: Pass through two multi - layer perceptrons to respectively map the node visual context and the directed - edge visual context and the relevant node multimodal knowledge and the directed - edge multimodal knowledge after screening into the same vector space, and obtain the knowledge - enhanced scene comprehensive feature representing node i and the knowledge - enhanced scene comprehensive feature representing the directed edge between node i and j

[0062] S43: Based on the knowledge - enhanced features of the object nodes and the relationship edges and Connect each object node in the hierarchical visual graph pair - by - pair to obtain a fully - connected graph. The object - node features in the graph are initialized as the knowledge - enhanced features The directed - edge features are initialized as the knowledge - enhanced features When the relationship edge in the fully - connected graph does not appear in the hierarchical visual graph then it is initialized as the feature splicing of the participating object nodes;

[0063] Use a standard graph convolutional neural network to perform information transfer on the object nodes and relationship edges in the fully connected graph, realize knowledge reasoning for three-dimensional visual scenes, and after three rounds of iteration, output the object node features and the relationship edge features are respectively sent into the object classification network and the relationship classification network to obtain the object category probability distribution corresponding to the object node i and the relationship category probability distribution corresponding to the relationship edge between object nodes i and j

[0064] According to a preferred embodiment, in step S43, the output object node features and the relationship edge features are respectively sent into the object classification network and the relationship classification network:

[0065]

[0066]

[0067] In the formula, φ o and φ s respectively represent two forward propagation networks responsible for object and relationship classification, represents the object category probability distribution corresponding to the object node i, |O| represents the number of object categories in the dataset, [[ID=�1]]represents the relationship category probability distribution corresponding to the relationship edge between object nodes i and j, |R| represents the number of relationship categories in the dataset.

[0068] The foregoing main solution of the present invention and its various further selection solutions can be freely combined to form multiple solutions, all of which are solutions that can be adopted and claimed by the present invention. Those skilled in the art can understand that there are various combinations according to the prior art and common knowledge after understanding the solution of the present invention, all of which are the technical solutions to be protected by the present invention, and will not be enumerated here.

[0069] Advantages of the present invention:

[0070] First, by mining the spatial hierarchical attributes of objects in the three-dimensional point cloud scene and deeply integrating with the large-scale commonsense knowledge graph, the present invention constructs a symbolic knowledge graph depicting the spatial hierarchical attributes of the three-dimensional visual scene, accurately depicts the unique hierarchical spatial structure and scene layout of the three-dimensional point cloud scene, effectively overcomes the problem of the complex and changeable three-dimensional visual scene, provides a data basis for the multi-modal knowledge extraction process, enables the algorithm in the present invention to systematically perceive and understand any given three-dimensional point cloud visual scene, and has the advantage of being widely applicable to three-dimensional visual scenes of different categories and forms.

[0071] Second, the present invention uses a symbolic knowledge graph as guiding information to construct a hierarchical visual graph representation of the input three-dimensional visual scene, and guides the encoding of visual context features of the input visual scene, realizing the deep fusion of the spatial hierarchical attributes of the three-dimensional visual scene and the feature encoding network, extracting scene visual information enhanced by prior knowledge, overcoming the over-reliance on visual information in existing work, and using prior knowledge to enhance the representation ability of scene visual features, making the algorithm in the present invention have the advantage of knowledge enhancement ability for visual features.

[0072] Third, the present invention designs a graph reasoning network to fuse symbolic knowledge from the text modality and scene visual context features from the visual modality in the symbolic knowledge graph, realizing the extraction and learning of multi-modal spatial hierarchical knowledge, and using the object categories in the visual scene as indexes to extract corresponding multi-modal knowledge features, realizing multi-modal knowledge-enhanced object and relationship detection, effectively overcoming the problem of high feature requirements for three-dimensional scene graph prediction, and improving the algorithm performance and efficiency by introducing more expressive multi-modal knowledge, making the three-dimensional scene graph prediction method in the present invention have the advantages of high precision, high efficiency, strong robustness, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 is the overall framework diagram of the three-dimensional semantic scene graph prediction method based on multi-modal spatial hierarchical knowledge of the present invention;

[0074] Figure 2 is a schematic diagram of the graph reasoning network extracting multi-modal knowledge. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0075] The following describes the specific embodiments of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the features in the following embodiments can be combined with each other without conflict.

[0076] It should be noted that: similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0077] Refer to the attached Figure 1 drawings, and further describe the specific steps of the present invention.

[0078] S1. Construction of Symbolic Knowledge Based on Spatial Hierarchical Structure

[0079] Given a set of object categories O in a dataset, a multilayer perceptron is used to perform hierarchical classification on each object category. Each object category is divided into three levels: bottom, middle, or top. The bottom level contains only ground objects, which provide physical support for the middle level object categories. The middle level object categories, in turn, provide physical support for the top level object categories.

[0080] Input the hierarchical classification information of object categories and the common sense knowledge graph ConceptNet. First, the ConceptNet data is filtered and cleaned. Common object categories in 3D visual scenes are extracted from the SUNRGBD, Scannet and 3RScan datasets. Then, the ConceptNet is filtered to find the relationship triplets containing only these object categories. The filtered ConceptNet is defined as Node Set Contains only common object categories in 3D visual scenes, a set of directed edges Represents relationships between common object categories.

[0081] Given a hierarchical classification of object categories, the GloVe word embedding vectors of the corresponding object categories and the embedding vectors of the corresponding levels are concatenated to represent the knowledge graph. The embedding vector of each object node in the high-dimensional vector space.

[0082] If the knowledge graph The hierarchical classification results of any two object nodes in are cross-layer, and the knowledge graphs of the two nodes are If there is at least one path with 2 hops, a supporting edge is added between the two nodes to represent the physical support relationship between the two object categories. The corresponding word embedding vector in GloVe is used to represent The features of the directed edges in . Finally, the updated ConceptNet knowledge graph As a symbolic knowledge graph

[0083] The present invention mines the spatial hierarchical attributes of objects in three-dimensional point cloud scenes, deeply integrates them with large-scale common sense knowledge graphs, constructs a symbolic knowledge graph that characterizes the spatial hierarchical attributes of three-dimensional visual scenes, accurately depicts the hierarchical spatial structure and scene layout unique to three-dimensional point cloud scenes, effectively overcomes the complex and changeable problems of three-dimensional visual scenes, and provides a data foundation for the multimodal knowledge extraction process, so that the algorithm in the present invention can systematically perceive and understand any given three-dimensional point cloud visual scene, and has the advantage of being universally applicable to three-dimensional visual scenes of different categories and forms.

[0084] S2. Visual context encoding under symbolic knowledge guidance strategy

[0085] Input 3D point cloud scene where x i is the three-dimensional coordinates of the point, and f i is the additional point feature; under the guidance of the symbolic knowledge graph that depicts the spatial hierarchical structure knowledge of the three-dimensional point cloud scene , construct the hierarchical visual graph of the input three-dimensional visual scene. Define the hierarchical visual graph where the node set represents the object instances in the input three-dimensional point cloud scene, and the directed edge set represents the directed edges between object instances in the input scene.

[0086] Use VoteNet as the object detection network to generate the bounding box information of 256 candidate object instances and the candidate object set corresponding to the object categories o i is represented as {v i , c i}, where v i represents the bounding box parameters of object o i , and c i represents the object category to which object o i belongs.

[0087] Input the bounding box parameters v of each object i = c x , c y , c z , l x , l y , l z}, where (c x , c y , c z ) is the center coordinates of the bounding box, and (l x , l y , l z ) is the length of the bounding box on the three coordinate axes. Map the bounding box parameters to a high-dimensional vector space through a multi-layer perceptron to obtain the object spatial feature

[0088] According to the bounding box parameters v of the object i extract the three-dimensional point set corresponding to the object instance and use Point Cloud Transformer as the visual feature extraction network to extract the visual features of each object in the input scene

[0089] Given the object category c corresponding to each candidate object instance i , obtain the word embedding vector of the object category through GloVe as the semantic feature of the object ​The visual features f of cascaded objects v , the spatial features f t , and the semantic features f w to obtain the hierarchical visual graph the set of nodes in the initial features of each node.

[0090] If there is at least one support edge between the nodes of any two object nodes in the hierarchical visual graph in the nodes of the symbolic knowledge graph , then a directed edge is added between the two object nodes in the hierarchical visual graph and added to the set of directed edges representing the physical support relationship between the two object instances in the input 3D point cloud scene. By cascading the features of the two participating nodes in and calculating through a multi-layer perceptron to obtain the initial features of each edge in

[0091] Given the hierarchical visual graph Use a graph neural network with regional perception ability to iteratively update the hidden features of the nodes and directed edges in the hierarchical visual graph . The local region in is defined as several object nodes sharing the same support and their related directed edges. At time t, the node features are enhanced according to each node and the local region it is located in:

[0092]

[0093] where represents the hidden feature of object node i at time t, N r (i) represents other object nodes in the same local region as i, represents the enhanced node hidden feature, and ψ represents the forward propagation network.

[0094] Similarly, the feature enhancement of the directed edges in each local region is defined as follows:

[0095]

[0096] where represents the hidden feature of the directed edge between nodes i and j at time t, represents the enhanced directed edge hidden feature.

[0097] After feature enhancement, the regional perception graph neural network performs information transfer for each node and directed edge in the hierarchical visual graph :

[0098]

[0099]

[0100] where and respectively represent the information for updating the node i and the directed edge between nodes i and j aggregated at time t, which is calculated by the following formula:

[0101]

[0102] where N v (i) represents the set of nodes adjacent to node i in the hierarchical visual graph and and respectively represent two different non - linear vector transformations.

[0103] Similarly, it can be calculated by the following formula:

[0104]

[0105] where and respectively represent two three - layer multi - layer perceptron networks.

[0106] After three rounds of iteration, the hidden features of the object nodes in the graph neural network are taken as the visual context features of the object nodes and the hidden features of the directed edges are taken as the visual context features of the directed edges

[0107] The present invention uses a symbolic knowledge graph as guiding information, constructs a hierarchical visual graph representation of the input three - dimensional visual scene, and guides the encoding of the visual context features of the input visual scene, realizing the deep fusion of the spatial hierarchical attributes of the three - dimensional visual scene and the feature encoding network, extracting the scene visual information enhanced by prior knowledge, overcoming the over - dependence on visual information in existing works, and using prior knowledge to enhance the representation ability of the scene visual features, so that the algorithm in the present invention has the advantage of knowledge - enhanced ability for visual features.

[0108] S3. Multi - modal knowledge extraction based on text symbolic knowledge and visual context features;

[0109] Input symbolic knowledge graph and visual context features and Adopt a graph inference network to integrate text knowledge and visual knowledge, and iteratively update the knowledge features of each node and edge in the symbolic knowledge graph The features of the nodes or edges in the cascaded symbolic knowledge graph and the hierarchical visual graph The corresponding visual context features in the hierarchical visual graph where the node or edge appears and the 0 / 1 flag bits are used as the input features of each node and edge in the graph reasoning network.

[0110] As Figure 2 shown, the update process of the graph reasoning network is defined as follows:

[0111]

[0112]

[0113] where and respectively represent the multi-modal knowledge features of nodes and edges in the symbolic knowledge graph at time t, and respectively represent the information used to update node i and the directed edge between nodes i and j at time t, and its calculation process is defined as:

[0114]

[0115]

[0116] N k (i) represents the set of nodes adjacent to node i in the symbolic knowledge graph and and represent non-linear transformations.

[0117] After three rounds of iteration, the multi-modal knowledge features of each node and directed edge output by the graph reasoning network are taken as the multi-modal knowledge of the object category and the multi-modal knowledge of the directed edge

[0118] S4. Object and Relationship Detection Guided by Multi-modal Knowledge

[0119] Input the visual context features in the hierarchical visual graph and and the corresponding multi-modal knowledge and With the help of the visual context of the object node, initially predict the object category corresponding to the object in the input scene. And according to the initial prediction result, screen out the node multi-modal knowledge corresponding to the top three confidence object categories For the hierarchical visual graph For the directed edges on it, according to the object category prediction results, select the top three confidence object categories for the two object nodes connected by the directed edge respectively, and select the corresponding directed edge multimodal knowledge as the index.

[0120] Through two multi-layer perceptrons, respectively map the node visual context and the directed edge visual context to the relevant node multimodal knowledge and the directed edge multimodal knowledge after screening, and uniformly map them to the same vector space. For the nodes on the hierarchical visual graph The mapping process is as follows:

[0121]

[0122] In the formula and represent two multi-layer perceptrons respectively, φ represents the forward propagation network, represents the scene comprehensive feature after knowledge enhancement of node i.

[0123] Similarly, the mapping process of the directed edges on the hierarchical visual graph is defined as follows:

[0124]

[0125] represents the scene comprehensive feature after knowledge enhancement of the directed edge between nodes i and j.

[0126] Given the features after knowledge enhancement of the object node and the relationship edge and Connect each object node in the hierarchical visual graph pairwise to obtain a fully connected graph. The feature of the object node in the graph is initialized as the knowledge enhancement feature The feature of the directed edge is initialized as the knowledge enhancement feature If the relationship edge in the fully connected graph does not appear in the hierarchical visual graph it is initialized as the concatenation of the features of the participating object nodes. Use the standard graph convolutional neural network to perform information transfer on the object nodes and relationship edges in this fully connected graph to achieve knowledge reasoning for 3D visual scenes.

[0127] After three rounds of iteration, the output object node feature and the relationship edge feature are respectively fed into the object classification network and the relationship classification network:

[0128]

[0129]

[0130] where φ o and φ s respectively represent two forward propagation networks responsible for object and relationship classification, represents the probability distribution of the object category corresponding to the object node i, and |O| represents the number of object categories in the dataset. represents the probability distribution of the relationship category corresponding to the relationship edge between object nodes i and j, and |R| represents the number of relationship categories in the dataset.

[0131] The final loss function is defined as follows:

[0132]

[0133] where is the preliminary object category classification loss according to the object visual context and represents the final object and relationship classification loss. Among them can be expanded as follows:

[0134]

[0135] represents the cross-entropy loss of the final object category classification, represents the cross-entropy loss of the final relationship category classification, w o = 0.75 and w r = 1 are the weighting coefficients for balancing the loss terms.

[0136] By designing a graph reasoning network, the present invention fuses symbolic knowledge from the text modality and scene visual context features from the visual modality in the symbolic knowledge graph, realizes the extraction and learning of multi-modal spatial hierarchical knowledge, and uses the object categories in the visual scene as indexes to extract corresponding multi-modal knowledge features, realizes object and relationship detection with multi-modal knowledge enhancement, effectively overcomes the problem that three-dimensional scene graph prediction has high requirements for features, and improves the algorithm performance and efficiency by introducing more expressive multi-modal knowledge, making the three-dimensional scene graph prediction method in the present invention have the advantages of high precision, high efficiency, strong robustness, etc.

[0137] The above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A 3D semantic scene graph prediction method based on multimodal spatial hierarchical knowledge, characterized in that: The three-dimensional semantic scene graph prediction method comprises the following steps: S1: Construction of symbolic knowledge graph based on spatial hierarchy; Step S1 includes: S11: Perform hierarchical classification processing on each object category in the object category set O in the given data set through a multi-layer perceptron; S12: Based on the hierarchical classification information of object categories and the common sense knowledge graph ConceptNet, the common sense knowledge graph ConceptNet is screened and cleaned, and the filtered common sense knowledge graph ConceptNet is defined as Node Set Contains only common object categories in 3D visual scenes, with a directed edge set ε e Represent relationships between common object categories; S13: Based on the hierarchical classification of object categories, the GloVe word embedding vectors of the corresponding object categories and the embedding vectors of the corresponding levels are concatenated to represent the common sense knowledge graph. The embedding vector of each object node in the high-dimensional vector space; When the knowledge graph The hierarchical classification results of any two object nodes in are cross-layer, and the knowledge graphs of the two nodes are If there is at least one path with 2 hops, a supporting edge is added between the two nodes to represent the physical support relationship between the two object categories, and the corresponding word embedding vector in GloVe is used to represent Characteristics of directed edges in ; Finally, the updated common sense knowledge graph ConceptNet knowledge graph As a symbolic knowledge graph S2: Visual context encoding under the symbolic knowledge-guided strategy; Step S2 includes: S21: Based on 3D point cloud scenes, a symbolic knowledge graph depicting the spatial hierarchical structure knowledge of 3D point cloud scenes Under the guidance of S22: Use VoteNet as an object detection network to generate bounding box information for several candidate object instances and a set of candidate objects for the corresponding object categories. Input the bounding box parameters of each object and map the bounding box parameters to a high-dimensional vector space through a multi-layer perceptron to obtain the object space features. According to the object's bounding box parameter v i Extract the 3D point set corresponding to the object instance and use Point Cloud Transformer as the visual feature extraction network to extract the input scene The visual features of each object in Based on the object category corresponding to each candidate object instance, the word embedding vector of the object category is obtained through GloVe as the semantic feature of the object; Cascade the visual features, spatial features, and semantic features of objects to obtain a hierarchical visual map The initial features of each node in the node set; Moreover, when the hierarchical visual map Any two object nodes in the symbolic knowledge graph There is at least one supporting edge between the nodes in the hierarchical visual graph. Add a directed edge between the two object nodes in and add the directed edge set ε v , represents the physical support relationship between two object instances in the input 3D point cloud scene, through cascading The features of the two participating nodes are calculated by the multi-layer perceptron to obtain the characteristics of each edge in Initial features in S23: Given a hierarchical visual map Iteratively update the hierarchical visual graph using a graph neural network with region-aware capabilities Hidden features of middle nodes and directed edges, The local area in is defined as several object nodes sharing the same support and their related directed edges; At time t, the features of each node are enhanced according to the local area where each node is located; and the features of the directed edges in each local area are enhanced; After feature enhancement, the region-aware graph neural network targets the hierarchical visual graph Each node and directed edge in the graph is used to transfer information. After iteration, the hidden features of the object node in the graph neural network are taken as the visual context features of the object node. Hidden features of directed edges as visual context features of directed edges S3: Multimodal knowledge extraction based on text symbol knowledge and visual context features; Step S3 includes: Based on symbolic knowledge graph and visual context features and Use graph reasoning networks to integrate textual knowledge and visual knowledge, and iteratively update symbolic knowledge graphs The knowledge characteristics of each node and edge in After iteration, the multimodal knowledge features of each node and directed edge output by the graph reasoning network are taken as the multimodal knowledge of the object category. and multimodal knowledge with directed edges S4: Object and relation detection guided by multimodal knowledge; Step S4 includes: S41: Based on hierarchical visual graph Visual context features in and and corresponding multimodal knowledge and Visual context via object nodes Preliminary prediction of the object category corresponding to the object in the input scene; Based on the preliminary prediction results, the node multimodal knowledge corresponding to the top three confidence object categories is selected Hierarchical visual map For the directed edges on the , according to the object category prediction results, the top three confidence object categories are selected for the two object nodes connected by the directed edge, and the corresponding directed edge multimodal knowledge is selected as the index S42: Through two multi-layer perceptrons, the node visual context is and directed edge visual context Multimodal knowledge of nodes related to screening and directed edge multimodal knowledge Mapped uniformly to the same vector space, the scene comprehensive features after knowledge enhancement representing node i are obtained And the scene comprehensive features after knowledge enhancement representing the directed edge between nodes i and j S43: Features enhanced based on knowledge of object nodes and relationship edges and The hierarchical visual map Each object node in the graph is connected to each other to obtain a fully connected graph, and the object node features in the graph are initialized as knowledge enhancement features. Directed edge features are initialized as knowledge-enhanced features When the relationship edges in the fully connected graph do not appear in the hierarchical visual graph In , it is initialized to the feature splicing of the participating object nodes; The standard graph convolutional neural network is used to transmit information between object nodes and relationship edges in the fully connected graph to realize knowledge reasoning for three-dimensional visual scenes. After three rounds of iteration, the output object node features and relationship edge features Send them to the object classification network and the relationship classification network respectively to obtain the probability distribution of the object category corresponding to the object node i And the probability distribution of the relationship category corresponding to the relationship edge between object nodes i and j 2. The three-dimensional semantic scene graph prediction method according to claim 1, wherein: Step S11 also includes: classifying each object category into three levels: bottom level, middle level, or top level; The bottom layer only contains one type of object, the ground. The ground provides physical support for the middle-layer object categories, and the middle-layer object categories provide physical support for the object categories in the top layer.

3. The three-dimensional semantic scene graph prediction method according to claim 1, wherein: In step S11, data screening and cleaning of the common sense knowledge graph ConceptNet includes: Common object categories in 3D visual scenes are extracted from the SUNRGBD, Scannet, and 3RScan datasets, and relation triplets containing only these object categories are filtered out in the common sense knowledge graph ConceptNet.

4. The three-dimensional semantic scene graph prediction method according to claim 1, wherein: In step S23, at time t, feature enhancement is performed on each node based on the local area where each node is located: in Represents the hidden features of object node i at time t, N r (i) represents other object nodes in the same local area as i, represents the enhanced node hidden features, and ψ represents the forward propagation network; The feature enhancement of the directed edges in each local area is: in, represents the hidden features of the directed edge between nodes i and j at time t, Represents the enhanced directed edge hidden features.

5. The three-dimensional semantic scene graph prediction method according to claim 4, characterized in that: In step S23, after feature enhancement, the region perception graph neural network is used to target the hierarchical visual graph. Each node and directed edge in transmits information: in and They represent the information for updating the directed edges between node i and nodes i and j after aggregation at time t, Calculated by the following formula: where N v (i) Represents the hierarchical visual diagram The set of nodes adjacent to node i in , and Represent two different nonlinear vector transformations; A similar calculation can be obtained by the following formula: in and Represent two three-layer multilayer perceptron networks respectively.

6. The three-dimensional semantic scene graph prediction method according to claim 1, wherein: In step S3, the update process of the graph reasoning network is defined as follows: in and Represents the symbolic knowledge graph at time t Multimodal knowledge features of nodes and edges, and They represent the information used to update the directed edges between node i and nodes i and j at time t, and the calculation process is defined as: N k (i) represents the symbolic knowledge graph The set of nodes adjacent to node i in , and Represents a nonlinear transformation.

7. The three-dimensional semantic scene graph prediction method according to claim 1, wherein: In step S43, the output object node features and relationship edge features Send them to the object classification network and the relationship classification network respectively: Where φ o and φ s Representing two forward propagation networks responsible for object and relationship classification, Represents the probability distribution of the object category corresponding to the object node i, |O| represents the number of object categories in the data set, represents the probability distribution of the relationship category corresponding to the relationship edge between object nodes i and j, and |R| represents the number of relationship categories in the dataset.

Images