Unified perception and reasoning based end-to-end three-dimensional scene graph generation method and system

Abstract

The present application relates to the technical field of end-to-end model, and particularly relates to an end-to-end three-dimensional scene graph generation method and system based on unified perception and reasoning. The method comprises the following steps: extracting multi-scale dense scene features of an original point cloud, inputting the multi-scale dense scene features into a super point pooling module, selecting a super point through farthest point sampling and initializing a set of learnable query vectors; constructing a double-branch parallel geometric relationship decoder, performing bidirectional information interaction between the two branches through cross attention; inputting the query vectors into the geometric relationship decoder; sending the bounding boxes and categories of objects output by the entity branch to a detection head and outputting detection results; sending the data output by the triple branch to a geometric enhancement module, outputting enhanced relationship features and sending the enhanced relationship features to a relationship head to predict semantic relationship predicates, and obtaining a three-dimensional scene graph result. Through the core GeoRel decoder, unified modeling of perception and reasoning is realized, and perception and reasoning are mutually enhanced, thereby significantly improving the accuracy and robustness of three-dimensional scene graph generation.

Description

Technical Field

[0001] This invention relates to the field of end-to-end modeling technology, and in particular to an end-to-end 3D scene graph generation method and system based on unified perception and reasoning. Background Technology

[0002] 3D scene graph generation aims to resolve a scene into a structured graph representation, where nodes correspond to objects and edges correspond to semantic relationships between objects. Inspired by research on scene graph generation in the image domain, 3D scene graph generation has gradually developed in recent years, but its progress has been relatively slow due to the difficulty in acquiring 3D data and the high cost of annotation. Existing methods can be mainly summarized into the following two technical routes.

[0003] The first category of methods, pioneered by 3DSSG, constructed the first 3D scene graph dataset by annotating semantic triples on the 3RScan dataset and proposed the SGPN model, which uses 3D point clouds and provided instance annotations to predict semantic relationships. Subsequent research has continuously improved upon this: SGPN uses RGB-D sequences as incremental input and generates dynamic 3D scene graphs by handling incomplete objects and occlusions; to enhance graph reasoning capabilities, SGGpoint employs edge-oriented graph convolutional networks for multi-dimensional relationship modeling; Zhang et al. designed a graph autoencoder that extracts class prototype knowledge based solely on semantic category information to alleviate the interference of perceptual confusion on relationship prediction; VL-SAT introduces a vision-language assisted training mechanism to transfer knowledge from multimodal large models to 3D models, effectively handling long-tail relationships. However, these methods generally suffer from a key limitation: they typically assume that object candidate boxes or ground truth bounding boxes are pre-provided during inference, which does not align with real-world application scenarios (where only raw sensor data is used as input), thus limiting their practical applicability.

[0004] Another type of research employs a 2D-to-3D "projection" strategy. VGFM explored this direction early on. Wu et al. obtained instance results by inputting RGB images into a 2D perception model, and then reconstructed the 3D scene using SLAM technology, thereby generating a 3D scene map. FROSS combines a 2D detector with a scene map generation model, first generating a 2D scene map, and then using a Gaussian distribution to upscale the 2D detection results to 3D space, achieving fast inference from RGB-D sequences. Similarly, PSG-4D extends this idea to the temporal domain, capturing dynamic 4D maps from video sequences. USG further attempts to unify the representation of different modalities. Although these methods have speed advantages, their performance is highly dependent on the quality of 2D segmentation and multi-view consistency. Compared to methods that directly process 3D data, they often produce errors due to geometric inconsistencies and occlusion problems, making it difficult to guarantee the accuracy and robustness of the 3D scene map.

[0005] 3D object detection, as a crucial pre-processing technique for 3D scene graph generation (especially in the absence of ground truth annotations), has a noteworthy development history. From voting-based methods (such as VoteNet) and voxel-based methods (such as FCAF3D) to recent Transformer-based end-to-end detectors (such as 3DETR and UniDet3D), detection accuracy and efficiency have continuously improved. In particular, the introduction of the "ensemble prediction" paradigm allows detectors to avoid post-processing such as non-maximum suppression, laying the foundation for unified modeling with scene graph generation. However, these detectors only focus on object localization, treating instances as isolated entities and lacking the ability to reason about semantic and spatial relationships between objects.

[0006] Existing 3D scene graph generation methods generally adopt a two-stage pipeline architecture of "detection first, inference later." This means that an independent detection module first extracts object candidate boxes from the point cloud, and then a subsequent inference module predicts the semantic relationships between objects based on these detection results. However, this decoupled architecture has inherent drawbacks: any errors in the detection stage (missed detections, false detections, localization deviations) are directly propagated to the inference stage and continuously amplified, forming irreversible cascading errors. Simultaneously, information flow is unidirectional; the high-level semantic context of the inference stage cannot be backpropagated to optimize the detection results, preventing the detector and inferencer from co-optimizing. As a result, the overall system performance is limited by the upstream detection quality, making it difficult to achieve globally consistent scene understanding.

[0007] In summary, existing 3D scene graph generation methods either rely on pre-provided object candidates (type 1) or are limited by the quality of 2D perception (type 2), and the detection and inference tasks are disconnected, leading to error accumulation, lack of information feedback, and difficulty in achieving globally consistent scene understanding. Therefore, how to directly start from the original point cloud and unify the modeling of object detection and relation reasoning in an end-to-end manner has become a pressing technical problem to be solved in this field. Summary of the Invention

[0008] To address the technical problem in existing technologies that cannot directly start from raw point clouds and uniformly model object detection and relation reasoning in an end-to-end manner, this invention provides an end-to-end 3D scene graph generation method and system based on unified perception and reasoning. The technical solution is as follows: On the one hand, an end-to-end 3D scene graph generation method based on unified perception and reasoning is provided, characterized in that the method includes: S1. Obtain the original point cloud as input data, input the original point cloud into the sparse convolutional backbone network, and extract multi-scale dense scene features. S2. Input the dense scene features of multiple scales into the superpoint pooling module, select superpoints by sampling the farthest point and initialize a set of learnable query vectors; S3. Construct a dual-branch parallel geometric relation decoder. After updating their own features, the two branches interact bidirectionally through cross-attention. The dual branches include an entity branch and a triplet branch. Input the query vector into the geometric relation decoder. S4. Send the bounding boxes and categories of the objects output by the entity branch to the detection head and output the 3D detection results; send the data output by the triplet branch to the geometry enhancement module. The geometry enhancement module aggregates local context features based on the bounding boxes of the objects and dynamically adjusts the weights of geometric and semantic information. It outputs the enhanced relation features and sends them to the relation head to predict the semantic relation predicates, thus obtaining the 3D scene graph results.

[0009] Optionally, in S2, dense scene features at multiple scales are input into the superpoint pooling module, superpoints are selected by sampling from the farthest point, and a set of learnable query vectors is initialized, including: The farthest point sampling FPS is used to select K "superpoints" on the input original point cloud, and each superpoint corresponds to a local point cloud cluster; A set of learnable query vectors Q is initialized using the features of these superpoints, where each query vector is anchored to a local region in physical space; The query vector includes entity query vector and triple query vector.

[0010] Optionally, in S3, a dual-branch parallel geometric relation decoder is constructed. After updating their own features, the two branches interact bidirectionally through cross-attention. The dual branches include an entity branch and a triplet branch, including: Construct a dual-branch parallel geometric relation decoder; Among them, the two branches include: entity branches and triplet branches; The entity branch is responsible for locating and classifying object instances, and maintains a set of entity query vectors; The triplet branch is responsible for modeling and predicting the relationships between object pairs, and maintains a set of triplet query vectors; In each decoding layer, the two branches first perform self-attention to update their own features, and then interact bidirectionally through a cross-attention mechanism.

[0011] Optionally, in S4, the bounding box and category of the object output from the entity branch are sent to the detection head and the 3D detection results are output, including: The entity branch receives a set of entity query vectors; wherein, the set of entity query vectors includes: subject entity query vector and object entity query vector; The entity query vector is used to capture the global contextual relationships between objects through self-attention; Through cross-attention processing and scene features Fscene Interact with the object to supplement detailed information and output the object's bounding box parameters and semantic category.

[0012] Optionally, in S4, the data output from the triplet branch is sent to the geometry enhancement module. The geometry enhancement module aggregates local contextual features based on the object's bounding box and dynamically adjusts the weights of geometric and semantic information. It then outputs enhanced relation features, which are sent to the relation head to predict semantic relation predicates, resulting in a 3D scene graph, including: The triplet branch receives triplet query vectors; each triplet query vector is formed by concatenating the features of a pair of entity query vectors. After processing the triple query vector with self-attention and cross-attention, the final output is the semantic relation predicate between the subject and the object. The semantic relationship predicate between the subject and the object is input into the geometry enhancement module. The geometry enhancement module aggregates local context features based on the bounding box of the object and dynamically adjusts the weights of geometric and semantic information. Through gating mechanism and volume interaction calculation, the enhanced relationship features are generated. The enhanced relation features are input into the relation head to predict the semantic relation predicate, and the scene graph results are obtained.

[0013] Optionally, cross-attention processing includes: Each triple query vector obtains the latest relation-related instance features from the corresponding subject entity query vector and object entity query vector; Each entity query vector derives its high-level semantic context from its associated triple query vectors; All queries again focus on the global scene feature F through cross-attention. scene It supplements the raw data with details that were not captured by the query.

[0014] Optionally, the geometry enhancement module includes: Explicit geometric coding layer, spatially constrained context aggregation layer, and adaptive gated fusion layer; The center point coordinates of a pair of subject and object bounding boxes predicted by the current decoding layer are extracted through an explicit geometric coding layer, and the displacement vector is calculated. The displacement vector is then mapped to a high-dimensional space through a multilayer perceptron (MLP) to obtain a geometric embedding, and the relative positional topological relationship between the two is encoded. Interaction anchors are defined through a spatially constrained context aggregation layer, and local context features are aggregated on the scene feature F_scene using a distance-aware attention mechanism centered on these anchors. By using an adaptive gating fusion layer to dynamically adjust the weights of geometric and semantic information according to the specific context, complex spatial relationships can be modeled.

[0015] On the other hand, an end-to-end 3D scene graph generation system based on unified perception and reasoning is provided. This system is applied to the end-to-end 3D scene graph generation method based on unified perception and reasoning, and includes: The feature extraction unit is used to acquire the original point cloud as input data, input the original point cloud into the sparse convolutional backbone network, and extract dense scene features at multiple scales. The super-pooling unit is used to input dense scene features of multiple scales into the super-pooling module, select super points by sampling the farthest point, and initialize a set of learnable query vectors. The decoder construction unit is used to build a dual-branch parallel geometric relation decoder. The two branches update their own features and then interact bidirectionally through cross attention. The dual branches include an entity branch and a triplet branch. The query vector is input into the geometric relation decoder. The scene graph generation unit sends the bounding boxes and categories of objects output from the entity branch to the detection head and outputs the 3D detection results; it sends the data output from the triplet branch to the geometry enhancement module, outputs the enhanced relation features, and sends them to the relation head to predict the semantic relation predicates to obtain the 3D scene graph results.

[0016] On the other hand, an end-to-end 3D scene graph generation device based on unified perception and reasoning is provided. The end-to-end 3D scene graph generation device based on unified perception and reasoning includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, they implement any of the methods described above for the end-to-end 3D scene graph generation method based on unified perception and reasoning.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described end-to-end 3D scene graph generation methods based on unified perception and reasoning.

[0018] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: In this invention, a unified architecture of Uni3DSG is proposed, which reconstructs 3D scene graph generation into a direct end-to-end ensemble prediction task, synchronously inferring object instances and their semantic relationships from the raw point cloud. It abandons the two-stage pipeline, introducing physically guided query initialization and a cascaded bidirectional attention mechanism, enabling object localization and relationship reasoning to mutually promote and co-evolve in multi-level decoders. Simultaneously, a geometry enhancement module is designed to explicitly model the spatial topological relationships between objects, allowing relationship prediction to obtain fine-grained geometric context support. This unified modeling enables perception and reasoning to mutually enhance each other, thereby significantly improving the accuracy and robustness of scene graph generation. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of an end-to-end 3D scene graph generation method based on unified perception and reasoning provided in an embodiment of the present invention; Figure 2 This is an overall architecture diagram of the end-to-end 3D scene graph generation model provided in this embodiment of the invention; Figure 3 This is a block diagram of an end-to-end 3D scene graph generation system based on unified perception and reasoning provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0022] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0023] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0024] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0025] This invention provides an end-to-end 3D scene graph generation method based on unified perception and reasoning. This method can be implemented by an end-to-end 3D scene graph generation device based on unified perception and reasoning, which can be a terminal or a server. Figure 1 The flowchart shown is for an end-to-end 3D scene graph generation method based on unified perception and reasoning. The processing flow of this method may include the following steps: S1. Obtain the original point cloud as input data, input the original point cloud into the sparse convolutional backbone network, and extract multi-scale dense scene features.

[0026] In one feasible implementation, the input is an original point cloud X containing N points, each with 3D coordinates and C-dimensional auxiliary features. First, a sparse 3D U-Net is used as the backbone network to encode the input point cloud, extracting multi-scale dense scene features F. scene This feature contains rich geometric structure and semantic context information.

[0027] In one feasible implementation, the overall architecture of the end-to-end 3D scene graph generation model provided by the present invention is as follows: Figure 2 As shown, the input point cloud is first processed by a sparse convolutional backbone network (Sparse 3D U-Net) to extract dense scene features F. scene The scene features are then input into the superpoint pooling module, which selects superpoints through farthest point sampling and initializes a set of learnable query vectors. The core processing unit is the GeoRel Decoder, which employs a dual-branch parallel design: the left branch is the entity branch, maintaining the entity query vector Q. ent Q ent After being updated by the self-attention module, scene cross-attention and scene feature F are used. scene The interaction process ultimately inputs the detection head and outputs the object's bounding box and category. The right side shows the triplet branch, maintaining the triplet query vector Q. tri Q tri From the main query vector Q sub and object query vector Q obj It is composed of two branches, which also undergo self-attention and scene cross-attention processing. The two branches achieve bidirectional information interaction through cross-attention: the triple query vector focuses on the entity query vector through the subject cross-attention, and the entity query vector also focuses on the triple query vector through a corresponding mechanism. All queries are processed by the feedforward network before entering the next layer decoder. In the relation prediction process, a geometry enhancement module (GEM) is introduced: this module uses the subject query vector Q... sub and object query vector Q obj As input, enhanced relation features are generated through gating mechanisms and volumetric interactive computation, and finally, the relation head is input to predict the semantic relation predicate. The entire architecture uses cascaded processing of multi-layer decoders to enable continuous collaborative optimization between entity branches and triple branches, achieving end-to-end joint modeling of object detection and relation reasoning.

[0028] S2. Input the dense scene features of multiple scales into the superpoint pooling module, select superpoints by sampling the farthest point and initialize a set of learnable query vectors; In one feasible implementation, in S2, dense scene features of multiple scales are input into the superpoint pooling module, superpoints are selected by sampling from the farthest point, and a set of learnable query vectors is initialized, including: The farthest point sampling FPS is used to select K "superpoints" on the input original point cloud, and each superpoint corresponds to a local point cloud cluster; A set of learnable query vectors Q is initialized using the features of these superpoints, where each query vector is anchored to a local region in physical space; The query vector includes entity query vector and triple query vector.

[0029] In one feasible implementation, physical-guided query initialization continues: Unlike the random initialization used in existing Transformer methods, this invention employs farthest point sampling (FPS) to select K "superpoints" on the input point cloud, each superpoint corresponding to a local point cloud cluster. A set of learnable query vectors Q, with dimensions K×D, is initialized using the features of these superpoints. Each query vector is anchored to a local region in physical space, aiming to represent a potential target instance. This initialization method provides good geometric priors and spatial anchors for the subsequent decoding process, helping to accelerate convergence and improve localization accuracy.

[0030] S3. Construct a dual-branch parallel geometric relation decoder (GeoRel decoder). After updating their own features, the two branches interact bidirectionally through cross-attention. The dual branches include an entity branch and a triplet branch. Input the query vector into the geometric relation decoder. In one feasible implementation, in S3, a dual-branch parallel geometric relation decoder is constructed. After updating their own features, the two branches interact bidirectionally through cross-attention. The dual branches include an entity branch and a triplet branch, comprising: Construct a dual-branch parallel geometric relation decoder; Among them, the two branches include: entity branches and triplet branches; The entity branch is responsible for locating and classifying object instances, and maintains a set of entity query vectors; The triplet branch is responsible for modeling and predicting the relationships between object pairs, and maintains a set of triplet query vectors; In each decoding layer, the two branches first perform self-attention to update their own features, and then interact bidirectionally through a cross-attention mechanism.

[0031] S4. Send the bounding boxes and categories of the objects output by the entity branch to the detection head and output the 3D detection results; send the data output by the triplet branch to the geometry enhancement module. The geometry enhancement module aggregates local context features based on the bounding boxes of the objects and dynamically adjusts the weights of geometric and semantic information. It outputs the enhanced relation features and sends them to the relation head to predict the semantic relation predicates, thus obtaining the 3D scene graph results.

[0032] In one feasible implementation, in step S4, the bounding box and category of the object output from the entity branch are sent to the detection head and the 3D detection result is output, including: The entity branch receives a set of entity query vectors; wherein, the set of entity query vectors includes: subject entity query vector and object entity query vector; The entity query vector is used to capture the global contextual relationships between objects through self-attention; Through cross-attention processing and scene features F scene Interact with the object to supplement detailed information and output the object's bounding box parameters and semantic category.

[0033] In one feasible implementation, in S4, the data output from the triplet branch is sent to the geometry enhancement module. The geometry enhancement module aggregates local contextual features based on the object's bounding box and dynamically adjusts the weights of geometric and semantic information. It then outputs enhanced relation features and sends them to the relation head to predict semantic relation predicates, obtaining a 3D scene graph result, including: The triplet branch receives triplet query vectors; each triplet query vector is formed by concatenating the features of a pair of entity query vectors. After processing the triple query vector with self-attention and cross-attention, the final output is the semantic relation predicate between the subject and the object. The semantic relationship predicate between the subject and the object is input into the geometry enhancement module, and the enhanced relationship features are generated through gating mechanism and volume interaction calculation. The enhanced relation features are input into the relation head to predict the semantic relation predicate, and the 3D scene graph result is obtained.

[0034] In one feasible implementation, cross-attention processing includes: Each triple query vector obtains the latest relation-related instance features from the corresponding subject entity query vector and object entity query vector; Each entity query vector derives its high-level semantic context from its associated triple query vectors; All queries again focus on the global scene feature F through cross-attention. scene It supplements the raw data with details that were not captured by the query.

[0035] One feasible implementation method is, for example Figure 2 As shown, this is the core module of the present invention, which is a multi-layer Transformer decoder consisting of two parallel branches, and achieves information interaction and collaborative evolution through a cascaded bidirectional attention mechanism.

[0036] Entity branch: Responsible for locating and classifying object instances. This branch maintains a set of entity query vectors Q. ent Each query corresponds to a potential object. Global contextual relationships between objects are captured through self-attention, and then cross-attention is used in conjunction with scene features F. scene Interact with the object to supplement detailed information, and finally output the bounding box parameters and semantic category of the object.

[0037] The triplet branch is responsible for modeling and predicting the relationships between object pairs. This branch maintains a set of triplet query vectors Q. tri Each triplet query vector is formed by concatenating the features of a pair of entity query vectors (subject and object). After self-attention and cross-attention, the final output is the semantic relation predicate between the subject and the object.

[0038] Cascaded bidirectional attention: In each decoding layer, the two branches first perform self-attention to update their own features. Then, a special cross-attention mechanism is designed to achieve bidirectional interaction. Triple query vectors focus on entity query vectors: enabling each triple query vector to obtain the latest, relation-related instance features from the corresponding subject and object entity query vectors, ensuring that relation predictions are based on accurate object states.

[0039] Entity query vectors focus on triple query vectors: enabling each entity query vector to obtain high-level semantic context from its associated triple query vectors, i.e., which relationships the object participates in. This contextual information can in turn guide the object's localization (e.g., for an object participating in a "suspended" relationship, its localization must satisfy physical support constraints).

[0040] Scene Feature Cross-Attention: Finally, all queries (including entities and triples) again focus on the global scene feature F through cross-attention. scene This mechanism supplements the raw data with detailed information that was not captured by the query. This multi-interaction mechanism of "entity-triple-scene" enables object localization and relation reasoning to promote and optimize each other in each layer of the decoder.

[0041] In one feasible implementation, the geometry enhancement module includes: Explicit geometric coding layer, spatially constrained context aggregation layer, and adaptive gated fusion layer; The center point coordinates of a pair of subject and object bounding boxes predicted by the current decoding layer are extracted through an explicit geometric coding layer, and the displacement vector is calculated. The displacement vector is then mapped to a high-dimensional space through a multilayer perceptron (MLP) to obtain a geometric embedding, and the relative positional topological relationship between the two is encoded. Interaction anchors are defined through a spatially constrained context aggregation layer, and interaction features F are defined around these anchors. scene The above employs a distance-aware attention mechanism to aggregate local contextual features; By using an adaptive gating fusion layer to dynamically adjust the weights of geometric and semantic information according to the specific context, complex spatial relationships can be modeled.

[0042] In one feasible implementation, to further enhance the ability to model spatial relationships, especially to identify relationships with clear geometric constraints such as "standing on," "attached to," and "suspended from," this invention introduces a geometry enhancement module into the relationship prediction head. This module explicitly utilizes the geometric information of the predicted bounding box to enhance relationship features.

[0043] Explicit geometric coding: For a pair of subject and object bounding boxes predicted by the current decoding layer, extract the coordinates c of their center points. sub and c obj Calculate the displacement vector Δ pos = c sub - c obj The displacement vector is mapped to a high-dimensional space using a multilayer perceptron (MLP) to obtain the geometric embedding E. geo This is used to encode the relative positional topological relationship between the two.

[0044] Spatial Constraint Context Aggregation: This invention proposes a key assumption—the physical interaction between objects is a local phenomenon, and the "blank space" between them contains important semantic clues. Therefore, an interaction anchor point c is defined. mid = (c sub +c obj j) / 2 (i.e., the midpoint of the line connecting the two center points), and using this anchor point as the center, in the scene feature F scene The above employs a distance-aware attention mechanism to aggregate local contextual features F. ctx The aggregation process introduces hard space constraints, meaning it only considers points within radius τ and bases them on distance d. k (c) mid With scene point c scene k The exponential decay of the Euclidean distance (the distance between the two points) assigns different weights, thereby ensuring that the aggregated features are strictly anchored to the interaction region in the physical space.

[0045] Adaptive Gated Fusion: To flexibly control the fusion ratio of visual semantic features and geometric context features, a learnable gating scalar g is used, which is determined by the subject query vector Q. sub Object query vector Q obj and geometric embedding E geo Calculated using MLP and the Sigmoid function: The final relation is represented by F. rel From the main query vector Q sub Object query vector Q obj Geometric Embedding E geo and the weighted contextual features g·F ctx Composed of multiple parts: F rel = Concat(Q sub Q ob E geo , g·F ctx This design enables the model to dynamically adjust the weights of geometric and semantic information according to the specific context, thereby more accurately modeling complex spatial relationships.

[0046] In one possible implementation, the present invention also relates to the following matching strategy: To address the issue of inconsistency between prediction results and the number of actual annotations in end-to-end ensemble prediction tasks, this invention employs a hybrid bipartite graph matching strategy to perform optimal matching for both the detection task and the scene graph generation task, and calculates a joint loss function (such as the classification loss and regression loss in the loss function described below).

[0047] Among them, the detection matching cost is calculated using the Hungarian algorithm to determine the optimal match between the predicted object and the real object. The matching cost C is... det Taking into account both the classification probability and the GIoU (Generalized Intersection over Union) similarity of the bounding boxes, specifically the classification item (the negative class prediction probability multiplied by the weight β) is considered. cls ) and bounding box regression term (GIoU loss multiplied by weight β) box The weighted sum is obtained by taking the sum of the weights. Where β cls Set to 0.5, β box Set it to 2.0.

[0048] Scene graph matching cost: Extending matching to the relation triple level, while considering the alignment of subjects, objects, and their relationships. Matching cost C rel It consists of a weighted sum of the subject classification term, the subject bounding box regression term, the object classification term, the object bounding box regression term, and the relationship classification term. The weights of each term are consistent with the detection task, and the weight β of the relationship classification term is... rel Set it to 0.5.

[0049] The matching strategy also includes geometric consistency constraints: to further address spatial ambiguity, this invention introduces a geometric consistency constraint based on the query's "origin"—only real instances spatially containing the query's origin superpoint are considered valid matches. By masking invalid pairings in the cost matrix, it ensures that the generated scene graph is strictly anchored to the physical geometry.

[0050] In one feasible implementation, the present invention designs a loss function for the above-described detection process, including: a. Classification loss: L cls = CE(p, y) Where CE is the cross-entropy loss function, p is the classification probability output by the network, and y is the class label.

[0051] b. Regression Loss

[0052] IoU = area(b p ∩ b gt ) / area(b p ∪ b gt )

[0053] L reg = 1 – IoU

[0054] Where IoU represents the predicted bounding box b p and annotation box b g The intersection-union ratio (IoU) between the two is used as the loss function because a larger ratio indicates greater similarity. Therefore, the upper bound of 1 minus IoU is used.

[0055] In this embodiment of the invention, a unified architecture is used to achieve end-to-end 3D object detection and 3D scene graph generation. The introduced geometry enhancement module (GEM) explicitly encodes the geometric topological relationships (displacement vectors) and local spatial context (midpoint neighborhood features) between objects, and dynamically fuses multi-source information using an adaptive gating mechanism. This effectively improves the ability to recognize spatial relationships with clear geometric constraints, such as "standing on," "attached to," and "suspended from," while reducing system complexity and engineering redundancy. This invention is a pure end-to-end model that directly processes raw point clouds without relying on any intermediate annotations (such as object proposals or ground truth boxes) or 2D projection results during the inference stage. This allows the invention to be directly applied to various real-world scenarios, such as robot navigation, autonomous driving, and AR / VR perception, and has broad application prospects and practical value.

[0056] Figure 2This is a block diagram of an end-to-end 3D scene graph generation system 300 based on unified perception and reasoning, according to an exemplary embodiment. The system 300 is used in an end-to-end 3D scene graph generation method based on unified perception and reasoning. (Refer to...) Figure 2 The system includes a feature extraction unit 310, a superpooling unit 320, a decoder construction unit 330, and a scene graph generation unit 340. Among them: The feature extraction unit 310 is used to acquire the original point cloud as input data, input the original point cloud into the sparse convolutional backbone network, and extract multi-scale dense scene features. The super-pooling unit 320 is used to input dense scene features of multiple scales into the super-pooling module, select super points by sampling the farthest point, and initialize a set of learnable query vectors. Decoder construction unit 330 is used to construct a dual-branch parallel geometric relation decoder. After updating their own features, the two branches interact bidirectionally through cross attention. The dual branches include an entity branch and a triplet branch. The query vector is input into the geometric relation decoder. The scene graph generation unit 340 is used to send the bounding boxes and categories of objects output from the entity branch to the detection head and output the detection results; it sends the data output from the triplet branch to the geometry enhancement module, which aggregates local context features based on the bounding boxes of objects and dynamically adjusts the weights of geometric and semantic information, outputs enhanced relation features and sends them to the relation head to predict semantic relation predicates, and obtains scene graph results.

[0057] Optionally, the superpoint pooling unit 320 is used to select K "superpoints" on the input original point cloud using the farthest point sampling FPS, with each superpoint corresponding to a local point cloud cluster; A set of learnable query vectors Q is initialized using the features of these superpoints, where each query vector is anchored to a local region in physical space; The query vector includes entity query vector and triple query vector.

[0058] Optionally, the decoder building unit 330 is used to build a dual-branch parallel geometric relation decoder; Among them, the two branches include: entity branches and triplet branches; The entity branch is responsible for locating and classifying object instances, and maintains a set of entity query vectors; The triplet branch is responsible for modeling and predicting the relationships between object pairs, and maintains a set of triplet query vectors; In each decoding layer, the two branches first perform self-attention to update their own features, and then interact bidirectionally through a cross-attention mechanism.

[0059] Optionally, the scene graph generation unit 340 is used to receive a set of entity query vectors in the entity branch; wherein, the set of entity query vectors includes: a subject entity query vector and an object entity query vector; The entity query vector is used to capture the global contextual relationships between objects through self-attention; By interacting with scene features F_scene through cross-attention processing to supplement detailed information, the bounding box parameters and semantic categories of objects are output.

[0060] Optionally, the scene graph generation unit 340 is used to receive triple query vectors in the triple branch; wherein each triple query vector is formed by concatenating the features of a pair of entity query vectors. After processing the triple query vector with self-attention and cross-attention, the final output is the semantic relation predicate between the subject and the object. The semantic relationship predicate between the subject and the object is input into the geometry enhancement module, and the enhanced relationship features are generated through gating mechanism and volume interaction calculation. The enhanced relation features are input into the relation head to predict the semantic relation predicate, and the scene graph results are obtained.

[0061] Optionally, cross-attention processing includes: Each triple query vector obtains the latest relation-related instance features from the corresponding subject entity query vector and object entity query vector; Each entity query vector derives its high-level semantic context from its associated triple query vectors; All queries again focus on the global scene feature F through cross-attention. scene It supplements the raw data with details that were not captured by the query.

[0062] Optionally, the geometry enhancement module includes: Explicit geometric coding layer, spatially constrained context aggregation layer, and adaptive gated fusion layer; The center point coordinates of a pair of subject and object bounding boxes predicted by the current decoding layer are extracted through an explicit geometric coding layer, and the displacement vector is calculated. The displacement vector is then mapped to a high-dimensional space through a multilayer perceptron (MLP) to obtain a geometric embedding, and the relative positional topological relationship between the two is encoded. Interaction anchors are defined through a spatially constrained context aggregation layer, and interaction features F are defined around these anchors. scene The above employs a distance-aware attention mechanism to aggregate local contextual features; By using an adaptive gating fusion layer to dynamically adjust the weights of geometric and semantic information according to the specific context, complex spatial relationships can be modeled.

[0063] In this embodiment of the invention, a unified architecture is used to achieve end-to-end 3D object detection and 3D scene graph generation. The introduced geometry enhancement module (GEM) explicitly encodes the geometric topological relationships (displacement vectors) and local spatial context (midpoint neighborhood features) between objects, and dynamically fuses multi-source information using an adaptive gating mechanism. This effectively improves the ability to recognize spatial relationships with clear geometric constraints, such as "standing on," "attached to," and "suspended from," while reducing system complexity and engineering redundancy. This invention is a pure end-to-end model that directly processes raw point clouds without relying on any intermediate annotations (such as object proposals or ground truth boxes) or 2D projection results during the inference stage. This allows the invention to be directly applied to various real-world scenarios, such as robot navigation, autonomous driving, and AR / VR perception, and has broad application prospects and practical value.

[0064] Figure 4 This is a schematic diagram of the structure of an end-to-end 3D scene graph generation device based on unified perception and reasoning provided in an embodiment of the present invention, as shown below. Figure 4 As shown, an end-to-end 3D scene graph generation device based on unified perception and reasoning may include the above-mentioned... Figure 3 The diagram illustrates an end-to-end 3D scene graph generation system based on unified perception and reasoning. Optionally, an end-to-end 3D scene graph generation device 410 based on unified perception and reasoning may include a first processor 2001.

[0065] Optionally, an end-to-end 3D scene graph generation device 410 based on unified perception and reasoning may further include a memory 2002 and a transceiver 2003.

[0066] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0067] The following is combined with Figure 4 The components of an end-to-end 3D scene graph generation device 410 based on unified perception and reasoning are described in detail below: The first processor 2001 is a control center of an end-to-end 3D scene graph generation device 410 based on unified perception and reasoning. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0068] Optionally, the first processor 2001 can execute various functions of an end-to-end 3D scene graph generation device 410 based on unified perception and reasoning by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0069] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0070] In a specific implementation, as one example, an end-to-end 3D scene graph generation device 410 based on unified perception and reasoning may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0071] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0072] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via an interface circuit of an end-to-end three-dimensional scene graph generation device 410 based on unified perception and reasoning. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0073] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0074] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0075] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to an interface circuit of an end-to-end 3D scene graph generation device 410 based on unified perception and reasoning. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0076] It should be noted that, Figure 4 The structure of an end-to-end 3D scene graph generation device 410 based on unified perception and reasoning shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0077] Furthermore, the technical effects of an end-to-end 3D scene graph generation device 410 based on unified perception and reasoning can be referred to the technical effects of an end-to-end 3D scene graph generation method based on unified perception and reasoning described in the above method embodiments, and will not be repeated here.

[0078] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0079] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0080] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable sensors. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0081] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0082] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0083] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0086] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0087] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An end-to-end 3D scene graph generation method based on unified perception and reasoning, characterized in that, The term includes: S1. Obtain the original point cloud as input data, input the original point cloud into the sparse convolutional backbone network, and extract multi-scale dense scene features. S2. Input the dense scene features of multiple scales into the superpoint pooling module, select superpoints by sampling the farthest point and initialize a set of learnable query vectors; S3. Construct a dual-branch parallel geometric relation decoder. After updating their own features, the two branches interact bidirectionally through cross-attention. The dual branches include an entity branch and a triplet branch. Input the query vector into the geometric relation decoder. S4. Send the bounding boxes and categories of the objects output by the entity branch to the detection head and output the 3D detection results; send the data output by the triplet branch to the geometry enhancement module. The geometry enhancement module aggregates local context features based on the bounding boxes of the objects and dynamically adjusts the weights of geometric and semantic information. It outputs the enhanced relation features and sends them to the relation head to predict the semantic relation predicates, thus obtaining the 3D scene graph results.

2. The end-to-end 3D scene graph generation method based on unified perception and reasoning according to claim 1, characterized in that, In step S2, dense scene features at multiple scales are input into the superpoint pooling module. Superpoints are selected through farthest point sampling, and a set of learnable query vectors is initialized, including: The farthest point sampling FPS is used to select K "superpoints" on the input original point cloud, and each superpoint corresponds to a local point cloud cluster; A set of learnable query vectors Q is initialized using the features of these superpoints, where each query vector is anchored to a local region in physical space; The query vector includes entity query vector and triple query vector.

3. The end-to-end 3D scene graph generation method based on unified perception and reasoning according to claim 2, characterized in that, In S3, a dual-branch parallel geometric relation decoder is constructed. After updating their own features, the two branches exchange information bidirectionally through cross-attention. The dual branches include an entity branch and a triplet branch, including: Construct a dual-branch parallel geometric relation decoder; Among them, the two branches include: entity branches and triplet branches; The entity branch is responsible for locating and classifying object instances, and maintains a set of entity query vectors; The triplet branch is responsible for modeling and predicting the relationships between object pairs, and maintains a set of triplet query vectors; In each decoding layer, the two branches first perform self-attention to update their own features, and then interact bidirectionally through a cross-attention mechanism.

4. The end-to-end 3D scene graph generation method based on unified perception and reasoning according to claim 3, characterized in that, In S4, the bounding boxes and categories of the objects output from the entity branch are sent to the detection head, and the 3D detection results are output, including: The entity branch receives a set of entity query vectors; wherein, the set of entity query vectors includes: subject entity query vector and object entity query vector; The entity query vector is used to capture the global contextual relationships between objects through self-attention; Through cross-attention processing and scene features F scene Interact with the object to supplement detailed information and output the object's bounding box parameters and semantic category.

5. The end-to-end 3D scene graph generation method based on unified perception and reasoning according to claim 4, characterized in that, In S4, the data output from the triplet branch is sent to the geometry enhancement module. This module aggregates local contextual features based on the object's bounding box and dynamically adjusts the weights of geometric and semantic information. It then outputs enhanced relation features, which are sent to the relation head to predict semantic relation predicates, resulting in a 3D scene graph, including: The triplet branch receives triplet query vectors; each triplet query vector is formed by concatenating the features of a pair of entity query vectors. After processing the triple query vector with self-attention and cross-attention, the final output is the semantic relation predicate between the subject and the object. The semantic relationship predicate between the subject and the object is input into the geometry enhancement module. The geometry enhancement module aggregates local context features based on the bounding box of the object and dynamically adjusts the weights of geometric and semantic information. Through gating mechanism and volume interaction calculation, the enhanced relationship features are generated. The enhanced relation features are input into the relation head to predict the semantic relation predicate, and the scene graph results are obtained.

6. The end-to-end 3D scene graph generation method based on unified perception and reasoning according to claim 5, characterized in that, The cross-attention processing includes: Each triple query vector obtains the latest relation-related instance features from the corresponding subject entity query vector and object entity query vector; Each entity query vector derives its high-level semantic context from its associated triple query vectors; All queries again focus on the global scene feature F through cross-attention. scene It supplements the raw data with details that were not captured by the query.

7. The end-to-end 3D scene graph generation method based on unified perception and reasoning according to claim 6, characterized in that, The geometry enhancement module includes: Explicit geometric coding layer, spatially constrained context aggregation layer, and adaptive gated fusion layer; The center point coordinates of a pair of subject and object bounding boxes predicted by the current decoding layer are extracted through an explicit geometric coding layer, and the displacement vector is calculated. The displacement vector is then mapped to a high-dimensional space through a multilayer perceptron (MLP) to obtain a geometric embedding, and the relative positional topological relationship between the two is encoded. Interaction anchors are defined through a spatially constrained context aggregation layer, and interaction features F are defined around these anchors. scene The above employs a distance-aware attention mechanism to aggregate local contextual features; By using an adaptive gating fusion layer to dynamically adjust the weights of geometric and semantic information according to the specific context, complex spatial relationships can be modeled.

8. An end-to-end 3D scene graph generation system based on unified perception and reasoning, wherein the end-to-end 3D scene graph generation system based on unified perception and reasoning is used to implement the end-to-end 3D scene graph generation method based on unified perception and reasoning as described in any one of claims 1-7, characterized in that, The system includes: The feature extraction unit is used to acquire the original point cloud as input data, input the original point cloud into the sparse convolutional backbone network, and extract dense scene features at multiple scales. The super-pooling unit is used to input dense scene features of multiple scales into the super-pooling module, select super points by sampling the farthest point, and initialize a set of learnable query vectors. The decoder construction unit is used to build a dual-branch parallel geometric relation decoder, with bidirectional information exchange between the two branches through cross attention; the dual branches include an entity branch and a triplet branch; the query vector is input into the geometric relation decoder; The scene graph generation unit sends the bounding boxes and categories of objects output from the entity branch to the detection head and outputs 3D detection results; it sends the data output from the triplet branch to the geometry enhancement module, which aggregates local context features based on the bounding boxes of objects and dynamically adjusts the weights of geometric and semantic information, outputs enhanced relation features, and sends them to the relation head to predict semantic relation predicates, thus obtaining 3D scene graph results.

9. An end-to-end 3D scene graph generation device based on unified perception and reasoning, characterized in that, The end-to-end 3D scene graph generation device based on unified perception and reasoning includes: A processor; a memory storing computer-readable instructions that, when executed by the processor, implement any one of the methods in the end-to-end 3D scene graph generation method based on unified perception and reasoning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement any one of the methods in the end-to-end 3D scene graph generation method based on unified perception and reasoning as described in any one of claims 1-7.

Images