Computationally effective compressed representation for three-dimensional scene understanding

Abstract

A system includes an object database configured to store three-dimensional (3D) objects, a graph database configured to store a time-series of graphs; and a processor unit. The processor unit is configured to: a) input sensor data from at least one sensor; b) generate a point cloud with class labels from the sensor data by semantic segmentation; c) extract current 3D objects from the point cloud with class labels; d) create a current graph based on nodes and their relationships in the point cloud to store the current graph on the graph database; and e) merge the current 3D objects with 3D objects existing in the object database to store merged 3D objects on the object database, wherein each node included in a graph of the time-series of graphs corresponds to one of the merged 3D objects.

Description

COMPUTATIONALLY EFFECTIVE COMPRESSED REPRESENTATION FOR THREE-DIMENSIONAL SCENE UNDERSTANDING

[0001] The present invention relates to techniques for three-dimensional (3D) scene understanding.Background

[0002] Scene understanding by 3D semantic segmentation is used in various applications such as autonomous driving, robotics, and inspection automation. 3D semantic segmentation typically includes labeling each point of 3D point cloud with a classification attribute (e.g., chair, table, road, etc.). It is believed that a deeper comprehension of a scene can be attained by using graphs that highlight the relationships between objects.

[0003] An example of graph-based 3D scene understanding is disclosed in non patent literature (NPL) 1. According to NPL 1, a sequence of RGB-D images is used to create a point cloud with segments as nodes, from which the properties of each segment and a neighbor graph between segments are constructed. The properties and neighbor graph of segments are used to compute node and edge features and to predict a 3D scene graph.

[0004] On the other hand, there is an increasing research interest to compress a tremendous amount of 3D point cloud data. For example, NPL 2 discloses a real-time compression of point cloud streams, more specifically a time-series point cloud compression technique utilizing voxel-level temporal redundancy.

[0005] Shun-Cheng Wu et al. “SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences”, arXiv:2103.14898v3 [cs.CV] 31 Mar 2021, <https: / / arxiv.org / abs / 2103.14898>J. Kammerl et al. “Real-time compression of point cloud streams” published in IEEE ICRA 2012Summary

[0006] NPL 1 discloses 3D scene graph creation from a single sequence of RGB-D images but is silent on handling multiple sequences with different time stamps. In the case of multiple sequences, the 3D scene graph may be created for each time stamp and stored on a storage. The 3D scene graph is a combination of graph and 3D point cloud data with class labels. In general, a tremendous amount of 3D point cloud data is stored on the storage. Accordingly, storing a 3D scene graph each time 3D data is captured causes heavy load on the storage.

[0007] Further it is difficult to effectively detect update from multiple 3D scene graphs of different time stamps. For example, when an object position and / or composition change at an instant of time, fast detection of such an update cannot be achieved partly because of data-heavy 3D scene graph. NPL 1 does not propose any means of effectively detecting a 3D scene change.

[0008] As described above, NPL 1 has the following shortcomings: 1) the increased load on a storage due to data-heavy 3D scene graph; and 2) update detection from multiple 3D scene graphs is inefficient and slow.

[0009] It may be possible to reduce the amount of point cloud data by employing point cloud compression as described in NPL 2. However, NPL 2 employs an octree-based compression technique. Accordingly, the increased size of octree voxel improves compression rate but causes undesirable data loss. NPL 2 does not propose any means of resolving such a trade-off between compression rate and data loss.

[0010] An object of the present invention is to provide a technique of effective storage and lossless compression for multiple sequences of 3D data.

[0011] The present inventors found out that 3D object point clouds at different instants of time included redundant 3D object point clouds. Accordingly, by merging 3D object point clouds through different instants of time, the redundancy can be eliminated, resulting in the dramatically reduced amount of 3D object point cloud data.

[0012] According to an aspect of the present disclosure, a system includes: an object database configured to store three-dimensional (3D) objects; a graph database configured to store a time-series of graphs; and at least one processor configured to: a) input sensor data from at least one sensor; b) generate a point cloud with class labels from the sensor data by semantic segmentation; c) extract current 3D objects from the point cloud with class labels; d) create a current graph based on nodes and their relationships in the point cloud to store the current graph on the graph database; and e) merge the current 3D objects with 3D objects existing in the object database to store merged 3D objects on the object database, wherein each node included in a graph of the time-series of graphs corresponds to one of the merged 3D objects.

[0013] According to another aspect of the present disclosure, a method for processing three-dimensional (3D) scene in a system including an object database configured to store 3D objects, a graph database configured to store a time-series of graphs, and at least one processor, the method include: by the at least one processor, a) inputting sensor data from at least one sensor; b) generating a point cloud with class labels from the sensor data by semantic segmentation; c) extracting current 3D objects from the point cloud with class labels; d) creating a current graph based on nodes and their relationships in the point cloud to store the current graph on the graph database; and e) merging the current 3D objects with the 3D objects existing in the object database to store merged 3D objects on the object database, wherein each node included in a graph of the time-series of graphs corresponds to one of the merged 3D objects.

[0014] According to a further aspect of the present disclosure, a computer-readable program for processing three-dimensional (3D) scene in a system including an object database configured to store 3D objects, and a graph database configured to store a time-series of graphs, the program includes instructions to: a) input sensor data from at least one sensor; b) generate a point cloud with class labels from the sensor data by semantic segmentation; c) extract current 3D objects from the point cloud with class labels; d) create a current graph based on nodes and their relationships in the point cloud to store the current graph on the graph database; and e) merge the current 3D objects with 3D objects existing in the object database to store merged 3D objects on the object database, wherein each node included in a graph of the time-series of graphs corresponds to one of the merged 3D objects.

[0015] According to a still another aspect of the present disclosure, a non-transitory recording medium storing a computer-readable program for processing three-dimensional (3D) scene in a system including an object database configured to store 3D objects, a graph database configured to store a time-series of graphs, and at least one processor, the program includes instructions to: a) input sensor data from at least one sensor; b) generate a point cloud with class labels from the sensor data by semantic segmentation; c) extract current 3D objects from the point cloud with class labels; d) create a current graph based on nodes and their relationships in the point cloud to store the current graph on the graph database; and e) merge the current 3D objects with 3D objects existing in the object database to store merged 3D objects on the object database, wherein each node included in a graph of the time-series of graphs corresponds to one of the merged 3D objects.

[0016] As described above, according to the present invention, effective storage and lossless compression for multiple sequences of 3D data can be achieved.

[0017] The invention accordingly comprises the several steps and the relation of one or more of such steps with respect to each of the others, and the apparatus embodying features of construction, combinations of elements and arrangement of parts that are adapted to affect such steps, all is exemplified in the following detailed disclosure, and the scope of the invention will be indicated in the claims. In addition to the objects mentioned, other obvious and apparent advantages of the invention will be reflected from the detailed specification and drawings.

[0018] FIG. 1 is a block diagram illustrating an example of the schematic configuration and operation of a system according to the present disclosure.FIG. 2 is a schematic diagram illustrating an example of lossless compression according to the present disclosure.FIG. 3 is a schematic diagram illustrating an example of lossless restoration according to the present disclosure.FIG. 4 is a schematic diagram illustrating a first example of 3D update detection according to the present disclosure.FIG. 5 is a schematic diagram illustrating a second example of 3D update detection according to the present disclosure.FIG. 6 is a schematic diagram illustrating a third example of 3D update detection according to the present disclosure.FIG. 7 is a schematic block diagram illustrating an example of the function configuration of a system according to the present disclosure.FIG. 8 is a flowchart illustrating an example of the operation of a system according to the present disclosure.FIG. 9 is a diagram illustrating an example of an object database employed in a system according to the present disclosure.FIG. 10 is a diagram illustrating an example of a graph database employed in a system according to the present disclosure.FIG. 11 is a schematic block diagram illustrating an example of the point cloud restoration implemented by a system according to the present disclosure.FIG. 12 is a schematic block diagram illustrating an example of the multi-modal node change detection implemented by a system according to the present disclosure.FIG. 13 is a schematic block diagram illustrating an example of the spatio-temporal scene understanding implemented by a system according to the present disclosure.Description of example embodiments

[0019] Hereinafter, the word "example" is used herein to mean "serving as an example, instance, or illustration". Any embodiment described herein as "example" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0020] In the present disclosure, a time stamp or timeline is assigned to a sequence of 3D data captured in a span of time. Hereinafter, a time stamp or timeline is defined as an instant of time when 3D data is captured. Although 3D data capture time indicates the start / end time of data capture in a span of time, it is generally treated as an instant of time. As appropriate, an instant of time may be also referred to as a time stamp or timeline. Different instants of time may be separated by hours, days, weeks, or months. Assuming the first instant of time when 3D data is captured is T0, the next 3D data may be captured two days later T1, the further next 3D data may be captured one month later T2, and so forth.

[0021] 1. Outline of example embodiment According to an example embodiment of the present disclosure, 3D objects and a time-series of graphs are separately stored on an object database and a graph database. The 3D objects are extracted from 3D point cloud with class labels generated by semantic segmentation. The time-series of graphs may have time stamps such that each graph corresponds to a scene captured at different instant of time T0, T1, T2, … .

[0022] Current 3D objects and a current graph are obtained based on 3D point cloud with class labels at a current instant of time. The current 3D objects correspond to the nodes of the current graph, respectively. The current 3D objects are merged with the existing 3D objects stored in the object database to store the merged 3D objects on the object database. In this manner, the redundant 3D objects present through the time-series of graphs may be merged into a single 3D object, thereby eliminating the redundancy to dramatically reduce the amount of 3D object point cloud data.

[0023] A 3D point cloud at an instant of time can be restored by: searching the object database for 3D objects corresponding respectively to the nodes of the graph of that instant of time; and arranging the found 3D objects according to the graph to restore the 3D point cloud. Since the above-mentioned merging operation is lossless compression, the 3D point cloud can be restored without data loss.

[0024] An object change can be detected by comparing the current 3D objects with the existing 3D objects in the above-mentioned merging step. Specifically, an object appearance change can be detected by comparing the object feature of a current 3D object with that of a corresponding existing 3D object. A node change can be detected by comparing the current graph and a previous graph. Object-based and graph-based comparisons are thus performed separately, allowing faster and efficient 3D update detection. Hereinafter, an example embodiment and examples of the present disclosure will be described referring to the accompanying drawings.

[0025] 2. System configuration As illustrated in Fig. 1, a system 100 includes a data capture unit 101 which sequentially captures the above-described scenes. The data capture unit 101 may be an RGB or RGB-D sensor, LiDAR (Light Detection And Ranging) sensor, other type of 3D sensor, or a combination of two or more sensors thereof.

[0026] The system 100 further includes an object database 102, a graph database 103, a tentative object database 104, a processor unit 105, and a program memory 106. The object database 102 is configured to store 3D object data with object identifier (ID). The graph database 103 is configured to store graph data with each node having object ID. The tentative object database 104 is configured to temporarily store current 3D object data generated at a current instant of time.

[0027] The processor unit 105 may include one or more processor, which executes programs stored in the program memory 106. The processor unit 105 may perform 3D structure generation, semantic segmentation, object extraction and graph creation as described later. The processor unit 105 may further perform node change detection, object change detection, and merging operation to implement effective and useful storage and lossless compression for multiple sequences of 3D data at different instants of time.

[0028] The system 100 may further include an input device such as a keyboard, pointing device, an output device such as a monitor, other storage device, and communication device. Since these devices are standard equipment on a computer system, they are omitted for simplicity in Fig. 1.

[0029] Referring to Fig. 1, it is assumed for simplicity that 3D data of a scene is sequentially captured at each time T0, T1, T2, …, where time T (time T0, T1, T2 and so on) denotes the instant of time, time stamp or timeline as described above. Taking an indoor scene as an example, the scene of time T0 may include three objects (CHAIR1, TABLE and DESK) in a first layout, the scene of time T1 may include the same objects (CHAIR1, TABLE and DESK) in a second layout, and the scene of time T2 may include the three objects (CHAIR1, TABLE, DESK) and a new object (CHAIR2) in a third layout. Hereinafter, the operations of the system 100 will be described referring to Figs. 2-5.

[0030] 3. Operation <Lossless compression> As illustrated in Fig. 2, the processor unit 105 generates a point cloud with labels at time T0 by 3D structure generation and semantic segmentation. Based on the point cloud with labels, three objects CHAIR1, DESK and TABLE are extracted and stored on the tentative object database 104. Further, a graph G(T0) including three nodes with class labels of CHAIR, DESK and TABLE is created and stored on the graph database 103. The relationship of three nodes of the graph G(T0) corresponds to the first layout of time T0. If no object is stored in the object database 102 at time T0, the objects CHAIR1, DESK and TABLE are transferred from the tentative object database 104 to the object database 102. Since the object database 102 and the graph database 103 are related to each other by object IDs, each node of the graph G(T0) corresponds to one of the objects stored in the object database 102.

[0031] Next, the processor unit 105 generates a point cloud with labels at time T1. Based on the point cloud with labels of time T1, three objects CHAIR1, DESK and TABLE are extracted and stored on the tentative object database 104. Further, a graph G(T1) including three nodes with class labels of CHAIR, DESK and TABLE is created and stored on the graph database 103. The relationship of three nodes in the graph G(T1) corresponds to the second layout of time T1 which is different from the first layout. The objects CHAIR1, DESK and TABLE of time T1 stored in the tentative object database 104 are the same as those stored in the object database 102. Accordingly, no additional object is registered in the object database 102. In this manner, the merging operation of the current objects stored in the tentative object database 104 and the existing objects stored in the object database 102 are performed.

[0032] Next, the processor unit 105 generates a point cloud with labels at time T2. Based on the point cloud with labels of time T2, four objects CHAIR1, DESK, TABLE and CHAIR2 are extracted and stored on the tentative object database 104. Further, a graph G(T2) including four nodes with class labels of CHAIR, DESK and TABLE is created and stored on the graph database 103. The relationship of four nodes in the graph G(T2) corresponds to the third layout of time T2 which is different from the first and second layouts. The objects CHAIR1, DESK, TABLE and CHAIR2 of time T2 stored in the tentative object database 104 are different from those stored in the object database 102. Since a new object CHAIR2 is added, only the object CHAIR2 is registered as a new entry in the object database 102 while the other objects CHAIR1, DESK, and TABLE remains in the object database 102. The new object CHAIR2 corresponds to the new node of the graph G(T2). Accordingly, each node of the graph G(T2) corresponds to one of the objects stored in the object database 102. In this manner, the current objects and the existing objects are merged.

[0033] As described above, merged objects are stored in the object database 102 and a time-series of graphs G(T0), G(T1), G(T2), … are stored in the graph database 103.　Since redundant objects are merged into a single object, each object is not subjected to lossy compression. Accordingly, the above-mentioned merging operation eliminates the redundancy of objects through time T0, T1, T2, … , allowing the load on the object database 102 to be significantly lightened without data loss.

[0034] <Lossless restoration> As illustrated in Fig. 3, the processor unit 105 can restore a point cloud without loss at a desired instant of time. As described above, the object database 102 stores the object data merged through the time-series of graphs and the graph database 103 stores the time-series of graphs. Given a certain time T1, the processor unit 105 reads the graph G(T1) from the graph database 103. Each node of the graph G(T1) has a corresponding object ID. The processor unit 105 uses these object IDs to read corresponding objects CHAIR1, DESK and TABLE from the object database 102. The lossless 3D object point clouds at time T1 can be restored from the graph G(T1) and the corresponding objects CHAIR1, DESK and TABLE. In this manner, the 3D point cloud at a desired instant of time can be restored without data loss.

[0035] <Update detection> As illustrated in Fig. 4, it is assumed that objects CHAIR1, DESK and TABLE change in layout from the first layout of time Ti to the second layout of time Tj. In this case, the objects CHAIR1, DESK and TABLE at time Ti have been stored on the object database 102. Further, a graph G(Ti) including three nodes with class labels of CHAIR, DESK and TABLE has been stored on the graph database 103. Subsequently, it is assumed that at time Tj, current objects CHAIR1, DESK and TABLE are generated and stored in the tentative object database 104 and a current graph G(Tj) is generated and stored in the graph database 103.

[0036] The processor unit 105 performs a graph comparison operation 201 of comparing the current graph G(Tj) with the stored previous graph G(Ti) to detect node change such that the node of CHAIR is relocated nearer to the node of DESK. On the other hand, the processor unit 105 performs an object feature comparison 202 of comparing the stored objects at time Ti and the current objects at time Tj to detect that no object change occurs. Accordingly, in this case, the processor unit 105 detects only the node change.

[0037] As illustrated in Fig. 5, it is assumed that a certain object changes in appearance from CHAIR1 of time Ti to CHAIR3 of time Tj. In this case, the objects CHAIR1, DESK and TABLE at time Ti have been stored on the object database 102. Further, a graph G(Ti) including three nodes with class labels of CHAIR, DESK and TABLE has been stored on the graph database 103. Subsequently, it is assumed that at time Tj, current objects CHAIR3, DESK and TABLE are generated and stored in the tentative object database 104 and a current graph G(Tj) is generated and stored in the graph database 103.

[0038] The processor unit 105 performs the graph comparison operation 201 of comparing the current graph G(Tj) with the stored previous graph G(Ti) to detect no node change. The processor unit 105 performs the object feature comparison 202 of comparing the stored objects at time Ti and the current objects at time Tj to detect that the appearance of object CHAIR changes from CHAIR1 to CHAIR3. Accordingly, in this case, the processor unit 105 detects only the object appearance change.

[0039] As illustrated in Fig. 6, it is assumed that a certain object changes in appearance from CHAIR1 of time Ti to CHAIR3 of time Tj and a new object CHAIR2 is added. In this case, the objects CHAIR1, DESK and TABLE at time Ti have been stored on the object database 102. Further, a graph G(Ti) including three nodes with class labels of CHAIR, DESK and TABLE has been stored on the graph database 103. Subsequently, it is assumed that at time Tj, current objects CHAIR2, DESK, TABLE, and CHAIR3 are generated and stored in the tentative object database 104 and a current graph G(Tj) is generated and stored in the graph database 103.

[0040] The processor unit 105 performs the graph comparison operation 201 of comparing the current graph G(Tj) with the stored previous graph G(Ti) to detect node change such that a new object 203 of CHAIR is added. The processor unit 105 performs the object feature comparison 202 of comparing the stored objects at time Ti and the current objects at time Tj to detect that the appearance of object CHAIR changes from CHAIR1 to CHAIR3 and a new object CHAIR2 is added. Accordingly, in this case, the processor unit 105 detects both the new node addition of CHAIR2 and the object appearance change from CHAIR1 to CHAIR3.

[0041] As described above, node change and / or object change can be detected early. Accordingly, in the case where the system 100 is installed in a factory, critical object changes in the factory (e.g. ladder disappeared) can be detected at an early stage, resulting in faster alarm activation for accident prevention.

[0042] 4. Example The effective storage and lossless compression function according to an example of the present disclosure can be implemented by the processor unit 105 executing programs thereon. A detailed function configuration of the processor unit 105 will be describe with reference to Figs. 7 and 8.

[0043] As illustrated in Fig. 7, the processor unit 105 implements the following functions: 3D structure generation 301, semantic segmentation 302, object extraction 303, graph creation 304, graph comparison 305, node change detection 306, new node registration 307, and node correspondence mapping 308.

[0044] Referring to Fig. 8, the processor unit 105 controls the data capture unit 101 to input sensor data from the data capture unit 101 and performs the 3D structure generation 301 to generate a 3D point cloud at a current instant of time TN(operation S401). Subsequently the semantic segmentation 302 generates a 3D point cloud with class labels (operation S402).

[0045] The object extraction 303 extracts current 3D objects OBJNand their features from the 3D point cloud with class labels and stores the current 3D objects OBJNon the tentative object database 104 (operation S403). Hereinafter, the current 3D objects OBJNare also referred to as detected objects as appropriate.

[0046] The graph creation 304 creates a current graph GRAPHNfrom the detected objects and stores the current graph GRAPHNon the graph database 103 (operation S404). Subsequently the graph comparison 305 reads a previous graph GRAPHN-1with a previous time TN-1from the graph database 103 and compares the current graph GRAPHNwith the previous graph GRAPHN-1. The node change detection 306 detects the presence or absence of node change from a result of the graph comparison 305 (operation S405).

[0047] In the node change detection 306, nodes existing in both of the current graph GRAPHNand the previous graph GRAPHN-1are notified as existing nodes to the object correspondence mapping 308. New nodes detected by the node change detection 306 are notified to the new node registration 307.

[0048] The object correspondence mapping 308 inputs the detected objects (current 3D objects OBJN) from the tentative object database 104 and the existing 3D objects OBJN-1from the object database 102. The object correspondence mapping 308 establishes object correspondence by feature comparison of the detected objects (current 3D objects OBJN) and the existing 3D objects OBJN-1for the existing nodes (operation S406). More specifically, the object correspondence mapping 308 compares the detected object features of the detected objects (current 3D objects OBJN) and the existing object features of the existing 3D objects OBJN-1for the existing nodes. Based on the feature comparison result, the object correspondence mapping 308 determines matched objects and no-match objects among objects corresponding to the existing nodes. The matched objects are notified to the object database 102 and no-match objects are notified to the new node registration 307.

[0049] The new node registration 307 inputs the new nodes from the node change detection 306 and / or no-match objects from the object correspondence mapping 308. Based on the new nodes and / or no-match objects, the new node registration 307 determines an update object to be registered from the detected objects (current 3D objects OBJN) and registers the update object(s) on the object database 102 (operation S407). More specifically, the new node registration 307 registers only objects which are new or which have change in appearance. Whereas, for the objects which are matched in node correspondence (no change), no action is taken for them as they already exist in the object database 102. The operations S406 and S407 performs the merging of the detected objects (current 3D objects OBJN) and the existing 3D objects OBJN-1.

[0050] <Object database> As illustrated in Fig. 9, the object database 102 stores a collection of 3D objects in a table containing rows and fields. The fields are object ID, class, object feature, and object. In the case of time stamp T2 (see Fig. 1), the object ID field indicates any one of CHAIR1, CHAIR2, TABLE and DESK. The class field indicates any one of CHAIR, TABLE and DESK. The object feature field indicates the feature of each object which is extracted by the object extraction 303. As described above, the object feature is used for object feature comparison 202 in the object correspondence mapping 308. The object field indicates the 3D point cloud, mesh, or other type of 3D data or a combination of two or more data of each object ID.

[0051] <Graph database> As illustrated in Fig. 10, the graph database 103 stores a collection of graphs in a table containing rows and fields. The fields includes object ID, neighbor ID, node feature (position), node feature (rotation), edge feature (distance), and edge feature (relationship). the object ID field indicates a 3D object extracted from 3D point cloud with labels. In the case of time stamp T2 (see Fig. 1), the object ID field indicates any one of CHAIR1, CHAIR2, TABLE and DESK.

[0052] The neighbor ID field indicates the ID of each object adjacent to each object. The node feature (position) field　indicates the position of each object as X, Y, and Z coordinates. The node feature (rotation) indicates the absolute rotation angle of each object in the 3D data as the angle of inclination with the X, Y and Z axis but other ways of representation can be used e.g. roll-pitch-yaw representation.. The edge feature (distance) field indicates the distance between each object and its neighboring object. The edge feature (relationship) field indicates the relationship between each object and its neighboring object in spatial representation.

[0053] Taking object ID = CHAIR1 as an example, neighbors of CHAIR1 are CHAIR2, DESK and TABLE; node feature (position) of CHAIR1 is denoted as (x1, y1, z1) in X, Y, and Z coordinate system; node feature (rotation) is the absolute angle of inclination of CHAIR1 with the X, Y and Z axis denoted as (θ1x, θ1y, θ1z); edge feature (distance) is the distances from CHAIR1 to neighbors CHAIR2, DESK and TABLE denoted as (d12, d13, d14); and edge feature (relationship) of neighbors CHAIR2, DESK and TABLE is denoted as (LEFT, RIGHT, FRONT).

[0054] <Restoration> As illustrated in Fig. 11, the processor unit 105 may implement the functions of object retrieval 309 and point cloud restoration 310. Given a desired time TN, the object retrieval 309 searches the graph database 103 for GRAPHNaccording to the time stamp TN. When obtaining GRAPHN, the object retrieval 309 uses the object IDs included in GRAPHNto search the object database 102 for corresponding object point clouds OBJN. The point cloud restoration 310 restores the point cloud at the time stamp TNby arranging the object point clouds OBJNcorresponding respectively to the nodes of GRAPHNaccording to the layout of GRAPHN. Specifically, the node features (e.g. position, rotation) and edge features (e.g. relationship) of the graph guide the object placement in the restored point cloud. The restored point cloud at the time stamp TNis stored in a storage 107 and may be displayed on the monitor of the system 100.

[0055] 5. Applications <Multi-modal comparison> The system 100 shown in Fig. 1 may be provided with a plurality of sensors as the data capture unit 101. These sensors may capture 3D data at different instants of time. For example, a first sensor captures 3D data at a previous instant of time and a second sensor captures 3D data at a current instant of time. Hereinafter, it is assumed that two sensors A and B are used as multi-modal sensors to capture 3D data at different instants of time.

[0056] A node change detection using the sensors A and B can be implemented by the processor unit 105 executing programs thereon. A function configuration of the processor unit 105 will be describe with reference to Fig. 12.

[0057] As illustrated in Fig. 12, the processor unit 105 further implements the following functions: pre-processing 501, semantic segmentation 502, object extraction 503, graph creation 504, graph comparison 505, and node change detection 506. The pre-processing 501 is configured to input 3D data DATAAand DATABfrom the respective sensors A and B and to process these 3D data in preparation for the semantic segmentation 502. Note that the functions of the semantic segmentation 502, object extraction 503, graph creation 504, graph comparison 505, and node change detection 506 are the same as those of the semantic segmentation 302, object extraction 303, graph creation 304, graph comparison 305, and node change detection 306 of Fig. 7, respectively.

[0058] The processor unit 105 controls the sensor A to input 3D data DATAAfrom the sensor A and performs the pre-processing 501 to generate a 3D point cloud at an instant of time TN-1. Subsequently the semantic segmentation 502 generates a 3D point cloud with class labels.

[0059] The object extraction 503 extracts 3D objects OBJN-1from the 3D point cloud with class labels and stores the 3D objects OBJN-1on the tentative object database 104. The graph creation 504 creates a graph GRAPHAN-1from the 3D objects OBJN-1and stores the graph GRAPHAN-1on the graph database 103.

[0060] At the next instant of time TN, the processor unit 105 controls the sensor B to input 3D data DATABfrom the sensor B and performs the pre-processing 501 to generate a 3D point cloud at the instant of time TN. Subsequently the semantic segmentation 502 generates a 3D point cloud with class labels.

[0061] The object extraction 503 extracts 3D objects OBJNfrom the 3D point cloud with class labels and stores the 3D objects OBJNon the tentative object database 104. The graph creation 504 creates a graph GRAPHBNfrom the 3D objects OBJN.

[0062] Subsequently the graph comparison 505 reads the previous graph GRAPHAN-1from the graph database 103 and compares the graph GRAPHBNwith the previous graph GRAPHAN-1. The node change detection 506 detects the presence or absence of node change from a result of the graph comparison 505. When the node change occurs, the changed node is stored on a cross-modal change database 108.

[0063] The multi-modal sensors A and B may employ any one of the following combinations: LiDAR sensor and RGB-D camera; LiDAR sensor and RGB camera; and RGB camera and RGB-D camera.

[0064] As described above, the multi-modal comparison can be done effectively in graph representation. Using the multi-modal sensors can reduce sensing cost and provide flexibility.

[0065] <Spatio-temporal scene understanding> A Spatio-temporal scene understanding with large language model (LLM) can be implemented by the processor unit 105 executing programs thereon. A function configuration of the processor unit 105 will be describe with reference to Fig. 13.

[0066] As illustrated in Fig. 13, the processor unit 105 further implements the functions of object retrieval 311 and LLM 312. The object retrieval 311 sequentially reads a time-series of graphs GRAPHT(T=0, 1, …, N) from the graph database 103. Using object IDs included in the graphs GRAPHT, the object retrieval 311 searches the object database 102 for 3D objects OBJ and features corresponding to the object IDs. The 3D objects OBJ, their features and the graph GRAPHTare output to the LLM 312. The LLM 312 inputs these data to generate scene understanding output, which may be displayed on the monitor of the system 100. For example, the scene understanding output may be “In a room, .. there was a chair to the left of desk at time T=N which moved behind the desk at time T=N-1”.

[0067] 6. Additional statements The present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The above-described illustrative embodiment and examples are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Part or all of the above-described illustrative embodiments can also be described as, but are not limited to, the following additional statements.

[0068] (Additional statement 1) A system comprising: an object database configured to store three-dimensional (3D) objects; a graph database configured to store a time-series of graphs; and at least one processor configured to: a) input sensor data from at least one sensor; b) generate a point cloud with class labels from the sensor data by semantic segmentation; c) extract current 3D objects from the point cloud with class labels; d) create a current graph based on nodes and their relationships in the point cloud to store the current graph on the graph database; and e) merge the current 3D objects with 3D objects existing in the object database to store merged 3D objects on the object database, wherein each node included in a graph of the time-series of graphs corresponds to one of the merged 3D objects.

[0069] (Additional statement 2) The system according to additional statement 1, wherein the at least one processor is further configured to detect a node change by comparing the current graph with a previous graph.

[0070] (Additional statement 3) The system according to additional statement　2, wherein the at least one processor is further configured to, when a new node is detected, register a 3D object corresponding to the new node in the object database.

[0071] (Additional statement 4) The system according to additional statement 2 or 3, wherein the at least one processor is further configured to detect an object change by comparing the current 3D objects with the existing 3D objects.

[0072] (Additional statement 5) The system according to additional statement 4, wherein the at least one processor is further configured to detect an object appearance change by comparing the current 3D objects with the existing 3D objects for nodes existing in both of the current graph and the previous graph.

[0073] (Additional statement 6) The system according to additional statement　5, wherein the at least one processor is further configured to, if a no-match 3D object is detected, register the no-match 3D object as an update object in the object database.

[0074] (Additional statement 7) The system according to any one of additional statements 1-6, wherein the at least one processor is further configured to restore the point cloud of a desired instant of time from 3D objects corresponding respectively to nodes included in a graph of the desired instant of time.

[0075] (Additional statement 8) The system according to any one of additional statements 1-7, further comprising a plurality of sensors including a first sensor and a second sensor, wherein the at least one processor is further configured to: perform the a) - d) to generate a first graph from first sensor data of the first sensor and a second graph from second sensor data of the second sensor; and compare the first graph and the second graph to detect an node change.

[0076] (Additional statement 9) The system according to any one of additional statements 1-7, wherein the at least one processor is further configured to: read the time-series of graphs from the graph database; read the merged 3D objects from the object database; and process the time-series of graphs and the merged 3D objects using large language model (LLM) for scene understanding.

[0077] (Additional statement 10) The system according to additional statement 1, wherein the e) comprises: e.1) comparing the current graph with a previous graph to determine whether a node change occurs; e.2) comparing the current 3D objects with the existing 3D objects to determine whether an object change occurs; and e.3) responsive to occurrence of at least one of the node change and the object change, generating the merged 3D objects by reflecting the at least one of the node change and the object change in the object database.

[0078] (Additional statement 11) The system according to additional statement 10, wherein in the e.1), if it is determined that the node change occurs, at least one new node is detected in the current graph with respect to the previous graph.

[0079] (Additional statement 12) The system according to additional statement 11, wherein comparison of the e.2) is performed for at least one existing node common between the current graph and the previous graph to determine whether an object appearance change occurs.

[0080] (Additional statement 13) The system according to additional statement　12, wherein if a no-match 3D object is detected, the no-match 3D object is registered as an update object in the object database.

[0081] (Additional statement 14) The system according to additional statement　11, wherein if a new node is detected, a 3D object corresponding to the new node is registered in the object database.

[0082] (Additional statement 15) The system according to any one of additional statements 10-14, wherein the at least one processor is further configured to restore the point cloud of a desired instant of time from 3D objects corresponding respectively to nodes included in a graph of the desired instant of time.

[0083] (Additional statement 16) The system according to any one of additional statements 10-15, further comprising a plurality of sensors including a first sensor and a second sensor, wherein the at least one processor is further configured to: perform the a) - d) to generate a first graph from first sensor data of the first sensor and a second graph from second sensor data of the second sensor; and compare the first graph and the second graph to detect an node change.

[0084] (Additional statement 17) The system according to any one of additional statements 10-15, wherein the at least one processor is further configured to: read the time-series of graphs from the graph database; read the merged 3D objects from the object database; and process the time-series of graphs and the merged 3D objects using large language model (LLM) for scene understanding.

[0085] (Additional statement 18) A method for processing three-dimensional (3D) scene in a system including an object database configured to store 3D objects, a graph database configured to store a time-series of graphs, and at least one processor, the method comprising: by the at least one processor, a) inputting sensor data from at least one sensor; b) generating a point cloud with class labels from the sensor data by semantic segmentation; c) extracting current 3D objects from the point cloud with class labels; d) creating a current graph based on nodes and their relationships in the point cloud to store the current graph on the graph database; and e) merging the current 3D objects with 3D objects existing in the object database to store merged 3D objects on the object database, wherein each node included in a graph of the time-series of graphs corresponds to one of the merged 3D objects.

[0086] (Additional statement 19) The method according to additional statement 18, further comprising: detecting a node change by comparing the current graph with a previous graph.

[0087] (Additional statement 20) The method according to additional statement　19, further comprising: when a new node is detected, registering a 3D object corresponding to the new node in the object database.

[0088] (Additional statement 21) The method according to additional statement 19 or 20, further comprising: detecting an object change by comparing the current 3D objects with the existing 3D objects.

[0089] (Additional statement 22) The method according to additional statement 21, further comprising: detecting an object appearance change by comparing the current 3D objects with the existing 3D objects for nodes existing in both of the current graph and the previous graph.

[0090] (Additional statement 23) The method according to additional statement　22, further comprising: when a no-match 3D object is detected, registering the no-match 3D object as an update object in the object database.

[0091] (Additional statement 24) The method according to any one of additional statements 18-23, further comprising:　restoring the point cloud of a desired instant of time from 3D objects corresponding respectively to nodes included in a graph of the desired instant of time.

[0092] (Additional statement 25) The method according to any one of additional statements 18-24, wherein the system further includes a plurality of sensors including a first sensor and a second sensor, the method further comprises: performing the a) - d) to generate a first graph from first sensor data of the first sensor and a second graph from second sensor data of the second sensor; and comparing the first graph and the second graph to detect an node change.

[0093] (Additional statement 26) The method according to any one of additional statements 18-25, further comprising:　reading the time-series of graphs from the graph database; reading the merged 3D objects from the object database; and processing the time-series of graphs and the merged 3D objects using large language model (LLM) for scene understanding.

[0094] (Additional statement 27) The method according to additional statement 18, wherein the e) comprises: e.1) comparing the current graph with a previous graph to determine whether a node change occurs; e.2) comparing the current 3D objects with the existing 3D objects to determine whether an object change occurs; and e.3) responsive to occurrence of at least one of the node change and the object change, generating the merged 3D objects by reflecting the at least one of the node change and the object change in the object database.

[0095] (Additional statement 28) The method according to additional statement 27, wherein in the e.1), if it is determined that the node change occurs, at least one new node is detected in the current graph with respect to the previous graph.

[0096] (Additional statement 29) The method according to additional statement 28, wherein comparison of the e.2) is performed for at least one existing node common between the current graph and the previous graph to determine whether an object appearance change occurs.

[0097] (Additional statement 30) The method according to additional statement　29, wherein if a no-match 3D object is detected, the no-match 3D object is registered as an update object in the object database.

[0098] (Additional statement 31) The method according to additional statement　28, wherein if a new node is detected, a 3D object corresponding to the new node is registered in the object database.

[0099] (Additional statement 32) The method according to any one of additional statements 27-31, wherein the at least one processor is further configured to restore the point cloud of a desired instant of time from 3D objects corresponding respectively to nodes included in a graph of the desired instant of time.

[0100] (Additional statement 33) A computer-readable program for processing three-dimensional (3D) scene in a system including an object database configured to store 3D objects, and a graph database configured to store a time-series of graphs, the program comprising instructions to: a) input sensor data from at least one sensor; b) generate a point cloud with class labels from the sensor data by semantic segmentation; c) extract current 3D objects from the point cloud with class labels; d) create a current graph based on nodes and their relationships in the point cloud to store the current graph on the graph database; and e) merge the current 3D objects with 3D objects existing in the object database to store merged 3D objects on the object database, wherein each node included in a graph of the time-series of graphs corresponds to one of the merged 3D objects.

[0101] (Additional statement 34) The computer-readable program according to additional statement 33, wherein the e) comprises: e.1) comparing the current graph with a previous graph to determine whether a node change occurs; e.2) comparing the current 3D objects with the existing 3D objects to determine whether an object change occurs; and e.3) responsive to occurrence of at least one of the node change and the object change, generating the merged 3D objects by reflecting the at least one of the node change and the object change in the object database.

[0102] (Additional statement 35) A non-transitory recording medium storing a computer-readable program for processing three-dimensional (3D) scene in a system including an object database configured to store 3D objects, a graph database configured to store a time-series of graphs, and at least one processor, the program comprising instructions to: a) input sensor data from at least one sensor; b) generate a point cloud with class labels from the sensor data by semantic segmentation; c) extract current 3D objects from the point cloud with class labels; d) create a current graph based on nodes and their relationships in the point cloud to store the current graph on the graph database; and e) merge the current 3D objects with 3D objects existing in the object database to store merged 3D objects on the object database, wherein each node included in a graph of the time-series of graphs corresponds to one of the merged 3D objects.

[0103] (Additional statement 36) The non-transitory recording medium according to additional statement 35, wherein the e) comprises: e.1) comparing the current graph with a previous graph to determine whether a node change occurs; e.2) comparing the current 3D objects with the existing 3D objects to determine whether an object change occurs; and e.3) responsive to occurrence of at least one of the node change and the object change, generating the merged 3D objects by reflecting the at least one of the node change and the object change in the object database.

[0104] Application software in accordance with the present disclosure, such as computer programs executed by the device and may be stored on one or more computer readable mediums. It is also contemplated that the steps identified herein may be implemented using one or more general purpose or specific purpose computers and / or computer systems, networked and / or otherwise. Where applicable, the ordering of various steps described herein may be changed, combined into composite steps, and / or separated into sub-steps to provide features described herein.

[0105] It should also be understood that example embodiments of the present disclosure should not be limited to these embodiments but that numerous modifications and variations may be made by one of ordinary skill in the art in accordance with the principles of the present disclosure and be included within the spirit and scope of the present disclosure as hereinafter claimed.

[0106] The above exemplary embodiments can be applied to 3D scene understanding systems.

[0107] 100 System 101 Data capture unit 102 Object database 103 Graph database 104 Tentative object database 105 Processor unit 106 Program memory 201 Graph comparison 202 Object feature comparison 203 New node 301 3D structure generation 302 Semantic segmentation 303 Object extraction 304 Graph creation 305 Graph comparison 306 Node change detection 307 New node registration 308 Node correspondence mapping

Claims

1. A system comprising: an object database configured to store three-dimensional (3D) objects; a graph database configured to store a time-series of graphs; and at least one processor configured to: a) input sensor data from at least one sensor; b) generate a point cloud with class labels from the sensor data by semantic segmentation; c) extract current 3D objects from the point cloud with class labels; d) create a current graph based on nodes and their relationships in the point cloud to store the current graph on the graph database; and e) merge the current 3D objects with 3D objects existing in the object database to store merged 3D objects on the object database, wherein each node included in a graph of the time-series of graphs corresponds to one of the merged 3D objects.

2. The system according to claim 1, wherein the at least one processor is further configured to detect a node change by comparing the current graph with a previous graph.

3. The system according to claim　2, wherein the at least one processor is further configured to, when a new node is detected, register a 3D object corresponding to the new node in the object database.

4. The system according to claim 2 or 3, wherein the at least one processor is further configured to detect an object change by comparing the current 3D objects with the existing 3D objects.

5. The system according to claim 4, wherein the at least one processor is further configured to detect an object appearance change by comparing the current 3D objects with the existing 3D objects for nodes existing in both of the current graph and the previous graph.

6. The system according to claim　5, wherein the at least one processor is further configured to, if a no-match 3D object is detected, register the no-match 3D object as an update object in the object database.

7. The system according to any one of claims 1-6, wherein the at least one processor is further configured to restore the point cloud of a desired instant of time from 3D objects corresponding respectively to nodes included in a graph of the desired instant of time.

8. The system according to any one of claims 1-7, further comprising a plurality of sensors including a first sensor and a second sensor, wherein the at least one processor is further configured to: perform the a) - d) to generate a first graph from first sensor data of the first sensor and a second graph from second sensor data of the second sensor; and compare the first graph and the second graph to detect an node change.

9. The system according to any one of claims 1-7, wherein the at least one processor is further configured to: read the time-series of graphs from the graph database; read the merged 3D objects from the object database; and process the time-series of graphs and the merged 3D objects using large language model (LLM) for scene understanding.

10. The system according to claim 1, wherein the e) comprises: e.1) comparing the current graph with a previous graph to determine whether a node change occurs; e.2) comparing the current 3D objects with the existing 3D objects to determine whether an object change occurs; and e.3) responsive to occurrence of at least one of the node change and the object change, generating the merged 3D objects by reflecting the at least one of the node change and the object change in the object database.

11. The system according to claim 10, wherein in the e.1), if it is determined that the node change occurs, at least one new node is detected in the current graph with respect to the previous graph.

12. The system according to claim 11, wherein comparison of the e.2) is performed for at least one existing node common between the current graph and the previous graph to determine whether an object appearance change occurs.

13. The system according to claim　12, wherein if a no-match 3D object is detected, the no-match 3D object is registered as an update object in the object database.

14. The system according to claim　11, wherein if a new node is detected, a 3D object corresponding to the new node is registered in the object database.

15. The system according to any one of claims 10-14, wherein the at least one processor is further configured to restore the point cloud of a desired instant of time from 3D objects corresponding respectively to nodes included in a graph of the desired instant of time.

16. The system according to any one of claims 10-15, further comprising a plurality of sensors including a first sensor and a second sensor, wherein the at least one processor is further configured to: perform the a) - d) to generate a first graph from first sensor data of the first sensor and a second graph from second sensor data of the second sensor; and compare the first graph and the second graph to detect an node change.

17. The system according to any one of claims 10-15, wherein the at least one processor is further configured to: read the time-series of graphs from the graph database; read the merged 3D objects from the object database; and process the time-series of graphs and the merged 3D objects using large language model (LLM) for scene understanding.

18. A method for processing three-dimensional (3D) scene in a system including an object database configured to store 3D objects, a graph database configured to store a time-series of graphs, and at least one processor, the method comprising: by the at least one processor, a) inputting sensor data from at least one sensor; b) generating a point cloud with class labels from the sensor data by semantic segmentation; c) extracting current 3D objects from the point cloud with class labels; d) creating a current graph based on nodes and their relationships in the point cloud to store the current graph on the graph database; and e) merging the current 3D objects with 3D objects existing in the object database to store merged 3D objects on the object database, wherein each node included in a graph of the time-series of graphs corresponds to one of the merged 3D objects.

19. The method according to claim 18, wherein the e) comprises: e.1) comparing the current graph with a previous graph to determine whether a node change occurs; e.2) comparing the current 3D objects with the existing 3D objects to determine whether an object change occurs; and e.3) responsive to occurrence of at least one of the node change and the object change, generating the merged 3D objects by reflecting the at least one of the node change and the object change in the object database.

20. A computer-readable program for processing three-dimensional (3D) scene in a system including an object database configured to store 3D objects, and a graph database configured to store a time-series of graphs, the program comprising instructions to: a) input sensor data from at least one sensor; b) generate a point cloud with class labels from the sensor data by semantic segmentation; c) extract current 3D objects from the point cloud with class labels; d) create a current graph based on nodes and their relationships in the point cloud to store the current graph on the graph database; and e) merge the current 3D objects with 3D objects existing in the object database to store merged 3D objects on the object database, wherein each node included in a graph of the time-series of graphs corresponds to one of the merged 3D objects.

Images