Apparatus and method for generating graphical representation from three-dimensional point cloud

Abstract

A method for training a first machine learning system for generating a graphical representation of objects and their relationships in a 3D environmental scene from 3D point cloud input data. For each object i and each object pair i and j in the scene, an initial node feature vector [phi] i and an initial edge feature vector [phi] ij are determined from the point cloud input data and arranged in an initial graphic structure ([phi] i, [phi] ij, [phi] j). And determining a refined graph structure through the graph neural network. From two-dimensional image sensor data of an environmental scene, a feature vector of an object i is determined by a second machine learning system and a feature vector of an object pair i and j is determined by a third machine learning system. Parameters of the first machine learning system are adjusted relative to a training target defined by optimizing a difference between a refined node feature vector of an object i and a corresponding feature vector of the object i and / or optimizing a difference between refined edge feature vectors of objects i and j and corresponding feature vectors of object pairs i and j.

Description

Technical Field

[0001] The present invention relates to a method for training a machine learning system, a corresponding computer program and a machine-readable storage medium for generating a graphical representation of instances and their relationships in a three-dimensional environment scene from three-dimensional point cloud input data. Background Art

[0002] Three-dimensional scene graphs are an emerging graph-based representation that facilitates various 3D scene understanding tasks. The advantage of 3D scene graphs is that they can represent relationships between scene entities, such as settings in indoor scenes (e.g., enclosed buildings or enclosed spaces (e.g., rooms)) or instance objects in outdoor scenes. However, the use of 3D scene graphs in navigation tasks, decision making, or task planning in robotics is limited by their usability.

[0003] Due to their complexity and high level of abstraction, predicting 3D scene graphs can be difficult for learning models that rely on small-scale training datasets that include instances and relationships between them. To address this issue, open vocabulary training settings such as CLIP (https: / / arxiv.org / abs / 2103.00020) that rely on visual language models (VLMs) have been proposed, which can allow processing and generating words and descriptions of objects and their relationships without being limited to a fixed vocabulary.

[0004] In https: / / arxiv.org / abs / 2309.15940, an implicit scene graph representation for planning navigation tasks is proposed, which relies on human input when determining the relationship between instances in the graph. Another approach (such as https: / / arxiv.org / abs / 2309.16650) relies on a two-dimensional VLM to process the input data at inference time. Summary of the Invention

[0005] According to a first aspect, the present invention relates to a method for training a first machine learning system for generating a graphical representation of instances and their relationships in a three-dimensional environment scene from three-dimensional point cloud input data P. The graphical representation may be a data structure representing instances and their relationships in a (three-dimensional) environment scene. The graphical representation may include nodes and edges, where a node may refer to or represent an instance. An edge may connect two nodes, where an edge may refer to / represent / indicate a relationship between two instances associated with the nodes connected by the corresponding edge. An instance may be an object in a (three-dimensional) environment scene, and an instance pair may be a pair of objects in the scene. The three-dimensional point cloud input data may be a set / group of data points representing the environment scene, where each data point has three spatial coordinates. The first machine learning system includes two preprocessing networks and a graph neural network. The two preprocessing networks may be provided by PointNet, https: / / arxiv.org / abs / 1612.00593. Preferably, the two preprocessing networks may be point encoding networks that may take a three-dimensional point cloud as input and determine feature vectors for instances and / or instance pairs represented by a (sub)set of points in the three-dimensional input point cloud. In general, in the context of the invention presented herein, a feature vector may be an embedding. An embedding may be a representation of (a subset of) input data, such as a three-dimensional point cloud or two-dimensional image sensor data (in a lower dimensional space). An embedding may be given by a mathematical transformation that maps the input data into a vector space, where each dimension of a vector in the vector space represents a specific feature or characteristic of the input data. The method comprises the following steps. In the method steps, for each instance i in the scene, an initial node feature vector φ is determined from the point cloud input data P by a first pre-processing network. i In addition, for each instance pair i and j in the scene, the initial edge feature vector φ is determined from the point cloud input data by the second pre-processing network ij The (initial) node feature vector may encode the features of an instance, and the (initial) edge feature vector may encode the features of an instance pair. In other words, the features of an instance pair may be given by the relationship between or between the two instances of the instance pair. The features of an instance pair i and j may be the spatial, comparative and / or semantic relationship between the two instances i and j. In another method step, by constructing a triple (φ i 、φ ij 、φ j) Arrange the initial node and initial edge feature vectors in an initial graph structure. In a subsequent method step, a graph neural network determines a refined graph structure based on the initial graph structure. The refined graph structure includes refined node feature vectors and refined edge feature vectors. The graph neural network can process and refine the edge feature vectors and the node feature vectors and can project them into a latent space suitable for the training objectives defined below. In another method step, a second machine learning system determines a feature vector of instance i for each instance i in a scene from two-dimensional image sensor data. The two-dimensional image sensor data refers to a three-dimensional environment scene. In other words, the two-dimensional image sensor data is aligned with the three-dimensional environment scene. Therefore, the two-dimensional image sensor data can depict / illustrate the same shape, pose, and / or perspective of instances and instance pairs in the scene, which can be included in the three-dimensional point cloud input data. The second machine learning system can be a visual language model (VLM). The VLM can be trained on a large dataset including pairs of different modalities describing the same context (e.g., an image and a corresponding text description). During training, the VLM can learn to map embeddings of different modalities (e.g., text and image) into a shared vector space. Similar or related concepts from different modalities can be represented by similar or neighboring vectors in a shared space. In addition, for each instance pair i and j, in a further step, a feature vector of the instance pair i and j is determined by a third machine learning system from the two-dimensional image sensor data. The third machine learning system can also be a VLM. In another method step, the parameters of the first machine learning system are adjusted relative to a training objective, wherein the training objective is defined by optimizing the difference between the refined node feature vector of instance i and the corresponding feature vector of instance i for all instances and / or optimizing the difference between the refined edge feature vector of instance i and j and the corresponding feature vector of instance pair i and j for all instance pairs i and j. Preferably, the training objective includes minimizing a loss function based on a cosine similarity loss, wherein for all instances, the cosine similarity between the refined node feature vector of instance i and the corresponding feature vector of instance i will be maximized, and / or for all instance pairs, the cosine similarity between the refined edge feature vector of instance i and j and the corresponding feature vector of instance pair i and j will be maximized.

[0006] Advantageously, the method proposed herein allows for predicting / constructing a graphical representation of instances and their relationships in a three-dimensional environment scene directly from three-dimensional point cloud input data. This data can be acquired by a LiDAR sensor or a RADAR sensor and can be processed by a first machine learning system directly or after preprocessing (e.g., removing artifacts in the raw data) to construct a refined graph structure, wherein the refined graph structure can encode instances and relationships between instances in the three-dimensional environment scene. This allows for determining the graph structure when only three-dimensional point cloud data from the sensor is available. In addition, the first machine learning system trained according to the method described herein does not require a computationally expensive VLM to construct a 3D map of the three-dimensional environment scene from two-dimensional (video) camera data at inference time. In addition, as part of the proposed method, explicit semantic relationships can be predicted, where multiple options for storing descriptions of relationships per edge may become obsolete.

[0007] Preferably, the first and second preprocessing networks may include PointNet. PointNet (https: / / arxiv.org / abs / 1612.00593) may take a 3D point cloud as input and determine an embedding based on the input point cloud. In the preprocessing step, the instance mask M may be used to i Extract point cloud P from input point cloud P i or P ij , which include subsets of points of the input point cloud P and refer to instance i or instance pair i and j respectively. The 3D point cloud P is preprocessed into point cloud P i This can be performed using instance segmentation methods such as Mask3D [https: / / arxiv.org / abs / 2210.03105]. The point cloud P can then be determined by selecting all points that fall within the union of their respective bounding boxes. ij Then, PointNet can receive the point cloud P i , and determine the corresponding initial node feature vectors from these point clouds. Each point cloud P ij can be connected with a mask, where the mask can be equal to 1 if the corresponding point corresponds to instance i, 2 if the point corresponds to instance j, and 0 otherwise. PointNet can then receive the connected point cloud P ij , to determine the initial edge feature vectors corresponding to instances and / or the encoded relations between instances.

[0008] Preferably, the second machine learning system can be an OpenSeg [https: / / arxiv.org / abs / 2112.12143] model, and the third machine learning system can be an InstructBLIP [https: / / arxiv.org / abs / 2305.06500] model. Using the OpenSeg model as the second machine learning system can be advantageous because it provides language-aligned feature embeddings for each pixel in the image sensor data. InstructBLIP may be particularly suitable as the third machine learning system due to its aligned image data encoder and its language output decoder. In this way, the relationship between instance pairs i and j can be encoded and thus extracted / predicted by the methods described herein at a level with a good compositional understanding of the scene, and in particular the relationship between the instances of instance pair i and j.

[0009] Preferably, the 3D point cloud input data is acquired using a LiDAR sensor, a RADAR sensor, a camera with a depth sensor, or a video camera with a depth sensor. If the 3D point cloud data is acquired using a LiDAR or RADAR sensor, this data can be used as input data directly or after preprocessing (e.g., to remove artifacts from the data). If the (video) camera sensor data contains depth information, these images need to be processed to convert the data into 3D point cloud input data.

[0010] Preferably, the method may further comprise the step of controlling a robot based on the refined graph structure, wherein the refined node feature vector and the refined edge feature vector are determined by the first machine learning system after adjusting parameters of the first machine learning system relative to a training objective. Examples of robots include industrial robots used in manufacturing, at least partially autonomous vehicles, and household or garden robots used for cleaning or mowing lawns.

[0011] Preferably, the step of determining, by the second machine learning system, for each instance i in the scene, a feature vector of instance i from the two-dimensional image sensor data as input may include the following sub-steps for each instance i. In the sub-steps, a set of k image sensor data including instance i may be determined from the two-dimensional image sensor data. Preferably, instance i as a whole is visible with a high percentage in each of the k image sensor data. Therefore, the k image sensor data may be selected as the image sensor data with the highest visibility percentage of instance i among all the image sensor data. In the next sub-step, a feature vector of instance i may be determined from each of the k image sensor data including instance i. Finally, the feature vector of instance i may be obtained by averaging the k determined feature vectors of instance i. Preferably, determining, by the third machine learning system, a feature vector for each instance pair i and j from two-dimensional image sensor data includes the following sub-steps. In a first step, a set of m image sensor data including the instance pair i and j is determined from the two-dimensional image sensor data. The m image sensor data may be determined as sensor data with the highest visibility of both instances i and j among all image sensor data. Furthermore, a bounding box of instance i and instance j is determined in each of the m image sensor data including the instance pair i and j. In a next step, each of the m image sensor data including the bounding box of instance i and instance j is cropped at n different scales to obtain n different cropped image sensor data for each of the m image sensor data. Therefore, each cropped image sensor data may include the bounding box of instance i and instance j. In a next step, a feature vector for the instance pair i and j is determined from each of the n different cropped image sensor data. Finally, the feature vector of instance pair i and j is obtained by first averaging n feature vectors from n different cropped image sensor data for each of the m image sensor data to obtain m average feature vectors for instance pair i and j, and then averaging the m obtained feature vectors for instance pair i and j.

[0012] Using n different cropped sensor data may be beneficial to encode and extract important contextual information between instances from the sensor data in order to determine the relationship between the two instances.

[0013] Preferably, after adjusting the parameters of the first machine learning system relative to the training objective, the refined node feature vectors and refined edge feature vectors of the refined graph structure are re-determined. This step can be performed during inference of the first machine learning system after the first machine learning system has been trained according to the previously described method steps to comply with the training objective. In other words, the trained first machine learning system can determine the refined node feature vectors and refined edge feature vectors from the 3D point cloud input data during inference. Furthermore, a candidate instance list can be provided, where each element of the candidate instance list is a word or text describing a possible instance in a 3D environment scene. Possible instances may not be included in the actual 3D environment scene, but may be instances that are typically likely to appear in 3D scenes. For example, in an indoor environment, possible instances may particularly be chairs, tables, beds, floors, etc., which may be part of a room or another enclosed environment. Furthermore, a fourth machine learning system, a third preprocessing network, and a fifth machine learning system can be provided. The third preprocessing network can be a QFormer (https: / / arxiv.org / abs / 2301.12597) network. The QFormer network is well suited for converting these features into the token space of the fifth machine learning system. The fourth machine learning system and the first machine learning system should map their respective input data to the same embedding space. In other words, the first and fourth machine learning systems should share the same embedding space. The first machine learning system may receive three-dimensional point cloud data as input and may be trained by a training objective to map this input data to a lower-dimensional vector representation in a corresponding embedding space, while the fourth machine learning system may receive words or text as input and map this input to the same embedding space. Therefore, the second and third machine learning systems also map their respective input data to the same embedding space as the fourth machine learning system. The fourth machine learning system may be a CLIP model. The method may further include the following steps: In this step, the fourth machine learning system determines an embedding for each element of a candidate instance list. The candidate instance list may be user-defined, wherein the user may input the candidate instance list through an interface, and the input list may be provided as input to the fourth machine learning system. Alternatively or additionally, the candidate instance list or a portion of the candidate instance list may be predefined based on a hard-coded list of words or text defining possible instances in the environment scenario, wherein the hard-coded list may be provided as input to the fourth machine learning system. In a further step, a graph structure having labeled nodes may be determined based on the refined graph structure. This can be achieved by assigning an element of the candidate list to the corresponding node of the refined node feature vector for each refined node feature vector having a corresponding node in the refined graph structure, wherein the assigned element can be a label of the corresponding node. The assignment can be based on the highest similarity between the embeddings of the refined node feature vector and the candidate list element.In other words, if the embedding of a list element has the highest similarity / lowest difference with the refined node feature vector corresponding to the corresponding node, then the list element can be selected and assigned as the label of the corresponding node. As a measure of the highest similarity / lowest difference, a cosine similarity score between the node feature vector and the text embedding from the list can be used. The list element with the highest cosine similarity can be assigned to the node, thereby providing a prediction for the node, where the prediction relates to the type or class of the instance associated with the node. In another method step, a third preprocessing network can determine input tokens based on the refined edge feature vectors, a predefined query, and relationship hints. The relationship hints can include labels assigned to nodes that are connected by corresponding edges with labeled nodes in the graph structure. Based on the determined input tokens, predefined query, and relationship hints, a fifth machine learning system can determine a textual description of each refined edge feature vector in a further step. In a next step, a scene graph can be determined from the graph structure with labeled nodes by assigning the determined textual description of each refined edge feature vector to the corresponding edge of the graph structure with labeled nodes.

[0014] The proposed method allows determining a scene graph through a two-step prediction, where instances in a three-dimensional scene are treated as nodes and the relationships between pairs of instances in the scene are treated as edges between two nodes. In the first step, the instance text labels, i.e., textual descriptions of objects in the scene in terms of words or a few words, can be determined. The textual descriptions, i.e., the words or a few words that describe the objects / instances in the scene can also be referred to as instance classes. In the second step, the relationships between instances can be predicted / determined by providing edge feature vectors, encoding the relationships between instances of a given pair of instances, and the previously determined instance labels as context to a large language model (LLM). Predicting relationships between instances requires a compositional understanding of the scene. The proposed method addresses this problem by leveraging the generative power of a pre-trained LLM, which may have a good compositional understanding of the environment scene and can be queried to give a textual description of the relationships between instances in the scene.

[0015] Preferably, the relationship between two instances may be a spatial, support, semantic and / or comparative relationship.

[0016] Preferably, the method may further comprise the step of verifying the scene graph by a user and / or controlling the robot based on the scene graph.

[0017] According to another aspect, the present invention relates to a system configured to perform the method according to the above steps and / or features.

[0018] According to another aspect, the present invention relates to a computer program having machine-readable instructions which, when executed on one or more computers, cause the computer(s) to perform one of the computer-implemented methods described above and below. Furthermore, according to another aspect, the present invention relates to a machine-readable storage medium having the above-described computer program stored thereon. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Embodiments of the present invention will be discussed in more detail with reference to the following drawings. These drawings show: Figure 1 An exemplary information flow diagram illustrating an embodiment of a method of training a first machine learning system for generating a graphical representation of instances and their relationships in a three-dimensional environment scene; Figure 2 Another flow chart illustrating another embodiment of the method described herein; Figure 3 Another flow chart showing another embodiment; Figure 4 Another flow chart showing another embodiment; Figure 5 Another flow chart showing another embodiment; Figure 6 shows a flow chart of a method according to one embodiment; Figure 7 A system is configured to perform the method according to the embodiments described herein. DETAILED DESCRIPTION

[0020] Figure 1 A flowchart of an embodiment of a method for training a first machine learning system 10 is shown, which is used to generate a graphical representation of instances and their relationships in a three-dimensional environment scene 1 from three-dimensional point cloud input data P. The first machine learning system 10 includes a pre-processing network 101 and a pre-processing network 102, which can be given by PointNet. The first machine learning system 10 also includes a graph neural network 103.

[0021] The first pre-processing network 101 can determine the initial node feature vector φ from the point cloud input data P for each instance i in scene 1 i The second pre-processing network 102 can determine the initial edge feature vector φ from the point cloud input data P for each instance pair i and j in scene 1 ij The initial node and initial edge feature vectors can be constructed by constructing a triplet (φ i 、φ ij 、φ j) are arranged in an initial graph structure, and a refined graph structure 11 including refined node feature vectors 12 and refined edge feature vectors 13 can be determined by the graph neural network 103 based on the initial graph structure. The second machine learning system 20 can determine a feature vector 21 of instance i from the two-dimensional image sensor data 2 for each instance i in the scene. The two-dimensional image sensor data can be aligned with the three-dimensional point cloud input data P of the environment scene 1. The third machine learning system 30 can determine a feature vector 31 of the instance pair i and j from the two-dimensional image sensor data 3 for each instance pair i and j. The two-dimensional image sensor data 2 and 3 can be the same data showing instances in the same pose and the same perspective. However, the image sensor data 3 can also be cropped relative to the image sensor data 2.

[0022] The parameters of the first machine learning system 10 can be adjusted relative to a training objective, where the training objective can be defined by a cosine similarity loss. More generally, the training objective can be defined by optimizing the difference between the refined node feature vector 12 of instance i and the corresponding feature vector 21 of instance i for all instances and / or optimizing the difference between the refined edge feature vector 13 of instance i and j and the corresponding feature vector 31 of the instance pair i and j for all pairs of instances.

[0023] Figure 2 A flowchart illustrating an embodiment of a computer-implemented method for training a first machine learning system 10 to generate a graphical representation 11 of instances and their relationships in a three-dimensional environment scene 1 from three-dimensional point cloud input data P is provided. The three-dimensional point cloud input data P may be provided with "class-agnostic instance annotations." In this context, "class-agnostic instance annotations" means that the point cloud input data P may be grouped by instances in the point cloud, but the class labels for the points grouped by instance are unknown. Therefore, the class of the instance may not be known at this point, but it may be known which points are grouped together to form an instance. The first machine learning system 10 may serve the purpose of determining the graphical representation 11 from the input point cloud p. The first machine learning system 10 may include first and second pre-processing networks 101 and 102, and a graph neural network 103. The first and second pre-processing networks 101 and 102 may function as point encoders that determine feature vectors for instances i and j (in the case of network 101) and for pairs of instances i, j (in the case of network 102). The first and second pre-processing networks 101 and 102 may share the same network architecture. The first and second processing networks 101 and 102 may be PointNets. However, the networks 101 and 102 may share the same architecture, but the weights may not be shared. The first pre-processing network 101 may determine an initial node feature vector φ for each instance i in the scene from the point cloud input data P. iSimilarly, the second pre-processing network 102 can determine the initial edge feature vector φ for each instance pair i and j in the scene from the point cloud input data P ij The determined initial node and initial edge feature vectors can be constructed by constructing a triplet (φ i 、φ ij 、φ j ) and are arranged in the initial graph structure and can be provided as input to the graph neural network 103. The graph neural network 103 can determine a refined graph structure including the refined node feature vectors 12 and the refined edge feature vectors 13 based on the initial graph structure. In other words, the initial node and edge feature vectors can be processed and refined by the graph neural network 103, and the graph neural network 103 can project the initial node and edge feature vectors into the refined node feature vectors 12 and the edge feature vectors 13. Figure 2 On the left side of , a second machine learning system 20 can determine a feature vector 21 of instance i for each instance i in the scene from the two-dimensional image sensor data 2. The second machine learning system 20 can be a visual language model that maps image sensor data into an embedding space. The VLM can map image sensor data and text input data into a joint embedding space, where images and text input data that show / describe the same context / subject are mapped to feature vectors with high similarity scores (i.e., high cosine similarity). Preferably, the second machine learning system 20 is given by an OpenSeg model. The feature vector 21 of instance i can also be referred to as the visual feature embedding of instance i. A third machine learning system 30 can determine a feature vector of instance pair i and j for each instance pair i and j from the two-dimensional image sensor data 3. Preferably, the third machine learning system 30 is given by a visual language model, such as, for example, BLIP / InstructBLIP. The feature vector 31 of instance pair i and j can also be referred to as the visual feature embedding of instance pair i, j. The two-dimensional image sensor data 2 and 3 should be aligned with the three-dimensional environment scene by projecting the three-dimensional environment scene into the image sensor views of the image sensor data systems 2 and 3 given the poses of the image sensor data systems 2 and 3 in 3D. The image sensor data 2 should show instance i. The image sensor data 3 should show not only a single instance, but also pairs of instances i and j. The parameters of the first machine learning system 10 can be adjusted relative to the training objective. The training objective can be defined by optimizing the difference between the refined node feature vector 12 of instance i and the corresponding feature vector 21 of instance i for all instances. And / or the training objective can be given by optimizing the difference between the refined edge feature vector 13 of instance pair i and j and the corresponding feature vector 31 of instance pair i and j for all instance pairs You can use the training target and / or Cosine similarity loss is used in order to adjust, i.e., “pull”, the graph feature space / embedding space of the first machine learning system 10 toward the embedding space of the second and third machine learning systems 20 and 30, i.e., the embedding space of the visual language model. Preferably, the second and third machine learning systems 20 and 30 should share the same embedding space.

[0024] Figure 3 An exemplary embodiment of sub-steps in the method described herein is shown, which involves determining, by a third machine learning system 30, a feature vector 31 for instance pair i and j from two-dimensional image sensor data 3 for each instance pair i and j. In a rough process of determining the feature vector 31, a set of m image sensor data including instance pair i and j can be determined from the two-dimensional image sensor data 3. In a next step, a bounding box of instance i and instance j can be determined in each of the m image sensor data 3 including instance pair i and j. In a next step, each of the m image sensor data with the bounding box of instance i and instance j can be cropped at n different scales to obtain n different cropped image sensor data for each of the m image sensor data. Therefore, each of the n cropped image sensor data can include the bounding box of instance i and instance j. From each of the n different cropped image sensor data, a feature vector φ for instance pair i and j can be determined. n By averaging n feature vectors φ from n different cropped image sensor data for each of the m image sensor data n , m average feature vectors 31a, 31b, 31c of instance pair i and j can be obtained. Finally, the feature vector 31a, 31b, 31c of instance pair i and j is averaged to obtain the feature vector 31 of instance pair i and j.

[0025] Figure 4 1 shows a specific part of an embodiment of the method described herein. The refined node feature vectors 12 and the refined edge feature vectors 13 of the refined graph structure 11 should be in accordance with the training target, for example, Figure 1-3The embodiments described in the context of [ 1 ] re-determine refined node features and refined edge feature vectors after adjusting the parameters of the first machine learning system. A candidate instance list 4 is provided, where each element of the candidate instance list is a word or text describing a possible instance in a three-dimensional environment scene. List 4 may, in particular, include "chair, table, bed, floor...", which describes possible elements in a three-dimensional environment scene. A fourth machine learning system 40 determines an embedding 41 for each element of candidate instance list 4. In other words, the fourth machine learning system maps the elements of list 4 to feature vectors / embeddings 41 in a lower-dimensional embedding space. Based on the refined graph structure 11, a graph structure 11A with labeled nodes is determined by assigning each refined node feature vector 12 with a corresponding node in the refined graph structure 11 to the corresponding node of the refined node feature vector 12. In this way, labeled nodes 12A labeled with the assigned elements are obtained. This assignment is based on the highest similarity between the refined node feature vector and the embedding of the candidate instance list element. In other words, the element of list 4 is selected and assigned to the corresponding node whose embedding has the highest similarity / lowest difference with the refined node feature vector 12. Cosine similarity can be used as a similarity metric. In other words, the list element with the highest cosine similarity between the text embedding provided by the fourth machine learning system 40 and the refined node feature vector 12 can be the prediction / class description of the corresponding node. The third preprocessing network 60 can determine the input token for the fifth machine learning system 50 based on the refined edge feature vector 13, the predefined query 6a, and the relationship hint 6b. The relationship hint 6b can include the labels 12A of the nodes connected by the corresponding edges in the graph structure with labeled nodes. An illustrative and non-limiting example of a relationship hint can be: "What is the relationship between [node i's label] and [node j's label]?" The third preprocessing network 60 can be a QFormer network. The fifth machine learning system 50 can determine a text description 13B for each refined edge feature vector 13 based on the determined input token, the predefined query 6a, and the relationship hint 6b. Therefore, a scene graph 11B can be determined from the graph structure with labeled nodes by assigning the text description 13B determined for each refined edge feature vector to the corresponding edge of the graph structure with labeled nodes. To enact this approach, the fourth machine learning system 40 and the first machine learning system 50 may map their respective input data to the same embedding space.

[0026] Figure 5 1 shows a portion of an embodiment of the method described herein. After adjusting the parameters of the first machine learning system relative to the training objective, a refined graph structure 11 having refined node feature vectors and refined edge feature vectors is determined. In other words, the refined graph structure 11 is generated by the trained machine learning model 10.

[0027] A candidate instance list 4 is provided, where each element of the candidate instance list is a word or text describing a possible instance in a three-dimensional environment scene. List 4 can be user-defined or system-provided. A fourth machine learning system 40 can determine an embedding 41 for each element of candidate instance list 4. Fourth machine learning system 40 can be provided, for example, by the language encoding portion of the VLM of CLIP. A graph structure 11A with labeled nodes 12A is determined based on the refined graph structure 11. This is done by assigning each element of candidate list 4 to a corresponding node in the refined node feature vector 12A of the refined graph structure 11 based on the highest similarity between the refined node feature vector 12 and the embeddings of the elements of candidate list 4. Graph 11A can include textual descriptions / words 12A describing instance i at its nodes. However, graph 11A still includes refined edge feature vectors at its edges. A third preprocessing network 60 can determine input tokens for a fifth machine learning system 50 based on the refined edge feature vectors 13, a predefined query 6a, and a relationship hint 6b. The relationship hint 6b may include labels of nodes that are connected by corresponding edges in the graph structure 11A with labeled nodes. The predefined query 6a may be pretrained and may guide the third preprocessing network 60 as well as the fifth machine learning system 50 to focus on relevant parts in the calculation. The predefined query 6a may be given by the IntructBLIP pre-include query. The third preprocessing network 60 may convert the refined edge feature vectors and the relationship hint into the token space of the fifth machine learning system 50. A non-limiting example of a relationship hint may be given by "What is the relationship between [label of node i] and [label of node j]?", where the labels are taken from the nodes of the graph 11A. Based on the determined input tokens, the predefined query 6a and the relationship hint 6b, the fifth machine learning system may determine a text description 13B for each refined edge feature vector. To formulate the method, preferably, the first, fourth and fifth machine learning systems may map their respective input data to the same embedding space. The fifth machine learning system may be the Vicuna 7B model (https: / / lmsys.org / blog / 2023-03-30-vicuna) using the Llama architecture (https: / / arxiv.org / abs / 2302.13971), which is arguably one of the best open-source language models available. Notably, the 7B refers to the 7 billion (trained) parameters of the Vicuna model. From a graph structure 11A with labeled nodes, a scene graph 11B may be determined by assigning a determined textual description 13B to a corresponding edge of a graph structure 12B with labeled nodes for each refined edge feature vector 13.

[0028] Figure 6In method step S1, an initial node feature vector φ for each instance i in the scene can be determined from point cloud input data P by a first pre-processing network. i , and the initial edge feature vector φ for each instance pair i and j in the scene can be determined from the point cloud input data by the second preprocessing network ij In method step S2, the triple (φ i 、φ ij 、φ j ) arrange the initial node and initial edge feature vectors in an initial graph structure. Method step S3 can be given by determining, by a graph neural network, a refined graph structure including refined node feature vectors and refined edge feature vectors based on the initial graph structure. In method steps S4a and S4b, which can be performed in parallel or not with method steps S1, S2 and / or S3, for each instance I in the scene, the feature vector of instance I can be determined by a second machine learning system from the two-dimensional image sensor data (step S4a), and for each instance pair i and j, the feature vector of the instance pair i and j can be determined by a third machine learning system from the two-dimensional image sensor data. In method step S5, the parameters of the first machine learning system can be adjusted relative to a training objective, wherein the training objective can be defined by optimizing the difference between the refined node feature vector of instance i and the corresponding feature vector of instance i for all instances and / or optimizing the difference between the refined edge feature vector of instance i and j and the corresponding feature vector of instance pair i and j for all instance pairs.

[0029] Figure 7 An exemplary embodiment of a data processing system 70 is shown, which includes at least one processor 71 and at least one machine-readable storage medium 72, the machine-readable storage medium 72 including instructions that, when executed by the processor 71, cause the data processing system 70 to perform a method according to one of the aspects or embodiments of the present invention described herein.

[0030] The term "computer" may be understood to cover any device for processing predefined computing rules. These computing rules may take the form of software, hardware, or a mixture of software and hardware.

[0031] In general, a complex number can be understood to be indexed, that is, each element in the complex number is assigned a unique index, preferably by assigning consecutive integers to the elements included in the complex number. Preferably, if the plurality includes N elements, where N is the number of elements in the plurality, then these elements are assigned integers from 1 to N. It will also be understood that the elements in the plurality can be accessed by their index.

Claims

1. A computer-implemented method for training a first machine learning system (10) for generating a graphical representation of instances and their relationships in a three-dimensional environment scene (1) from three-dimensional point cloud input data P, The first machine learning system (10) includes two preprocessing networks (101, 102) and a graph neural network (103), The method comprises the following steps: - The first pre-processing network (101) determines the initial node feature vector φ from the point cloud input data P for each instance i in the scene i , and the initial edge feature vector φ is determined from the point cloud input data P for each instance pair i and j in the scene by the second pre-processing network (102) ij (S1), -By constructing the triple (φ i 、φ ij 、φ j ) Arrange the initial node feature vectors and the initial edge feature vectors (S2) in the initial graph structure, - determining, by a graph neural network (103) based on the initial graph structure, a refined graph structure (11) comprising refined node feature vectors (12) and refined edge feature vectors (13) (S3), - determining, by a second machine learning system (20), for each instance i in the scene from the two-dimensional image sensor data (2) a feature vector (21) for the instance i (S4a), wherein the two-dimensional image sensor data (2) relates to the three-dimensional environment scene, and determining, by a third machine learning system (30), for each instance pair i and j from the two-dimensional image sensor data (3) a feature vector (31) for the instance pair i and j (S4b), - adjusting parameters of the first machine learning system (10) relative to a training objective, wherein the training objective is defined by optimizing the difference between the refined node feature vector (12) of instance i and the corresponding feature vector (21) of instance i for all instances and / or optimizing the difference between the refined edge feature vector (13) of instances i and j and the corresponding feature vector (31) of the instance pair i and j for all instance pairs (S5).

2. The method according to claim 1, wherein The first and second pre-processing networks (101, 102) are PointNets.

3. A method according to any one of the preceding claims, wherein The second machine learning system (20) is an OpenSeg model, and wherein the third machine learning system (30) is an InstructBLIP model.

4. A method according to any one of the preceding claims, wherein The 3D point cloud input data (P) is acquired using a LiDAR sensor, a RADAR sensor, a camera with a depth sensor, or a video camera with a depth sensor.

5. The method according to any one of the preceding claims, further comprising the steps of: - controlling a robot based on a refined graph structure (11), wherein refined node feature vectors (12) and refined edge feature vectors (13) are determined by the first machine learning system (10) after adjusting parameters of the first machine learning system (10) relative to the training objective.

6. The method according to any of the preceding claims, wherein determining, by the second machine learning system (20), for each instance i in the scene from the two-dimensional image sensor data (2) as input, a feature vector (21) of the instance i comprises the following steps for each instance i: - determining a set of k image sensor data comprising instance i from the two-dimensional image sensor data (2), - determining a feature vector for instance i from each of the k image sensor data comprising instance i, - The feature vector of instance i is obtained by averaging the k determined feature vectors of instance i (21).

7. A method according to any one of the preceding claims, wherein Determining a feature vector (31) for each instance pair i and j from the two-dimensional image sensor data (3) by a third machine learning system (30) comprises the following steps: - determining a set of m image sensor data comprising instance pairs i and j from the two-dimensional image sensor data (3), - determining a bounding box for instance i and instance j in each of the m image sensor data comprising the instance pair i and j, - cropping each of the m image sensor data having the bounding boxes of instance i and instance j at n different scales to obtain n different cropped image sensor data for each of the m image sensor data, wherein each cropped image sensor data includes the bounding boxes of instance i and instance j, - from each of n different cropped image sensor data, determine the feature vector for instance pair i and j, - Obtaining a feature vector (31) for instance pair i and j by first averaging n feature vectors from n different cropped image sensor data for each of the m image sensor data to obtain m averaged feature vectors (31a) for instance pair i and j, and then averaging the m obtained feature vectors for instance pair i and j.

8. The method according to any one of the preceding claims, in, After adjusting the parameters of the first machine learning system relative to the training objective, re-determining the refined node feature vector (12) and the refined edge feature vector (13) of the refined graph structure (11), wherein a candidate instance list (4) is provided, wherein each element in the candidate instance list (4) is a word or text describing a possible instance in the three-dimensional environment scene (1), and wherein a fourth machine learning system (40), a third preprocessing network (60), and a fifth machine learning system (50) are provided, wherein the fourth machine learning system (40) and the first machine learning system (50) map their respective input data to the same embedding space, The method further comprises the following steps: - determining, by a fourth machine learning system (40), an embedding for each element of the list of candidate instances (4), - determining a graph structure (11A) having labeled nodes based on the refined graph structure (11) by assigning, for each refined node feature vector (12) having a corresponding node in the refined graph structure (11), an element of the candidate list (4) as a label to the corresponding node of the refined node feature vector (12) based on the highest similarity between the embeddings of the refined node feature vector (12) and the elements of the candidate list (4), - determining input tokens by a third pre-processing network based on the refined edge feature vectors (13), a predefined query (6a) and relation hints (6b), wherein the relation hints (6b) include labels of nodes connected by corresponding edges in a graph structure (11A) with labeled nodes, - determining, by a fifth machine learning system (50), a textual description for each refined edge feature vector (13) based on the determined input tokens, the predefined query (6a) and the relationship hint (6b), - determining a scene graph (11B) from the graph structure (11A) with labeled nodes by assigning the determined textual description of each refined edge feature vector (13) to a corresponding edge of the graph structure (11A) with labeled nodes.

9. The method according to claim 8, further comprising the steps of: - verifying the scene graph (11B) by a user and / or controlling a robot based on the scene graph.

10. A system configured to perform the method according to any one of claims 1 to 9.

11. A computer program configured to cause a computer to perform the method according to any one of claims 1 to 9 when the computer program is executed by a processor.

12. A machine-readable storage medium having stored thereon the computer program according to claim 11.

Images