Device and method for determining relationships between objects in a 3D scene of the real world

Abstract

Computer-implemented method for training a machine learning system (60) for 3D scene graph prediction, wherein the machine learning system (60) comprises: • a neuronal radiation field module (61); • an object detection module with an open vocabulary (62); • a relationship prediction (63); wherein the procedure includes the following: • Providing a training dataset comprising a plurality of two-dimensional images of a 3D scene and positions and orientations of a camera recording the corresponding images, relative to the 3D scene; • Obtaining a desired relationship feature vector (r̂ i ), which characterizes a relationship between two objects in an image of the training dataset; • Training the machine learning system (60) by training the NeRF module (61), the open vocabulary object detection module (62) and the relationship prediction module (63) using the training dataset and the desired relationship embeddings (r̂), wherein: o the NeRF module (61) is trained by minimizing a loss function that compares rendered 2D images produced by the NeRF module with the provided 2D images; o the object detection module is trained with an open vocabulary (62) to detect object feature vectors (s i , i i ) predicting the object information derived from the 2D images (ŝ i , î i ) correlate; and o the relationship prediction module is trained to predict relationship feature vectors (r i ) predicting which are compatible with the desired relationship feature vectors (r̂ i ) correlate.

Description

Technical field: State of the art

[0001] Engelmann et al. “OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-wise Features and Rendered Novel Views”, 2024, https: / / arxiv.org / pdf / 2404.03650 reveals a machine learning model that operates on set images and encodes VLM features within a NeRF.

[0002] Kim et al. “GARField: Group anything with radiance fields”, 2024, https: / / arxiv.org / abs / 2401.09419, reveals an approach to decomposing 3D scenes into a hierarchy of semantically meaningful groups of placed input images. Advantage of the invention

[0003] Understanding the relationships between objects in a 3D scene is crucial for various applications, including robotics, augmented reality, and virtual reality. Conventional methods for understanding 3D scenes often rely on explicit 3D representations, such as point clouds or meshes, which can be computationally intensive to generate and process. Furthermore, these methods often struggle with open-vocabulary scenarios, where the sets of possible objects and relationships are not known in advance.

[0004] Recent advances in neural radiance fields (NeRFs) have enabled implicit 3D scene representations that can be learned directly from 2D images of those scenes. NeRFs excel at synthesizing novel views and 3D reconstruction, but typically focus on representing geometry and appearance, with limited capability for semantic understanding or inferring relationships. Some approaches have extended NeRFs to include object-centric semantic features, enabling open-vocabulary object detection and segmentation. However, these methods typically do not explicitly model relationships between objects. Existing 3D scene graph construction methods usually rely on explicit 3D data and also do not perform well for open-vocabulary objects.

[0005] The present invention addresses these limitations by introducing a novel machine learning system that enables the understanding of relationships. The invention provides a method for training this system, enabling it to learn both object representations and relationships between objects directly from 2D images of a 3D scene of the real world. Specifically, the invention includes a relationship prediction module within the NeRF architecture, which is trained using relationship information preferably extracted from a pre-trained multimodal LLM. This approach allows the system to understand complex, open-vocabulary relationships between objects. Disclosure of the invention

[0006] In a first aspect, the invention relates to a computer-implemented method for training a machine learning system for 3D scene graph prediction, wherein the machine learning system comprises the following: • a neural radiation field module designed to receive, as input, a first 3D point of a scene and a viewing direction within the scene and to output color and density values ​​for the first 3D point; • an open-vocabular object detection module designed to receive, as input, the first 3D point and to output an object feature vector representing semantic information for an object located at the first 3D point; • a relationship prediction module designed to receive, as input, the first 3D point, a second 3D point, and to output a relationship feature vector representing a relationship between an object located at the first 3D point and an object located at the second 3D point; wherein the method comprises: • Providing a training dataset comprising a plurality of two-dimensional images of a 3D scene and positions and orientations of a camera recording the corresponding images, relative to the 3D scene; • Obtaining a desired relationship feature vector that characterizes a relationship between two objects in an image of the training dataset; • Training the machine learning system by training the NeRF module, the open vocabulary object detection module, and the relationship prediction module using the training dataset and relationship embeddings, wherein: The NeRF module is trained by minimizing a loss function that compares rendered 2D images generated by the NeRF module with the provided 2D images; The object detection module is trained with an open vocabulary to predict object feature vectors that correlate with object information derived from the 2D images; and o the relationship prediction module is trained to predict relationship feature vectors that correlate with the desired relationship feature vectors.

[0007] The NeRF module takes a 3D point x and a viewing direction d as input and can output a prediction of the color and density at that point in the 3D scene. This module can be trained using a standard NeRF training approach, minimizing a loss function that compares the rendered 2D images generated by NeRF with the provided 2D images in the training dataset. The 3D point x can determine the position of a point within the scene. The direction d typically refers to the viewing direction; for example, it could correspond to a ray from a camera image pixel position to the 3D scene.

[0008] The open-vocabulary object detection module can be understood as responsible for identifying objects within the 3D scene. Specifically, it takes the same input as the NeRF module and can output an object feature vector representing semantic information for the object located at that point. This module is particularly capable of open-vocabulary object detection, meaning it can identify objects that were not visible during training. This module can be trained to predict object feature vectors that correlate with object information derived from the 2D images in the training dataset. This object information can be obtained using pre-trained vision language models and / or instance segmentation models.Using an "open vocabulary" can allow the model to process any concept without special training on specific objects.

[0009] The relationship prediction module can be considered the key innovation of the present invention. It is specifically responsible for predicting relationships between objects in the 3D scene. It takes as input a first 3D point x and a second 3D point z. These two 3D points can represent the objects whose relationship is being queried. The output of this module is a relationship feature vector r, which can encode the relationship between the two objects. This module can be trained to predict relationship feature vectors that correlate with the desired relationship feature vectors obtained using a multimodal large language model.

[0010] The three modules can each be implemented as a neural network, in particular as a multi-layered perceptron or any other form of a fully connected neural network.

[0011] In preferred embodiments, obtaining the desired relationship feature vectors comprises the following: • Providing the image and a textual prompt to a pre-trained multimodal large language model, where the textual prompt queries the multimodal large language model about a relationship between two objects depicted in the image; • Receiving a textual response from the multimodal large language model, where the textual response describes a relationship between the two objects; • Providing an embedding of the textual response as the desired relationship feature vector.

[0012] The method of claim 2, wherein providing the image and textual prompt to the pre-trained multimodal large language model comprises applying a set-of-mark (SoM) prompt technique in which visual markers are placed over the image to identify the two objects before querying the multimodal LLM about the relationship between the two objects.

[0013] In the SoM prompting technique, visual markers can be overlaid on the image to identify the two objects before the multimodal LLM is queried about their relationship. This technique can improve the accuracy and specificity of the LLM's response by clearly indicating which objects are being referenced in the query. The markers can take various forms, such as bounding boxes, segmentation masks, or simple alphanumeric labels. The specific type of marker used may depend on the application and the capabilities of the LLM. This prompting technique is used in Fig. 2(e) [or a relevant figure from the paper]. The marker-setting prompt technique can allow the query of the multimodal large language model to focus on specific relationships in the image. For example, if the image shows a table with a cup on top of it, a generic relationship query might provide an answer such as "table with cup". However, using the marker-setting prompt allows queries for specific relationships, such as "the cup on the table". This preferably reduces ambiguity and improves relationship prediction.

[0014] Advantageously, these embodiments enable the automatic extraction of relationships between objects in the image without relying on manual human labeling. Even more advantageously, these embodiments of the invention improve the accuracy and specificity of the relationship extraction process with regard to the application of a labeling prompt.

[0015] In preferred embodiments, the object detection module is trained with an open vocabulary to output a tuple comprising a semantic feature vector and an instance embedding for a given 3D point, where s represents a semantic feature vector derived from a pre-trained vision language model for the 3D point, and i represents an instance embedding vector for grouping rays associated with the same object instance as the 3D point.

[0016] In other words, the open-vocabulary object detection module can be designed to output a tuple (s, i) for a given 3D point x. The first element of the tuple, s, can represent a semantic feature vector derived from a pre-trained vision-language model. This vector encodes semantic information about the object located at the 3D point x. The second element of the tuple, i, represents an instance embedding vector. This vector is used to group rays associated with the same object instance as the 3D point x. The semantic information can be extracted by providing an image, as well as a textual query such as "which objects are in the image," to a vision-language model.This information can then be used as training data for the open-vocabulary object detection module, for example, by minimizing the cosine difference between predicted labels and labels extracted from the vision language model. The instance embedding vector preferably provides a method for grouping pixels in the image based on their instance class. For example, if the image shows two chairs, it may be preferable to assign individual instance classes to each chair. This instance embedding can be trained using known instance segmentation algorithms or derived from pre-trained instance segmentation modules.Training the instance embedding by minimizing, for example, a cosine difference between the predicted embedding vector and an instance embedding vector from a pre-trained model can further improve the accuracy of instance segmentation. By jointly outputting the semantic feature vector and the instance embedding vector, the present invention enables object detection in open-vocabulary configurations, while simultaneously being able to segment the detected objects according to their instance class.

[0017] Advantageously, these embodiments of the invention, by using a tuple (s, i) as the output of the open-vocabulary object detection module, enable a more comprehensive and nuanced understanding of objects within the 3D scene. The semantic feature vector s provides rich semantic information derived from a powerful vision language model, allowing the system to recognize and categorize objects in an open-vocabulary configuration. Furthermore, the instance embedding vector i enables accurate instance segmentation by grouping rays associated with the same object, even when multiple instances of the same object class are present in the scene. The combination of both semantic and instance information ensures that detected objects in the image not only correspond to the object class but are also grouped according to their instance class.

[0018] In preferred embodiments, the textual prompt instructs the multimodal LLM to identify objects in the 2D image and to describe relationships between identified objects using at least one relationship category selected from the group consisting of: spatial relationships, comparative relationships, functional relationships, and support relationships.

[0019] The prompt instructs the LLM to perform two main tasks: identifying objects in the image and describing the relationships between these identified objects. Critically, the prompt leads the LLM to categorize the relationships using predefined categories, encompassing spatial relationships (e.g., "above," "below," "next to"), comparative relationships (e.g., "larger than," "smaller than," "exactly the same size as"), functional relationships (e.g., "part of," "used for"), and support relationships (e.g., "standing on," "attached to").

[0020] Advantageously, these embodiments of the invention, which define the structure of the textual prompt provided to the LLM, enable the acquisition of more informative and relevant relationship features. By specifically instructing the LLM to identify objects and describe their relationships using predefined categories, the generated textual responses are more structured and easier to process. This structure can improve the quality of the extracted relationship features because it leads the LLM to focus on more relevant aspects of the scene. Furthermore, using these predefined relationship categories ensures that the extracted features capture a wide range of possible relationships, resulting in a more comprehensive understanding of the 3D scene.This structured approach can be contrasted with methods that use generic prompts or no prompts at all, resulting in less informative and more ambiguous responses from the LLM. Therefore, these implementations improve the overall effectiveness of the training process and the resulting performance of the machine learning system.

[0021] In another aspect, the invention relates to a computer-implemented method for determining a second object in a 3D scene with respect to a first object in the 3D scene by a specific desired relationship using the machine learning system according to one of claims 1 to 5, wherein the method comprises the following steps: • Receiving a relationship query, including a textual description of the specific desired relationship; • Obtaining a 3D point corresponding to the first object; • For each point of a plurality of other 3D points (z) in the 3D scene: o For a plurality of rays: o Generating a relationship feature vector (r) using the relationship prediction module, wherein the relationship query includes a 3D point (x) of the ray, the 3D point (z) and a viewing direction (d) of the ray; o Calculating a rating based on the cosine similarity between an embedding of the textual description of a relationship and the relationship feature vector (r); o Aggregating the ratings for the majority of rays to obtain an aggregate rating for the 3D point (z); • Determining the second object by providing an object located at one of the points as the second object.

[0022] This method enables efficient and accurate relationship queries within the 3D scene. The query includes a textual description of the specific desired relationship, such as "standing on," "next to," "part of," or any other relationship of interest. The method then receives a 3D point x corresponding to the first object. This 3D point can be provided as input by a user, or it can be determined automatically using object detection techniques, preferably the object detection module of the machine learning system. After the 3D point 'x' has been determined for the first object, the method proceeds by evaluating the relationship query for different candidate objects. This can be done by iterating over a plurality of other 3D points z in the 3D scene. For each of these candidate points z, multiple rays are evaluated. This multi-view approach improves the robustness of the relationship prediction.For each pair of points x and z, a relationship feature vector r is generated by querying the relationship prediction module of the machine learning system. Each relationship vector 'r' obtained by the relationship prediction module can be evaluated by calculating a score based on the cosine similarity between an embedding of the textual description of the relationship—obtained, for example, using natural language processing techniques well-established in the field—and the relationship feature vector r. Cosine similarity can be understood as a metric that evaluates the distance between the generated relationship vector and the embedding vector of the provided textual description. Higher cosine similarity indicates a closer semantic match between the predicted relationship and the desired relationship.For each point z, the resulting ratings obtained for the majority of rays can be aggregated to obtain an aggregate rating. The aggregation procedure is not particularly restrictive and can be understood as any method for reducing the dimensionality of multiple rating values ​​into a single value. Advantageously, aggregating ratings based on rays improves the robustness and reliability of the relationship query. Finally, after obtaining an aggregate rating for each of the candidate points 'z', the second object can be determined by selecting an object located at one of the points that has an aggregate rating above a given threshold. This threshold can be chosen empirically based on the desired degree of precision and retrieval for the relationship query.An example of the relationship query procedure can be understood by considering a scenario where the first object is a "table" and the relationship query is "cup on the table". The procedure then determines a 3D point 'x' corresponding to the object "table". Using the relationship prediction module, several "candidate" objects, represented by points 'z' in the scene, can be evaluated, and a score can be calculated for each of the candidate points. The second object can then be determined to be a "cup" located near the table, preferably on top of the table.

[0023] Advantageously, the computer-implemented object relationship query method according to the present invention provides an efficient and flexible way to query relationships between objects in a 3D scene. The use of a textual description enables relationship queries with an open vocabulary, meaning that the system can handle relationships that are not explicitly visible during training. The multi-view approach, using multiple rays for each candidate object, improves the robustness of relationship prediction. Furthermore, the iterative evaluation and aggregation process ensures that the identified second object is highly likely to be related to the first object in the desired way. This approach enables a wide range of applications, from object search and retrieval to scene understanding and robotic manipulation.

[0024] In preferred embodiments, the majority of other 3D points is limited to 3D points for which the object feature vector obtained from the open vocabulary object detection module indicates the presence of an object.

[0025] Specifically, the 3D points can be limited to those points for which the object feature vector, obtained from the open-vocabulary object detection module, indicates the presence of an object. This object feature vector, as described in previous claims, provides information about the semantic class and instance of an object located at a given 3D point. By limiting the candidate points to those where an object is detected, the search space for the second object is significantly reduced. This advantageously leads to higher computational efficiency, especially in scenes with many points that do not correspond to any objects of interest in the 3D scene.

[0026] Embodiments of the invention are discussed in more detail with reference to the following figures. The figures show: Fig. 1 a machine learning system; Fig. 2. A training system for training the machine learning system. Description of the embodiments

[0027] Fig. Figure 1 schematically shows a machine learning system (60) comprising a neural radiation field module (61), also referred to as the NeRF module (61), an open-vocabulary object detection module (62), and a relationship prediction module (63). The NeRF module (61) can be configured to receive a 3D point (x) and a viewing direction (d) as input and can output color (c) and density values ​​(σ). The open-vocabulary object detection module (62) can be configured to receive the 3D point (x) as input and can output a semantic feature vector (s) and an instance embedding vector (i). The relationship prediction module (63) can be configured to receive the 3D point (x) and a second 3D point (z) as input and can output a relationship feature vector (r).The second 3D point (z) can represent a query location, and the relationship feature vector (r) can represent a relationship between an object located at 3D point (x) and a second object located at 3D point (z). The output of the open-vocabulary object detection module (62), which is a tuple comprising a semantic feature vector (s) and an instance embedding vector (i), can enable the identification of objects within the image according to an object class as well as an object instance. The output of the relationship prediction module (63) can be a relationship embedding vector (r) that can characterize a relationship between two objects based on the positions of the two objects (x,z).

[0028] Fig. Figure 2 shows an embodiment of a training system (140) for training the in Fig.1 machine learning system (60) described. The training system (140) uses a training dataset (T) to train the machine learning system (60) for 3D scene graph prediction. The training dataset (T) comprises a plurality of 2D images of a 3D scene and the corresponding positions and orientations of the camera that captured each corresponding image.

[0029] For training purposes, a training data unit (150) can access a computer-implemented database (St2) that can provide the training data set (T). The training data unit (150) can select at least one image and the corresponding camera position and orientation from the training data set (T), preferably randomly. Furthermore, the training data unit (150) can determine suitable training tuples for the image, comprising a 3D point (x). i ), a direction of view (d i ) and a second 3D point (z i ). The 3D point (xi ) and the viewing direction (d) can, for example, be obtained by randomly sampling a pixel from the 2D image and calculating the corresponding ray through the 3D scene based on the camera parameters. The training data unit (150) can also contain target relationship features (r̂ i ) for the selected tuple based on the image of a textual prompt, for example using a pre-trained large language multimodal model (LLM), as described previously. In addition, the training unit (150) can provide target values ​​for the other outputs of the machine learning system: target color values ​​(ĉ i ), target density values ​​(σ̂ i ), semantic feature target vectors (ŝ i ), and instance embedding target vectors (î i These target values ​​can be derived from ground-truth data, pre-trained models, or other suitable sources.

[0030] To extract pixel-aligned target relationship features (r̂ i It is preferred to use marker-setting prompts. To extract textual relationships, it is preferred to use a multimodal GPT-4 model or a multimodal Llama model. Preferably, a combination of visual and textual prompts can be used. For visual prompts, semi-transparent masks and numeric markers can be overlaid on the image. The numeric markers are preferably extracted from a SAM model. The textual prompt can then include a two-step approach that queries the model to first extract objects by their marker identifier and then extract relationships referenced by the previously extracted object identifiers, along with a relationship label. In particular, the prompt can be written as follows: --- Request ---

[0031] 1. Object identification: Identifying all objects in the image by their identifier. Creating a dictionary that maps the identifier to the class name.

[0032] 2. Affordance / Relationship Detection: For each pair of tagged objects that are clearly related, describe the semantic relationships and affordances as a list of dictionaries using the format [s id: #n1, subject class: x, o id: #n2, object class: y, predicates: [p1, p2, ...]]. For Subjects and objects that share multiple relationships / affordances: Concatenate predicates with a comma in the [predicate] field. Avoid generic expressions like "next to" for ambiguous relationships. Instead, specify relationships with precise relationships and affordances that describe spatial relationships [above / below, etc.], comparative relationships [greater than / less than, similar / same type / color], functional relationships [part of / belonging to, turns on], and support relationships [standing on, hanging on, lying on, attached to]. Do not use left / right; always use 3D-consistent relationships. Always use a spatial relationship with a semantic, comparative, functional, or support relationship, separated by a comma (e.g., [A] [above, lying on] [B]). For symmetrical relationships, include both directions (e.g., [A] [above] [B] and [B] [below] [A]).Highlight even distant objects if they have [same / similar / identical color / same object type]. --- Request ---

[0033] The output of the multimodal large language model is textual in nature. To be used in training, the textual output can be embedded in a certain vector space. The inventors found that the choice of encoder determines the structure and queryability of the embedding space in the machine learning system (60). Encoders such as CLIP, BERT, Jina-v3, RoBERTa, and GPT-2 can be used, for example. The inventors found that Jina-v3 embeddings produce the best-structured feature space, with related concepts exhibiting strong similarity, whereas the majority of relationships describing distinct concepts show a high degree of dissimilarity. Preferably, a Jina-v3 model is therefore intended for embedding. The resulting embedding is then called the desired relationship vector (r̂). i) provided. These vectors can further be projected onto the image plane using the SAM segmentation masks and the tagging settings, as a reference to generate a high-dimensional feature representation of the extracted relationships, aligned with the pixel locations of the objects in the image.

[0034] The semantic feature target vectors (ŝ i ) can preferably be determined according to the method of Engelmann et al. The instance embedding target vectors (I g ) can preferably be determined according to the method of Kim et al.

[0035] The training data unit (150) then sends the training tuples (x i , d i , z i ) to the machine learning system (60). The machine learning system (60) then generates corresponding output values ​​(c i , σ i ,s i , i i , r i ) using its respective modules.

[0036] The generated output values ​​(c i , σ i , s i , i i , r i ) and the corresponding target values ​​(ĉi,σ̂ i ,ŝ i ,î i ,r̂ i The data are sent to a modification unit (180). The modification unit (180) then determines new parameters (Φ') for the machine learning system (60) based on the differences between the target values ​​and the generated output values. For this purpose, the modification unit (180) can compare the target values ​​and the generated output values ​​using suitable loss functions for each value. Specifically, the loss function for color and density can be a photometric reconstruction loss, which affects rendered images generated by the NeRF module (61) using c i and σ i are generated with ĉ i and σ̂ i, which are determined from the image, compare. The loss function for the semantic features and the instance embedding can be used for comparisons of s i and i i with the target values ​​ŝ i and i These losses can include cross-entropy or cosine similarity losses. The loss for the generated reference vectors r i can be based on the cosine similarity or another suitable distance metric between r i and the target relationship characteristics (r̂ i The individual losses can then be calculated. These losses can then be combined, for example using a weighted sum, to obtain a combined loss. The modification unit can then determine new parameters (Φ') for the machine learning system based on the combined loss.

[0037] Preferably, the described training can be repeated iteratively for a predefined number of iteration steps or iteratively until the combined loss value falls below a predefined threshold. Alternatively or additionally, it is also conceivable that the training is terminated when an average combined loss value with respect to a test or validation data set falls below a predefined threshold. In at least one of the iterations, the new parameters (Φ') determined in a previous iteration can be used as parameters (Φ) of the machine learning system (60).

[0038] In another embodiment (not shown), the trained machine learning system (60) is queried to find objects with respect to a specific relationship within a 3D scene. The procedure can begin by receiving a relationship query. This query can include a textual description of the desired relationship, such as "on," "next to," "part of," or other relationships. A 3D point (x) representing the first object in the relationship can also be received. The 3D point (x) can be provided, for example, through user interaction, such as when a user clicks on an object in a visualized representation of the 3D scene, or through other object localization methods. The procedure can then proceed to identify candidate objects for the second object in the relationship. The candidate objects can be determined, for example, from the object detection module (62).In particular, only candidate objects identified as such by the object detection module can be considered. This can be achieved, for example, by providing all possible 3D points (z) to the object detection module (62) and considering only those points (z) as candidates for which the object detection module (62) determines a valid object instance. This filtering step can significantly reduce the number of candidate objects and thus lower the computational costs.

[0039] For each candidate object, represented by a 3D point (z), the procedure can then evaluate the relationship between the first object (represented by point x) and the candidate object. This evaluation can include multiple rays, i.e., lines within the 3D scene encompassing a 3D point (x), and a viewing direction (d). For each ray, the relationship prediction module (63) can be queried with the 3D point (x), the 3D point (z), and the viewing direction (d). The relationship prediction module (63) can then provide a relationship feature vector (r) as output. An evaluation can be calculated, for example, based on the cosine similarity, between the relationship feature vector (r) and an embedding of the textual description of the relationship provided in the relationship query. These embeddings can be generated, for example, using natural language processing techniques.The ratings obtained for the different perspectives (d) can then be aggregated for each candidate object (z) to produce an aggregate rating. The aggregation procedure is not particularly restricted and can, for example, be a simple average or a weighted average. The aggregate rating can represent a probability or confidence that the desired relationship exists between the first object at point (x) and the candidate object at point (z).

[0040] Finally, the method can identify the second object by selecting the candidate object with the highest aggregated score, optionally considering only those above a threshold. The object can be identified, for example, by providing the 3D coordinate of point 'z'. In embodiments of the invention, it may be preferred that the relationship prediction module (63) is trained not only with positive examples of relationships but also with negative examples. Negative examples may include pairs of points (x, z) for which objects located at these points are known not to satisfy the relationship, as demonstrated, for example, by manual inspection of the scene.

[0041] The term "computer" can be understood as encompassing any device for processing predefined calculation rules. These calculation rules can be in the form of software, hardware, or a combination of both.

[0042] In general, a plural can be understood as indexed, meaning that each element of the plural is assigned a unique index, preferably by assigning consecutive integers to the elements contained in the plural. Preferably, if a plural comprises N elements, where N is the number of elements in the plural, the elements are assigned the integers from 1 to N. It can also be understood that elements of the plural can be accessed via their index. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Engelman et al. “OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-wise Features and Rendered Novel Views”, 2024, https: / / arxiv.org / pdf / 2404.03650

[0001] Kim et al. “GARField: Group anything with radiance fields”, 2024, https: / / arxiv.org / abs / 2401.09419

[0002]

Claims

[1] Computer-implemented method for training a machine learning system (60) for 3D scene graph prediction, wherein the machine learning system (60) comprises: • a neural radiation field module (61) designed to receive, as input, a first 3D point (x) of a scene and a viewing direction (d) within the scene and to output color and density values ​​for the first 3D point (x); • an open vocabulary object detection module (62) designed to receive, as input, the first 3D point (x) and to output an object feature vector representing semantic information (s) for an object located at the first 3D point; • a relationship prediction module (63) designed to receive, as input, the first 3D point (x), a second 3D point (z), and to output a relationship feature vector (r) representing a relationship between an object located at the first 3D point (x) and an object located at the second 3D point (z); wherein the method comprises: • Providing a training dataset comprising a plurality of two-dimensional images of a 3D scene and positions and orientations of a camera recording the corresponding images, relative to the 3D scene; • Obtaining a desired relationship feature vector (r̂ i ), which characterizes a relationship between two objects in an image of the training dataset; • Training the machine learning system (60) by training the NeRF module (61), the open vocabulary object detection module (62) and the relationship prediction module (63) using the training dataset and the desired relationship embeddings (r̂), wherein: ◯ the NeRF module (61) is trained by minimizing a loss function that compares rendered 2D images produced by the NeRF module with the provided 2D images; ◯ the object detection module is trained with an open vocabulary (62) to detect object feature vectors (s i , i i ) predicting the object information derived from the 2D images (ŝ i , l̂ i ) correlate; and ◯ the relationship prediction module is trained to predict relationship feature vectors (r i ) predicting which are compatible with the desired relationship feature vectors (r̂ i ) correlate. [2] Method according to claim 1, wherein obtaining the desired relationship feature vectors (r̂ i The following is included: • Providing the image and a textual prompt to a pre-trained multimodal large language model, where the textual prompt queries the multimodal large language model about a relationship between two objects depicted in the image; • Receiving a textual response from the multimodal large language model, where the textual response describes a relationship between the two objects; • Providing an embedding of the textual response as the desired relationship feature vector (r̂) i ). [3] Method according to claim 2, wherein providing the image and textual prompt to the pre-trained multimodal large language model comprises applying a set-of-mark (SoM) prompt technique in which visual markers are placed over the image to identify the two objects before querying the multimodal LLM about the relationship between the two objects. [4] Method according to any one of claims 1 to 3, wherein the open vocabulary object detection module (62) is trained to output a tuple comprising a semantic feature vector (s) and an instance embedding vector (i) for a given 3D point (x), wherein s represents a semantic feature vector derived from a pre-trained vision language model for the 3D point (x), and the instance embedding vector (i) represents grouping rays associated with the same object instance as the 3D point (x). [5] Method according to any one of claims 2 to 4, wherein the textual prompt instructs the multimodal LLM to identify objects in the 2D image and to describe relationships between identified objects using at least one relationship category selected from the group consisting of: spatial relationships, comparative relationships, functional relationships and support relationships. [6] Computer-implemented method for determining a second object in a 3D scene with respect to a first object in the 3D scene by a specific desired relationship using the machine learning system (60) according to any one of claims 1 to 5, wherein the method comprises the following steps: • Receiving a relationship query, including a textual description of the specific desired relationship; • Obtaining a 3D point corresponding to the first object; • For each point of a plurality of other 3D points (z) in the 3D scene: o For a plurality of rays: o Generating a relationship feature vector (r) using the relationship prediction module, wherein the relationship query includes a 3D point (x) of the ray, the 3D point (z) and a viewing direction (d) of the ray; o Calculating a rating based on the cosine similarity between an embedding of the textual description of a relationship and the relationship feature vector (r); o Aggregating the ratings for the majority of rays to obtain an aggregate rating for the 3D point (z); • Determining the second object by providing an object located at one of the points as the second object. [7] Method according to claim 6, wherein the plurality of other 3D points (z) are limited to 3D points for which the object feature vector obtained from the open vocabulary object detection module indicates the presence of an object. [8] Training system (140) designed to perform the training procedure according to any one of claims 1 to 5. [9] Computer program designed to cause a computer to execute the method according to any one of claims 1 to 7, including all its steps, when the computer program is executed by a processor (145). [10] Machine-readable storage medium (146) on which the computer program according to claim 9 is stored.

Images