Generating robotic assembly instructions from three-dimensional computer-aided design models

Abstract

To provide a device and a method for generating robotic assembly instructions from three-dimensional (3D) computer-aided design (CAD) models.SOLUTION: A method may include: generating component graphs for components and generating an assembly descriptor based on an assembly graph and the component graphs; processing the assembly descriptor with a graph convolution network model to generate node embeddings; applying pooling to the node embeddings to generate graph embeddings; calculating a cross attention between the components to generate component interrelations; utilizing the graph embeddings and the component interrelations to predict links between the components; predicting poses and joint axes for the components; and generating assembly instructions based on the graph embeddings, the component interrelations, the links, the poses, and the joint axes.SELECTED DRAWING: Figure 5

Description

[Background technology]

[0001] Automating tasks using robotic devices is highly dependent on correctly programming the robotic device. For example, during an assembly process, a robotic arm must be programmed to move into position to properly grasp an item and then move it to the appropriate location. In another example, during an assembly process, a robotic drill must be programmed to move into position to tighten a fastener, such as a screw or bolt. Summary of the Invention [Problem to be solved by the invention]

[0002] Some implementations described herein relate to a method that can include receiving a three-dimensional (3D) computer-aided design (CAD) model and generating an assembly graph having nodes representing components of the 3D CAD model and edges representing contacts between the components. [Means for solving the problem]

[0003] The method can include generating a component graph for each component of the assembly graph, and generating an assembly descriptor based on the assembly graph and the component graph. The method can include processing the assembly descriptor with a graph convolutional network model to generate node embeddings, and applying pooling to the node embeddings to generate a graph embedding. The method can include computing cross-attention between the components based on the graph embeddings to generate component interrelationships, and predicting links between the components using the graph embeddings and the component interrelationships. The method can include predicting poses and joint axes for the components based on the graph embeddings, the component interrelationships, and the links, and generating assembly instructions based on the graph embeddings, the component interrelationships, the links, the poses, and the joint axes. The method can cause a robotic device to execute the assembly instructions to assemble an object represented by the 3D CAD model.

[0004] Some implementations described herein relate to an apparatus. The apparatus may include one or more memories and one or more processors coupled to the one or more memories. The one or more processors may receive a 3D CAD model and generate an assembly graph having nodes representing components of the 3D CAD model and edges representing contacts between the components. The one or more processors may be configured to generate a component graph for each component of the assembly graph and generate an assembly descriptor based on the assembly graph and the component graph, where the assembly descriptor includes the component graph and a corresponding adjacency matrix. The one or more processors may be configured to process the assembly descriptor using a graph convolutional network model to generate a node embedding and apply pooling to the node embedding to generate a graph embedding. The one or more processors may be configured to calculate cross-attention between components based on the graph embedding to generate component interrelationships and predict links between components utilizing the graph embedding and the component interrelationships. The one or more processors may be configured to predict poses and joint axes for the components based on the graph embedding, the component interrelationships, and the links, and generate assembly instructions based on the graph embedding, the component interrelationships, the links, the poses, and the joint axes. The one or more processors are configurable to cause the robotic device to execute the assembly instructions to assemble an object represented in the 3D CAD model.

[0005] Some implementations described herein relate to a non-transitory computer-readable medium storing a set of instructions. The set of instructions, when executed by one or more processors of the device, can cause the device to receive a 3D CAD model and generate an assembly graph having nodes representing components of the 3D CAD model and edges representing contacts between the components. The set of instructions, when executed by one or more processors of the device, can cause the device to generate a component graph for each component of the assembly graph, where each component graph includes one or more nodes representing one or more faces of a corresponding component and one or more edges connecting the one or more nodes representing the one or more faces. The set of instructions, when executed by one or more processors of the device, can cause the device to generate an assembly descriptor based on the assembly graph and the component graph, and process the assembly descriptor with a graph convolutional network model to generate a node embedding. The set of instructions, when executed by one or more processors of the device, can cause the device to apply pooling to the node embedding to generate a graph embedding, and compute cross-attention between components based on the graph embedding to generate component interrelationships. The set of instructions, when executed by one or more processors of the device, can cause the device to predict links between components using the graph embedding and the component interrelationships, and predict poses and joint axes for the components based on the graph embedding, the component interrelationships, and the links. The set of instructions, when executed by one or more processors of the device, can cause the device to generate assembly instructions based on the graph embedding, the component interrelationships, the links, the pose, and the joint axes, and to cause the robotic device to execute the assembly instructions to assemble an object represented by the 3D CAD model. [Brief description of the drawings]

[0006] [Figure 1A] FIG. 2 is a diagram of an example implementation described herein. [Figure 1B] FIG. 2 is a diagram of an example implementation described herein. [Figure 1C] FIG. 2 is a diagram of an example implementation described herein. [Figure 1D] FIG. 2 is a diagram of an example implementation described herein. [Figure 1E] FIG. 2 is a diagram of an example implementation described herein. [Figure 1F] FIG. 2 is a diagram of an example implementation described herein. [Figure 1G] FIG. 2 is a diagram of an example implementation described herein. [Figure 1H] FIG. 2 is a diagram of an example implementation described herein. [Figure 1I] FIG. 2 is a diagram of an example implementation described herein. [Figure 1J] FIG. 2 is a diagram of an example implementation described herein. [Figure 1K] FIG. 2 is a diagram of an example implementation described herein. [Figure 1L] FIG. 2 is a diagram of an example implementation described herein. [Figure 1M] FIG. 2 is a diagram of an example implementation described herein. [Figure 2A] FIG. 1 illustrates an example of training and using a machine learning model. [Figure 2B] FIG. 1 illustrates an example of training and using a machine learning model. [Figure 2C] FIG. 1 illustrates an example of training and using a machine learning model. [Diagram 3] FIG. 1 illustrates an example environment in which the systems and / or methods described herein may be implemented. [Figure 4] FIG. 4 illustrates example components of one or more of the devices of FIG. 3. [Diagram 5] 1 is a flowchart of an exemplary process for generating robot assembly instructions from a three-dimensional (3D) computer-aided design (CAD) model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The following detailed description of implementations refers to the accompanying drawings, in which the same reference numbers in different drawings may indicate the same or similar elements.

[0008] Programming a robotic device to assemble an object is a time-consuming process. For example, a computer may refine the program code for the robotic device through multiple iterations based on user input, consuming power and processing resources each time the program code is adjusted. Additionally, the program code may be debugged over multiple test iterations, thus consuming power and processing resources each time the program code is recompiled and retried.

[0009] Furthermore, automation techniques that generate robotic instructions often consume large amounts of power and processing resources. For example, using an augmented reality (AR) marker when recording a video of an assembly process provides a large amount of data for generating program code for a robotic device. However, using the AR marker to determine the large amount of data consumes a large amount of power, processing resources, and memory space. In another example, using a motion sensor with a specific coordinate marker also provides data for generating program code for a robotic device. However, using the motion sensor to determine the large amount of data also consumes a large amount of power, processing resources, and memory space.

[0010] Furthermore, the generation of robotic instructions typically depends on the profiles of the items used during the assembly process. For example, existing data structures for screws, bolts, and other items used during the assembly process may allow a robotic device to properly grasp and manipulate the items used during the assembly process. However, pre-generating the profiles of the items is a time-consuming process. Furthermore, the generation of the profiles may depend on capturing and processing scans of the items, which also consumes power and processing resources.

[0011] Therefore, current techniques for generating robot assembly instructions consume computational resources (e.g., processing resources, memory resources, communication resources, etc.), network resources, etc. associated with generating and refining program code for the robot assembly instructions over multiple iterations, utilizing large amounts of augmented reality and motion sensor resources to generate the robot assembly instructions, generating profiles of the object to be assembled, etc.

[0012] Some implementations described herein relate to a planning system that generates assembly instructions for a robot from a three-dimensional (3D) computer-aided design (CAD) model. For example, the planning system may receive the 3D CAD model and generate an assembly graph having nodes representing components of the 3D CAD model and edges representing contacts between the components. The planning system may generate a component graph for each component of the assembly graph and generate an assembly descriptor based on the assembly graph and the component graph. The planning system may process the assemble descriptor using a graph convolutional network model to generate node embeddings and apply pooling to the node embeddings to generate a graph embedding. The planning system may calculate cross-attention between components based on the graph embeddings to generate component interrelationships and utilize the graph embeddings and the component interrelationships to predict links between the components. The planning system may predict poses and joint axes for the components based on the graph embeddings, the component interrelationships, and the links, and generate assembly instructions based on the graph embeddings, the component interrelationships, the links, the poses, and the joint axes. The planning system may cause a robotic device to execute the assembly instructions to assemble an object represented by the 3D CAD model.

[0013] Thus, the planning system generates robot assembly instructions from the 3D CAD model. For example, the planning system can provide a data-driven learning model that utilizes 3D CAD models of separate components of an object to provide an efficient and executable assembly plan and pose values ​​of the components. The planning system can obtain component information using the 3D CAD model of the object, and process the component information along with the set of models to generate an assembly plan for the object. This saves computational resources, network resources, etc., that would otherwise be spent on generating and refining program code for the robot assembly instructions in multiple iterations, utilizing extensive augmented reality and motion sensors to generate the robot assembly instructions, generating profiles of the object to be assembled, etc.

[0014] 1A-1M are diagrams of an example 100 relating to generating building instructions for a robot from a 3D CAD model. As shown in FIGS. 1A-1M, the example 100 includes a server device and a robotic device that participate in a planning system. The planning system includes a system for generating building instructions for the robot from the 3D CAD model. Further details of the server device, the robotic device, and the planning system are provided elsewhere herein.

[0015] As shown by reference numeral 105 in FIG. 1A, the planning system can receive a 3D CAD model. For example, a server device can store a 3D CAD model of an object for which assembly instructions are to be generated. The assembly instructions can include instructions that, when executed by a robotic device, can cause the robotic device to assemble the object. The 3D CAD model can include a functional virtual prototype of a 3D object (e.g., chair parts). The 3D CAD model allows a designer to dynamically generate and correct every detail of the object. In some embodiments, the server device can provide the 3D CAD model to the planning system and the planning system can receive the 3D CAD model from the server device. The planning system can store the 3D CAD model in a data structure (e.g., a database, a table, a list, etc.) associated with the planning system.

[0016] As further shown in FIG. 1A by reference numeral 110, the planning system can generate an assembly graph having nodes representing components of the 3D CAD model and edges representing contacts between the components. For example, the planning system can utilize the 3D CAD model to determine information identifying components of an object (e.g., parts of a chair such as the backrest, side legs, seat, rear connector, front connector, etc.). The planning system can utilize the information identifying the components to generate an assembly graph. The assembly graph can include nodes representing components of the object (e.g., the 3D model) and edges representing contacts (e.g., connections) between the components. For example, the backrest of the chair can be represented by node A, the first side leg by node B, the front connector by node C, the second side leg by node D, the seat by node E, the rear connector by node F, etc. In some implementations, the nodes of the assembly graph may not initially include discrete features. The edges of the assembly graph between any two nodes can be classified into a non-edge type when there is no contact relationship between the components represented by the nodes, a hard edge type when there is direct contact between the components represented by the nodes, etc.

[0017] As shown by reference numeral 115 in FIG. 1B, the planning system can generate a component graph for each component (e.g., node) of the assembly graph. For example, a component (e.g., node) of the assembly graph can include one or more faces and one or more connections between the one or more faces. For each component, the planning system can generate a component graph including nodes representing one or more faces of the component and edges representing one or more connections between the one or more faces. For example, the front connector of a chair can include six faces (e.g., front, back, top, bottom, left side, right side) and eight connections between the six faces. In such an example, the planning system can generate a component graph (e.g., for the front connector of the chair) including six nodes and eight edges (e.g., links) provided between the nodes. Thus, each node of the assembly graph can be represented by a component graph including the nodes and edges.

[0018] In some implementations, each node (e.g., face) in the component graph can include a set of features, such as spatial features, structural features, etc. Spatial features can include the spatial position of the vertices (P). Structural features can include the vertex normals (N v ), Gaussian curvature (gc), surface normal (N f ), the angle (θ) between the surface and a surface neighboring it, etc. The Gaussian curvature is determined as follows: JPEG2025024671000002.jpg2067 Here, the angle θ i is in the angle set and is the angle between the normals (n) of the two faces. JPEG2025024671000003.jpg2237 Here, the normal set (N v ) is mathematically defined as follows: JPEG2025024671000004.jpg2052

[0019] In FIG. 1C , as indicated by reference numeral 120, the planning system can generate an assembly descriptor based on an assembly graph and a component graph. For example, the planning system can generate an assembly descriptor by combining the component graphs based on the assembly graph (e.g., nodes of the assembly graph correspond to the component graphs). In some implementations, the assembly descriptor includes the component graphs and corresponding adjacency matrices. Each component graph is represented by a corresponding adjacency matrix. The adjacency matrix (e.g., connectivity matrix) of a component graph (e.g., a labeled graph) is calculated by dividing the vertices v of the graph by the number of nodes v. i and v j matrices with rows and columns labeled with v i and v j The position (v i ,v j ) will be 1 or 0.

[0020] As shown in FIG. 1D by reference numeral 125, the planning system processes the assembly descriptor with a graph convolutional network model to generate node embeddings. For example, the planning system can be associated with a graph convolutional network model having multiple graph convolutional layers. The graph convolutional network model can generate updated node features (e.g., node embeddings) of the input graph (e.g., assembly descriptor). The graph convolutional network model can include a first portion having a first convolutional layer (e.g., an S layer) and a second portion having a second convolutional layer (e.g., a T layer) different from the first convolutional layer. In some implementations, the assembly descriptor includes a component graph, and when processing the assembly descriptor with the graph convolutional network model to generate node embeddings, the planning system can encode features of the component graphs in the assembly descriptor to generate the node embeddings. Further details of the graph convolutional network model are described below in conjunction with FIGS. 1E-1G and 2.

[0021] The top part of Figure 1E shows a multi-layer graph convolutional network. For example, a two-layer graph convolutional network can include an input layer and an output layer. The input layer can include an amount of input channels (C) and the output layer can include an amount of feature maps (F). The input to a two-layer graph convolutional network can include a graph represented as G(X,A), where X is the feature embedding of the nodes in the graph and A is the adjacency matrix. The bottom left of Figure 1E shows the propagation of a message (m) to a node (v3) at step t, where N(v) is the set of adjacent nodes of v, and h denotes the hidden state of the node. JPEG2025024671000005.jpg2278 The bottom right of Fig. 1E shows messages transmitted from nodes v1 and v2 to node v3, and the resulting adjacency matrix m ν t+1 (e.g. 15,15,-10).

[0022] The top part of Figure 1F shows the final adjacency matrix h ν t+1 The bottom of Figure 1F shows that all neighbors i of node 1 (e.g., h1) obtain an attention coefficient α ij k We show how to transmit features with the following structure. The features of head k from each neighbor are aggregated and sent to node 1 (e.g., h i ) features of the next level: JPEG2025024671000006.jpg2879

[0023] FIG. 1G illustrates a message passing network for a two-layer graph convolutional network. A graph denoted as G(X,A) may be provided to the message passing network, where X is the feature embedding of the nodes of the graph and A is the adjacency matrix. As shown, features of nodes A and C may be passed to node B in the first layer, features of nodes A, B, E, and F may be passed to node C in the first layer, features of node A may be passed to node D in the first layer, and features of nodes B, C, and D may be passed to node A in the second layer.

[0024] As further shown in FIG. 1H at 130, the planning system can apply pooling to the node embeddings to generate the graph embeddings. For example, the planning system can utilize a first convolutional layer (e.g., S-layer) of the graph convolutional network model to encode features of the component graphs into the component graph embeddings (e.g., node embeddings). The node embeddings are available as features of the assembly graph nodes. The planning system can apply a type of pooling to the node embeddings to generate the graph embeddings. In some implementations, when applying pooling to the node embeddings to generate the graph embeddings, the planning system can apply average pooling to the node embeddings to generate the graph embeddings, max pooling to the node embeddings to generate the graph embeddings, min pooling to the node embeddings to generate the graph embeddings, etc.

[0025] In some implementations, each node (e.g., component) of the assembly graph can include a feature vector that represents the component. The planning system can propagate the assembly graph through a second convolutional layer (e.g., T-layer) of the graph convolutional network model to obtain an updated set of node features. In some implementations, the planning system can utilize a recurrent neural network model (e.g., a long short-term memory (LSTM) model, a gated recurrent unit (GRU) model, etc.) to aggregate messages in the message-passing network. The recurrent neural network model can receive the hidden state of the aggregated message from the previous step and the input graph to generate an output graph with new hidden states of the aggregated message and updated node embeddings.

[0026] As shown in FIG. 1I at 135, the planning system can compute cross-attention between components based on graph embedding to generate component interrelationships. For example, the planning system can capture the relationships between components (e.g., component interrelationships) by computing cross-attention between components based on graph embedding. Given feature vectors of components, the planning system can focus on the most informative descriptors to generate a single feature vector that captures the interrelationships between the parts. For example, a feature vector (F a ), the planning system can learn a self-attention feature map using scaled dot product attention to convert the features of each part into a self-attention vector. Thus, F a can be transformed into three feature subspaces Q, K, and V as follows: Q=W q F a K=W k F a V=W v F a Here, W q , W k, W v is the learning weight matrix of dimension d*d (e.g., d=1024), and the self-attention weights are calculated as follows: JPEG2025024671000007.jpg2145 The planning system converts the self-attention feature vector into a matrix S a can be summarized as: S a =W a V The planning system is the matrix S a To convert x into a weight (e.g., probability) vector, a row-wise softmax can be applied. The planning system can compute the element-wise dot product between the component features and the self-attention vector as follows: JPEG2025024671000008.jpg938 The planning system sums the weighted features to produce the output feature vector f a can be generated. JPEG2025024671000009.jpg2241

[0027] As shown in FIG. 1J by reference numeral 140, the planning system can use the graph embedding and component interrelationships to predict links between components. For example, if the graph embedding and component interrelationships are used to predict links between components, the planning system can process the graph embedding and component interrelationships with a link prediction model or a link classification model to predict links (e.g., contacts) between components. The planning system can model the link (e.g., contact) generation problem as a link prediction problem or a link classification problem in the graph. The planning system can predict links using pairs of node embeddings by applying some prediction head (ŷ) such as: JPEG2025024671000010.jpg1161 Here, the prediction head is a concatenated and multi-layer perceptron (MLP) classifier or a dot product (e.g., y = (hv ) T h u ) can be used to determine

[0028] As shown in FIG. 1K by reference numeral 145, the planning system can predict poses and joint axes for components based on graph embedding, component interrelationships, and links. For example, the planning system can regress translation, rotation, and joint axis predictions using a combination of Euclidean norm (e.g., L2) and chamfer distance loss terms. The planning system can utilize MLP to predict the 6-DOF poses of each component. In some implementations, the planning system can represent the joint axes as unit vectors in 3D space (e.g., three standard unit vectors i, j, k along the x-, y-, and z-axes, respectively). The planning system can represent any direction in the form of a scalar multiple of the standard unit vector, and can normalize any direction (a) to unit magnitude as follows: JPEG2025024671000011.jpg10128 The planning system is capable of aligning all 3D assemblies (e.g. objects) along the main axes (e.g. directions) of the 3D space.

[0029] As shown in FIG. 1L by reference numeral 150, the planning system can generate assembly instructions based on the graph embedding, component interrelationships, links, poses, and joint axes. For example, the planning system can generate assembly instructions for a robotic device (e.g., including one or more robotic machines) for each of a plurality of operations based on the graph embedding, component interrelationships, links, poses, and joint axes. For example, the planning system can generate assembly instructions for the robotic device to grasp each component of an object associated with an operation represented in the assembly instructions and manipulate the component according to a coordinate set associated with the component and operation. By iteratively generating assembly instructions according to the operation sequence and the components of the object, the planning system generates assembly instructions for the robotic device to assemble the object from the components according to the 3D CAD model.

[0030] In some implementations, the planning system can apply rapid search random trees to assembly instructions (e.g., robot motion plans), where the motions represented by the assembly instructions are associated with state transitions (e.g., multiple state transitions). For example, the motions can represent state transitions from an initial state (before the motion) to a final state (after the motion) of a component associated with the motion. Thus, a sequence of motions represented by the assembly instructions can correspond to a sequence of state transitions. The planning system can start from an initial state associated with a first motion represented by the assembly instructions and iteratively attempt to reduce the distance from the initial state to a final state associated with an ultimate motion represented by the assembly instructions. The planning system thus applies rapid search random trees to iteratively find the shortest (at least locally) path of the assembly instructions through the motion sequence. The planning system can thus generate machine-level instructions corresponding to the state transitions based on the graph embedding, component interrelationships, links, poses, and joint axes.

[0031] As shown in FIG. 1M by reference numeral 155, the planning system can cause the assembly instructions to be executed by the robotic device to assemble the object represented by the 3D CAD model. For example, the planning system can provide the assembly instructions to the robotic device and command the robotic device to execute the assembly instructions. The robotic device can execute the assembly instructions and assemble the object represented by the 3D CAD model based on the execution of the assembly instructions. For example, the planning system can send the instructions as commands to the robotic device. Thus, the robotic device can execute the assembly steps corresponding to the 3D CAD model. In some implementations, the planning system can send the assembly instructions to program the robotic device. Thus, the robotic device can execute the assembly steps repeatedly (e.g., in an environmental setting such as a factory or another assembly line) according to the assembly instructions. For example, an operator of the robotic device can trigger the robotic device to repeatedly execute the assembly steps according to the assembly instructions. In some implementations, the robotic device can execute the assembly steps each time it receives a trigger (e.g., from an operator) or can repeat the assembly steps according to a schedule. The schedule may be preconfigured (e.g., with a time interval between each execution of an assembly step) or may be sensor-based (e.g., each execution of an assembly step is triggered by the movement of the assembly line or another type of environmental trigger).

[0032] Thus, the planning system generates robot assembly instructions from the 3D CAD model. For example, the planning system can provide a data-driven learning model that utilizes 3D CAD models of separate components of an object to provide an efficient and executable assembly plan and pose values ​​of the components. The planning system can obtain component information using the 3D CAD model of the object, and process the component information along with the set of models to generate an assembly plan for the object. This saves computational resources, network resources, etc., that would otherwise be spent on generating and refining program code for the robot assembly instructions in multiple iterations, utilizing extensive augmented reality and motion sensors to generate the robot assembly instructions, generating profiles of the object to be assembled, etc.

[0033] As noted above, FIGS. 1A-1M are provided as examples. Other examples may differ from those described with respect to FIGS. 1A-1M. The number and configuration of devices shown in FIGS. 1A-1M are provided as examples. In practice, there may be additional devices, fewer devices, different devices, or different configurations of devices than those shown in FIGS. 1A-1M. Furthermore, two or more devices shown in FIGS. 1A-1M may be implemented within a single device, or a single device shown in FIGS. 1A-1M may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) shown in FIGS. 1A-1M may perform one or more functions that are described as being performed by another set of devices shown in FIGS. 1A-1M.

[0034] 2A illustrates an example 200 of training and using a machine learning model to generate assembly instructions for a robot from a 3D CAD model. The training and use of the machine learning models described herein may be performed using a machine learning system that may include or be included in a computing device, server, cloud computing environment, etc., such as a planning system described in more detail elsewhere herein.

[0035] As indicated by reference numeral 205, the machine learning model can be trained using a set of observations. The set of observations can be obtained from historical data, such as data collected during one or more processes described herein. In some implementations, the machine learning system can receive the set of observations (e.g., as input) from a planning system, such as those described elsewhere herein.

[0036] As indicated by reference numeral 210, the set of observations includes a feature set. The feature set may include a set of variables, which may be referred to as features. A particular observation may include a set of variable values ​​(or feature values) corresponding to the set of variables. In some implementations, the machine learning system may determine the variables for the set of observations and / or the variable values ​​for the particular observation based on input received from the planning system. For example, the machine learning system may identify the feature set (e.g., one or more features and / or feature values) by extracting the feature set from structured data, performing natural language processing to extract the feature set from unstructured data, receiving input from an operator, etc.

[0037] As an example, a feature set for a set of observations may include a first feature of an assembly graph, a second feature of a component graph, a third feature of an assembly descriptor, etc. As shown, for a first observation, the first feature may have a value of assembly graph 1, the second feature may have a value of component graph 1, the third feature may have a value of assembly descriptor 1, etc. These features and feature values ​​are provided as examples and may differ in other examples.

[0038] As indicated by reference numeral 215, a set of observations may be associated with a target variable. The target variable may represent a variable having a numerical value, a numerical variable that may fall within a range of values, a number of discrete values, a variable that may be selected from one of a number of options (e.g., one of a number of classes, classifications, labels, etc.), a variable having a Boolean value, etc. The target variable may be associated with a target variable value, which may be specific to the observation. In example 200, the target variable has a value of a node embedding for the observation.

[0039] The target variable may represent values ​​that the machine learning model is being trained to predict, and the feature set may represent variables that are input to the trained machine learning model to predict values ​​for the target variable. The set of observations may include target variable values ​​such that the machine learning model can be trained to recognize patterns in the feature set that lead to the target variable values. Machine learning models that are trained to predict target variable values ​​are sometimes referred to as supervised learning models.

[0040] In some implementations, the machine learning model may be trained on a set of observations that does not include a target value. This may be referred to as an unsupervised learning model. In this case, the machine learning model can learn patterns from the set of observations without labeling or supervision and can provide output indicative of such patterns, such as by clustering and / or association to identify related items in the set of observations.

[0041] As indicated by reference numeral 220, the machine learning system can train a machine learning model using the set of observations and one or more machine learning algorithms, such as a regression algorithm, a decision tree algorithm, a neural network algorithm, a k-nearest neighbor algorithm, a support vector machine algorithm, and / or the like. After training, the machine learning system can save the machine learning model as a trained machine learning model for use in analyzing new observations.

[0042] As indicated by reference numeral 230, the machine learning system may receive a new observation and apply the trained machine learning model 225 to the new observation, such as by inputting the new observation into the trained machine learning model 225. As shown, the new observation may include, by way of example, a first feature of assembly graph X, a second feature of component graph Y, a third feature of assembly descriptor Z, etc. The machine learning system may apply the trained machine learning model 225 to the new observation to generate an output (e.g., a result). The type of output may depend on the type of machine learning model and / or the type of machine learning task being performed. For example, the output may include a predicted value of a target variable, such as when supervised learning is employed. Additionally or alternatively, the output may include information identifying a cluster to which the new observation belongs, information indicating a degree of similarity between the new observation and one or more other observations, such as when unsupervised learning is employed.

[0043] As an example, the trained machine learning model 225 can predict a node embedding value A of a target variable of the node embedding for a new observation, as shown at 235. Based on the prediction, the machine learning system can provide a first recommendation, provide an output for determining the first recommendation, perform a first automated action, cause the first automated action to be performed (e.g., by instructing another device to perform the automated action), etc.

[0044] In some implementations, the trained machine learning model 225 can classify (e.g., cluster) new observations into clusters, as indicated by reference numeral 240. Observations within a cluster can have a threshold similarity. As an example, if the machine learning system classifies the new observation into a first cluster (e.g., an assembly graph cluster), the machine learning system can provide a first recommendation. Additionally or alternatively, the machine learning system can perform and / or cause the first automated action to be performed (e.g., by instructing another device to perform the automated action) based on the classification of the new observation into the first cluster.

[0045] As another example, if the machine learning system classifies a new observation into a second cluster (e.g., a part graph cluster), the machine learning system can provide a second (e.g., different) recommendation and / or perform or cause a second (e.g., different) automated action.

[0046] In some implementations, recommended and / or automated actions associated with a new observation may be based on a target variable value having a particular label (e.g., classification, category, etc.), based on whether the target variable value satisfies one or more thresholds (e.g., whether the target variable value is greater than a threshold, less than a threshold, equal to a threshold, within a threshold range, etc.), based on the cluster into which the new observation falls, etc.

[0047] In this manner, machine learning systems can apply a rigorous, automated process to determine target variables (e.g., node embeddings) that can recognize and / or identify tens, hundreds, thousands, or millions of features and / or feature values ​​for tens, hundreds, thousands, or millions of observations, thereby improving the accuracy, consistency, and reducing delays associated with determining target values ​​relative to the computational resources that would be required to have tens, hundreds, or thousands of operators manually determine the target variables.

[0048] 2B and 2C are diagrams illustrating an example 200 of using a deep learning model to generate assembly instructions for a robot from a 3D CAD model. As shown in FIG. 2B, the deep learning model can include a recurrent graph encoder, a cross attention component, and a contact generation component. The recurrent graph encoder generates an assembly graph ( The assembly graph component can receive the current state of the assembly graph (e.g., JPEG2025024671000012.jpg1034, where v is the set of component nodes and ε is the set of edges) and can sequentially generate updated node embeddings (e.g., partial embeddings). The cross attention component can receive the partial embeddings and can compute the relationships between the various partial embeddings (e.g., part inter-relationships) and the cross attention between the partial embeddings. The contact generation component can receive the partial embeddings and the cross attention and can process the partial embeddings and the cross attention. The contact generation component can update the state of the assembly graph by adding new links (e.g., edges) based on the processing of the partial embeddings and the cross attention. The contact generation component can generate the updated assembly graph ( Given partial feature vectors, the deep learning model can generate a single feature vector that captures the interrelationships between the partial feature vectors by focusing on the most informative descriptors.

[0049] As shown in Figure 2C, the deep learning model estimates the current state of the assembly graph ( The deep learning model can receive the state of the assembly graph (at each time period) by adding new links (e.g. edges) and the hidden states of the nodes of the assembly graph (at each time period) and generate an updated assembly graph ( For example, the deep learning model can generate the current state of the assembly graph ( The deep learning model can receive the state of the assembly graph (e.g., in the first time period) by adding a new link (e.g., between node E and node F) and the hidden state (h0) of the nodes of the assembly graph. The deep learning model can update the state of the assembly graph (e.g., in the first time period) by adding a new link (e.g., between node E and node F) to generate an updated assembly graph ( The deep learning model can generate the current state of the assembly graph ( JPEG2025024671000018.jpg1034) and the hidden state of the nodes in the assembly graph at the final period (h t The deep learning model can continue this process until it receives the final assembly graph ( JPEG2025024671000019.jpg1046) (for example, interconnecting nodes A to F) can be generated.

[0050] As noted above, Figures 2A-2C are provided as examples, and other examples may differ from those described with respect to Figures 2A-2C.

[0051] FIG 3 illustrates an example environment 300 in which systems and / or methods described herein may be implemented. As shown in FIG 3, environment 300 may include a planning system 301, which may include and / or be executable within one or more elements of a cloud computing system 302. Cloud computing system 302 may include one or more elements 303-313, as described in more detail below. As further shown in FIG 3, environment 300 may include a network 320, a server device 330, and a robotic device 340. The devices and / or elements of environment 300 may be interconnected by wired and / or wireless connections.

[0052] The cloud computing system 302 includes computing hardware 303, a resource management component 304, a host operating system (OS) 305, and / or one or more virtual computing systems 306. The resource management component 304 can perform virtualization (e.g., abstraction) of the computing hardware 303 to create one or more virtual computing systems 306. Using virtualization, the resource management component 304 can allow a single computing device (e.g., a computer, a server, and / or the like) to operate like multiple computing devices, such as by creating multiple separate virtual computing systems 306 from the computing hardware 303 of the single computing device. In this manner, the computing hardware 303 can operate more efficiently with lower power consumption, higher reliability, higher availability, higher utilization, greater flexibility, and lower cost than using separate computing devices.

[0053] Computing hardware 303 includes hardware and corresponding resources from one or more computing devices. For example, computing hardware 303 can include hardware from a single computing device (e.g., a single server) or from multiple computing devices (e.g., multiple servers), such as multiple computing devices in one or more data centers. As shown, computing hardware 303 can include one or more processors 307, one or more memories 308, one or more storage components 309, and / or one or more network components 310. Examples of processors, memories, storage components, and network components (e.g., communication components) are described elsewhere herein.

[0054] The resource management component 304 includes a virtualization application (e.g., running on hardware, such as the computing hardware 303) that virtualizes the computing hardware 303 and enables starting, stopping, and / or managing one or more virtual computing systems 306. For example, the resource management component 304 can include a hypervisor (e.g., a bare metal or type 1 hypervisor, a hosted or type 2 hypervisor, etc.) or a virtual machine monitor, such as when the virtual computing system 306 is a virtual machine 311. Additionally or alternatively, the resource management component 304 can include a container manager, such as when the virtual computing system 306 is a container 312. In some implementations, the resource management component 304 operates within and / or in conjunction with the host operating system 305.

[0055] Virtual computing system 306 includes a virtual environment that enables cloud-based execution of the operations and / or processes described herein using computing hardware 303. As shown, virtual computing system 306 can include virtual machines 311, containers 312, hybrid environments 313 including virtual machines 311 and containers 312, etc. Virtual computing system 306 can execute one or more applications using a guest operating system (e.g., within virtual computing system 306) or a file system that includes binary files, software libraries, and / or other resources required for application execution on host operating system 305.

[0056] The planning system 301 may include one or more elements 303-313 of a cloud computing system 302, may be executable within, and / or may be hosted within the cloud computing system 302, although in some implementations the planning system 301 may not be cloud-based (e.g., may be implemented outside of a cloud computing system) or may be partially cloud-based. For example, the planning system 301 may include one or more devices that are not part of the cloud computing system 302, such as device 400 of FIG. 4, which may include a standalone server or another type of computing device. The planning system 301 may perform one or more operations and / or processes described in more detail elsewhere herein.

[0057] Network 320 may include one or more wired and / or wireless networks. For example, network 320 may include a cellular network, a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a private network, the Internet, etc., and / or a combination of these or other types of networks. Network 320 may enable communication between devices of environment 300.

[0058] Server device 330 may include one or more devices capable of receiving, generating, storing, processing, providing, and / or routing information, as described elsewhere herein. Server device 330 may include communication devices and / or computing devices. For example, server device 330 may include a server, such as an application server, a client server, a web server, a database server, a host server, a proxy server, a virtual server (e.g., running on computing hardware), or a server in a cloud computing system. In some implementations, server device 330 may include computing hardware used in a cloud computing environment.

[0059] The robotic device 340 may include one or more devices capable of receiving, generating, storing, processing, providing, and / or routing information, as described elsewhere herein. For example, the robotic device 340 may include an autonomous machine that replaces human labor associated with building a product, assembling a product, etc. In some implementations, the robotic device 340 may include an autonomous mobile robot (AMR), an automated guided vehicle (AGV), an articulated robot, a humanoid, a collaborative robot, a hybrid robot, etc. In some implementations, the robotic device 340 may include a communication device and / or a computing device.

[0060] The number and configuration of devices and networks shown in Figure 3 are provided as an example. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different configurations of devices and / or networks than those shown in Figure 3. Furthermore, two or more of the devices shown in Figure 3 may be implemented within a single device, or a single device shown in Figure 3 may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) of environment 300 may perform one or more functions that are described as being performed by another set of devices in environment 300.

[0061] 4 is a diagram of example components of an apparatus 400, which may correspond to the planning system 301, the server device 330, and / or the robotic device 340. In some implementations, the planning system 301, the server device 330, and / or the robotic device 340 may include one or more of the apparatus 400 and / or one or more components of the apparatus 400. As shown in FIG. 4, the apparatus 400 may include a bus 410, a processor 420, a memory 430, an input component 440, an output component 450, and a communication component 460.

[0062] The bus 410 includes components that enable wired and / or wireless communication between the components of the device 400. The processor 420 includes a central processing unit, a graphics processing unit, a microprocessor, a controller, a microcontroller, a digital signal processor, a field programmable gate array, an application specific integrated circuit, and / or another type of processing component. The processor 420 is implemented in hardware, firmware, or a combination of hardware and software. In some implementations, the processor 420 includes one or more processors that are programmable to perform functions. The memory 430 includes random access memory, read-only memory, and / or another type of memory (e.g., flash memory, magnetic memory, and / or optical memory).

[0063] The input component 440 enables the device 400 to receive input, such as user input and / or sensed input. For example, the input component 440 can include a touch screen, a keyboard, a keypad, a mouse, buttons, a microphone, switches, sensors, a global positioning system component, an accelerometer, a gyroscope, actuators, etc. The output component 450 enables the device 400 to provide output, such as via a display, a speaker, and / or one or more light emitting diodes. The communication component 460 enables the device 400 to communicate with other devices, such as via wired and / or wireless connections. For example, the communication component 460 can include a receiver, a transmitter, a transceiver, a modem, a network interface card, an antenna, etc.

[0064] The device 400 can execute one or more processes described herein. For example, a non-transitory computer-readable medium (e.g., memory 430) can store a set of instructions (e.g., one or more instructions, code, software code, program code, etc.) for execution by the processor 420. The processor 420 can execute the set of instructions to execute one or more processes described herein. In some implementations, execution of the set of instructions by the one or more processors 420 causes the one or more processors 420 and / or the device 400 to execute one or more processes described herein. In some implementations, hardwired circuitry may be used in place of or in combination with instructions to execute one or more processes described herein. Thus, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0065] The number and arrangement of components shown in Figure 4 are provided as an example. Apparatus 400 may include additional, fewer, different, or differently arranged components than those shown in Figure 4. Additionally or alternatively, a set of units (e.g., one or more components) of apparatus 400 may perform one or more functions that are described as being performed by another set of components of apparatus 400.

[0066] FIG. 5 is a flow chart illustrating an example process 500 for generating robotic assembly instructions from a 3D CAD model. In some implementations, one or more process blocks in FIG. 5 may be performed by an apparatus (e.g., planning system 301). In some implementations, one or more process blocks in FIG. 5 may be performed by another apparatus or apparatuses, such as a server apparatus (e.g., server apparatus 330) and / or a robotic apparatus (e.g., robotic apparatus 340). Additionally or alternatively, one or more process blocks in FIG. 5 may be performed by one or more components of apparatus 400, such as processor 420, memory 430, input component 440, output component 450, and / or communication component 460.

[0067] 5, process 500 may include receiving a 3D CAD model (block 505). For example, a device may receive the 3D CAD model as described above.

[0068] 5, the process 500 may include generating an assembly graph (block 510) having nodes representing components of the 3D CAD model and edges representing contacts between the components. For example, the apparatus may generate an assembly graph having nodes representing components of the 3D CAD model and edges representing contacts between the components as described above. In some implementations, each of the edges indicates one of contact between the respective components or non-contact between the respective components.

[0069] As further shown in FIG. 5, process 500 may include generating a component graph for each component of the assembly graph (block 515). For example, the apparatus may generate a component graph for each component of the assembly graph as described above. In some implementations, each of the component graphs includes one or more nodes representing one or more faces of the corresponding component and one or more edges connecting the one or more nodes representing the one or more faces. In some implementations, each component graph includes at least one node representing a face of the corresponding component, and at least one node includes a feature set, and the feature set includes at least one spatial feature and at least one structural feature.

[0070] 5, the process 500 may include generating an assembly descriptor based on the assembly graph and the component graph (block 520). For example, the apparatus may generate the assembly descriptor based on the assembly graph and the component graph, as described above. In some implementations, the assembly descriptor includes the component graph and a corresponding adjacency matrix.

[0071] As further shown in FIG. 5, the process 500 can include processing the assembly descriptor with a graph convolutional network model to generate a node embedding (block 525). For example, the apparatus can process the assembly descriptor with a graph convolutional network model to generate a node embedding, as described above. In some implementations, the graph convolutional network model includes a first portion having a first convolutional layer and a second portion having a second convolutional layer different from the first convolutional layer. In some implementations, the assembly descriptor includes a component graph, and processing the assembly descriptor with the graph convolutional network model to generate a node embedding includes encoding features of the component graph in the assembly descriptor to generate the node embedding.

[0072] 5, process 500 may include applying pooling to the node embeddings to generate a graph embedding (block 530). For example, the apparatus may apply pooling to the node embeddings to generate the graph embedding as described above. In some implementations, applying pooling to the node embeddings to generate the graph embedding may include one of applying average pooling to the node embeddings to generate the graph embedding, applying max pooling to the node embeddings to generate the graph embedding, or applying average pooling to the node embeddings to generate the graph embedding.

[0073] As further shown in FIG. 5, the process 500 may include computing cross-attention between components based on the graph embedding to generate component interrelationships (block 535). For example, the device may compute cross-attention between components based on the graph embedding to generate component interrelationships as described above. In some implementations, computing cross-attention between components based on the graph embedding to generate component interrelationships includes computing cross-attention between components based on the graph embedding to generate a single feature vector representing the component interrelationships. In some implementations, computing cross-attention between components based on the graph embedding to generate component interrelationships includes converting the graph embedding into a self-attention feature vector using scaled dot product attention with the graph embedding, aggregating the self-attention feature vector into a matrix, and processing the matrix to generate a single feature vector representing the interrelationships between the components.

[0074] 5, the process 500 may include utilizing the graph embedding and the component interrelationships to predict links between the components (block 540). For example, the apparatus may predict links between the components using the graph embedding and the component interrelationships as described above. In some implementations, predicting links between the components using the graph embedding and the component interrelationships includes processing the graph embedding and the component interrelationships with a link prediction model or a link classification model to predict links between the components.

[0075] 5, the process 500 may include predicting poses and joint axes for the components based on the graph embedding, the component interrelationships, and the links (block 545). For example, the apparatus may predict the poses and joint axes for the components based on the graph embedding, the component interrelationships, and the links, as described above. In some implementations, each of the poses is a six-degree-of-freedom pose of the corresponding component. In some implementations, each of the joint axes is a unit vector in three-dimensional space.

[0076] 5, the process 500 can include generating building instructions based on the graph embedding, the component interrelationships, the links, the pose, and the joint axes (Block 550). For example, the apparatus can generate building instructions based on the graph embedding, the component interrelationships, the links, the pose, and the joint axes, as described above.

[0077] 5, the process 500 can include having the assembly instructions executed by a robot to assemble the object represented by the 3D CAD model (block 555). For example, the device can cause a robotic device to execute the assembly instructions to assemble the object represented by the 3D CAD model, as described above.

[0078] The above disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications may be made in light of the above disclosure or may be acquired from practicing the implementation.

[0079] As used herein, the term "component" is intended to be broadly understood as hardware, firmware, or a combination of hardware and software. It will be apparent that the systems and / or methods described herein can be implemented in different forms of hardware, firmware, and / or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code. It should be understood that software and / or hardware can be used to implement the systems and / or methods based on the description herein.

[0080] As used herein, satisfying a threshold may refer to a value being greater than the threshold, greater than or equal to the threshold, less than the threshold, less than or equal to the threshold, equal to the threshold, etc., depending on the context.

[0081] Although particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of various implementations. Indeed, many of these features can be combined in other ways than those specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below depends directly on only one claim, the disclosure of various implementations includes each dependent claim in combination with every other claim in the claim group.

[0082] No element, act, or instruction used herein should be construed as critical or essential unless expressly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and can be used interchangeably with "one or more." Additionally, as used herein, the article "the" is intended to include one or more items referenced in relation to the article "the" and can be used interchangeably with "one or more of." Additionally, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and can be used interchangeably with "one or more." When only one item is intended, the term "only one" or similar language is used. Also, as used herein, the terms "has," "have," "having," and the like are intended to be open-ended terms. Additionally, the term "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Also, as used in this specification, the term "or" when used in a tandem is intended to be inclusive and can be used interchangeably with "and / or" unless expressly stated otherwise (e.g., when used in combination with "either" or "only one of").

[0083] In the foregoing specification, various embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the broader scope of the invention as defined in the following claims. The specification and drawings are therefore to be regarded as illustrative rather than restrictive.

Claims

1. receiving, by the device, a three-dimensional (3D) computer-aided design (CAD) model; generating, by the apparatus, an assembly graph having nodes representing components of the 3D CAD model and edges representing contacts between the components; generating, by the apparatus, a component graph for each component of the assembly graph; generating, by the apparatus, an assembly descriptor based on the assembly graph and the component graph; processing, by the apparatus, the assembly descriptor using a graph convolutional network model to generate a node embedding; applying pooling to the node embeddings to generate a graph embedding; calculating cross-attention between the components based on the graph embedding to generate component interrelationships; predicting, by the apparatus, links between components using the graph embedding and the component interrelationships; predicting, with the apparatus, poses and joint axes for the components based on the graph embedding and the component interrelationships and the links; generating, by the apparatus, assembly instructions based on the graph embedding, the component interrelationships, the links, the poses, and the joint axes; causing a robotic device to execute the assembly instructions to assemble an object represented by the 3D CAD model; and A method comprising:

2. The method of claim 1 , wherein each of the edges indicates one of a contact between the respective components or a non-contact between the respective components.

3. The method of claim 1 , wherein each of the component graphs includes one or more nodes that represent one or more faces of a corresponding component, and one or more edges that connect the one or more nodes that represent the one or more faces.

4. each of the component graphs includes at least one node representing a face of a corresponding component; the at least one node comprises a feature set; The method of claim 1 , wherein the feature set includes at least one spatial feature and at least one structural feature.

5. The method of claim 1 , wherein the assembly descriptor includes the component graph and a corresponding adjacency matrix.

6. 2. The method of claim 1 , wherein the graph convolutional network model includes a first portion having a first convolutional layer and a second portion having a second convolutional layer different from the first convolutional layer.

7. the assembly descriptor includes the component graph; 2. The method of claim 1 , wherein processing the assembly descriptor with the graph convolutional network model to generate the node embedding comprises encoding features of the component graph in the assembly descriptor to generate the node embedding.

8. One or more memories; one or more processors coupled to the one or more memories, Receiving a three-dimensional (3D) computer-aided design (CAD) model; generating an assembly graph having nodes representing components of the 3D CAD model and edges representing contacts between the components; generating a component graph for each component of the assembly graph; generating an assembly descriptor based on the assembly graph and the component graphs, the assembly descriptor including the component graph and a corresponding adjacency matrix; processing the assembly descriptor with a graph convolutional network model to generate node embeddings; applying pooling to the node embeddings to generate a graph embedding; calculating cross-attention between the components based on the graph embedding to generate component interrelationships; predicting links between the components using the graph embedding and the interrelationships of the components; and predicting poses and joint axes for the components based on the graph embedding and the component interrelationships and the links; generating assembly instructions based on the graph embedding, the component interrelationships, the links, the poses, and the joint axes; executing the assembly instructions by a robotic device to assemble an object represented by the 3D CAD model; and one or more processors configured to:

13. An apparatus comprising:

9. The one or more processors that apply the pooling to the node embeddings to generate the graph embeddings include: applying average pooling to the node embeddings to generate the graph embedding; applying max pooling to the node embeddings to generate the graph embedding; applying min pooling to the node embeddings to generate a graph embedding; The apparatus of claim 8 , configured to perform one of the following:

10. The one or more processors calculate the cross-attention between the components based on the graph embedding to generate interrelationships of the components. The apparatus of claim 8 , configured to compute the cross-attention between the components based on the graph embedding to generate a single feature vector that represents inter-relationships of the components.

11. The one or more processors calculate the cross-attention between the components based on the graph embedding to generate interrelationships of the components. converting the graph embedding into a self-attention feature vector using scaled dot product attention with the graph embedding; Aggregating the self-attention feature vectors into a matrix; processing the matrix to generate a single feature vector representing interrelationships between the components; The apparatus of claim 8 configured to:

12. Using the graph embedding and the component interrelationships to predict links between the components, The apparatus of claim 8 , wherein the one or more processors are configured to process the graph embedding and the component interrelationships with a link prediction model or a link classification model to predict links between the components.

13. The apparatus of claim 8 , wherein each of the poses is a six degree of freedom pose of a corresponding component.

14. The apparatus of claim 8 , wherein each of the joint axes is a unit vector in three-dimensional space.

15. A non-transitory computer readable medium storing a set of instructions, comprising: The set of instructions, when executed by one or more processors of a device, causes the device to: Receiving a three-dimensional (3D) computer-aided design (CAD) model; generating an assembly graph having nodes representing components of the 3D CAD model and edges representing contacts between the components; generating, for each component of the assembly graph, a component graph including one or more nodes, each node representing one or more faces of a corresponding component, and one or more edges connecting the one or more nodes representing the one or more faces; generating an assembly descriptor based on the assembly graph and the component graph; processing the assembly descriptor with a graph convolutional network model to generate node embeddings; applying pooling to the node embeddings to generate a graph embedding; calculating cross-attention between the components based on the graph embedding to generate component interrelationships; predicting links between the components using the graph embedding and the interrelationships of the components; and predicting poses and joint axes for the components based on the graph embedding and the component interrelationships and the links; generating assembly instructions based on the graph embedding, the component interrelationships, the links, the poses, and the joint axes; executing the assembly instructions by a robotic device to assemble an object represented by the 3D CAD model; and A non-transitory computer-readable medium comprising one or more instructions for causing a

16. 16. The non-transitory computer-readable medium of claim 15, wherein each of the component graphs includes at least one node representing a face of a corresponding component, the at least one node including a set of features, the set of features including at least one spatial feature and at least one structural feature.

17. the assembly descriptor includes the component graph; 16. The non-transitory computer-readable medium of claim 15, wherein the one or more instructions cause the device to process the assembly descriptor with the graph convolutional network model to generate the node embeddings, the one or more instructions causing the device to encode features of the component graph in the assembly descriptor to generate the node embeddings.

18. The one or more instructions for causing the apparatus to apply the pooling to the node embeddings to generate the graph embedding include: applying average pooling to the node embeddings to generate the graph embedding; applying max pooling to the node embeddings to generate the graph embedding; applying min pooling to the node embeddings to generate a graph embedding; 20. The non-transitory computer-readable medium of claim 15, further comprising:

19. The one or more instructions for causing the apparatus to compute the cross-attention between the components based on the graph embedding to generate interrelationships of the components include:

16. The non-transitory computer-readable medium of claim 15, further comprising computing the cross-attention between the components based on the graph embedding to generate a single feature vector representing interrelationships of the components.

20. The one or more instructions for causing the apparatus to compute the cross-attention between the components based on the graph embedding to generate interrelationships of the components include: converting the graph embedding into a self-attention feature vector using scaled dot product attention with the graph embedding; Aggregating the self-attention feature vectors into a matrix; The non-transitory computer readable medium of claim 15 , further comprising processing the matrix to generate a single feature vector representing interrelationships of the components.

Images