Method for determining object arrangement

Abstract

A method of determining an arrangement of objects includes obtaining first data representing a first arrangement of first objects in a scene; and inputting the obtained first data into a trained machine learning model to determine a first cost value of the first arrangement. The first cost value indicates a degree of difference between the first arrangement and an optimal arrangement of the first object according to a cost function that the machine learning model has been trained and provided. The method includes determining a second arrangement of the first object based on the first cost value.

Description

Technical Field

[0001] The invention relates to a method for determining an arrangement of objects. Background Art

[0002] It may be desirable for a robot to perform or assist with tasks, such as housework. Many such tasks involve the rearrangement of objects. For example, tidying up a room, setting cutlery and crockery on a table, loading a dishwasher and stacking a fridge are all tasks that involve placing certain objects in a particular or 'correct' arrangement. In order for a robot to complete such a task, it may be important to determine the 'correct' arrangement in which the objects are to be placed. However, this is challenging for a robot or computer to do, as the 'correct' arrangement may involve a complex mixture of factors. Furthermore, it may be desirable to determine such an arrangement in a flexible manner, for example in a manner that allows other factors to be taken into account, for example when controlling the robot to move objects. Summary of the invention

[0003] According to a first aspect of the present invention, there is provided a method for determining an arrangement of objects, the method comprising: obtaining first data representing a first arrangement of first objects in a scene, the first data comprising data representing the first objects and data representing a relative posture between the first objects; inputting the obtained first data into a trained machine learning model to determine a first cost value for the first arrangement, the trained machine learning model having been trained to provide a cost function which, based on the input of data representing the arrangement of objects, outputs a cost value indicating the degree of difference between the arrangement and an optimal arrangement, wherein the first cost value indicates the degree of difference between the first arrangement and the optimal arrangement of the first object according to the cost function; and determining a second arrangement of the first object based on the first cost value.

[0004] Determining a second arrangement of the first objects (e.g., an arrangement into which the robot is to arrange the first objects and / or an arrangement that is, for example, closer to a 'neat' arrangement than the first arrangement) based on a cost value indicating how much the first arrangement differs from the optimal arrangement (e.g., the output of a learned 'neatness' cost function from a machine learning model) may allow the second arrangement to be determined flexibly. For example, this may provide a more flexible determination than directly predicting the optimal arrangement. For example, determining the second arrangement based on a first cost value output by a cost function (e.g., a 'neatness' cost function) may allow the cost function to be combined with other cost functions (such as a time cost function) when determining the second arrangement, and / or may allow one or more of the object positions to be fixed when determining the second arrangement. Thus, a flexible determination of the second arrangement may be provided.

[0005] Optionally, the method comprises generating control instructions configured to cause the robot to move at least one of the first objects to a posture that the at least one first object has in the second arrangement. This may allow the robot to change the arrangement of the objects, for example to make the arrangement more neat. Thus, this may allow automatic rearrangement, for example to tidy up the objects to be provided.

[0006] Optionally, the method includes providing the control instructions to the robot so that the robot moves at least one of the first objects to a posture of the at least one first object in the second arrangement. This may allow objects in the scene to be placed in a neater arrangement.

[0007] Optionally, determining the second arrangement includes determining a second arrangement of the first object having a second cost value, the second cost value indicating that the second arrangement differs less from the optimal arrangement than the first arrangement. This can be determined by, for example, gradient descent and / or by sampling the cost function to identify such a second arrangement. This can allow the second arrangement to be an arrangement with a lower cost value than the first (initial) arrangement. Therefore, this can allow a second arrangement to be provided that is closer to the optimal value of the cost function.

[0008] Optionally, determining the second arrangement includes determining a gradient of the cost function at the first cost value. For example, descent following the determined gradient may allow automatic determination of a second arrangement (e.g., a 'tidier' ​​arrangement) having a lower cost value than the first arrangement. In some examples, gradient descent may be applied only once or a few times to determine a second arrangement having a cost value that is closer to a minimum of the cost function than the first arrangement. In some examples, gradient descent may be applied iteratively until a second arrangement is determined that is at or near a minimum of the cost function. In any case, determining the gradient may allow determination of the second arrangement in a relatively fast and resource-efficient manner.

[0009] Optionally, determining the second arrangement includes determining a minimum of the cost function. For example, the second arrangement can be determined as an arrangement whose cost value is at or near a minimum (e.g., a local or global minimum) of the cost function. For example, this can help allow identification of particularly neat arrangements.

[0010] Optionally, determining the second arrangement includes fixing the pose of one or more of the first objects of the first arrangement. For example, this can allow physical constraints (such as objects that are large and therefore cannot be moved) or user-specified constraints (such as objects that the user does not want to move) to be implemented when determining the second arrangement. This in turn can allow flexibility in determining the second arrangement.

[0011] Optionally, determining the second arrangement based on the first cost value comprises: combining the first cost value with one or more further cost values ​​of the first arrangement, thereby generating a first combined cost value, each of the one or more further cost values ​​being determined according to a respective further cost function and indicating a respective cost of the first arrangement compared to a respective further best according to the respective further cost function; and determining the second arrangement based on the first combined cost value. This may provide a balancing of a cost function (e.g. a 'neatness' cost function) with other costs, such as the time taken to produce a given arrangement and / or the extent to which certain arrangements may be impossible because certain objects may not be placed in certain spaces. This may provide a flexible and practical determination of the second arrangement, and may therefore, for example, provide a flexible and practical implementation of a tidying robot.

[0012] Optionally, the one or more additional cost values ​​include one or more of the following: an occupation cost value indicating the extent to which one or more first objects in the first arrangement occupy unoccupied space; and a time cost value indicating an estimate of the time required for the robot to interact with one or more first objects in the first arrangement. The occupation cost value may provide for incorporating physical realities or constraints of objects and / or scenes into the determination of the second arrangement. For example, the occupation cost value may be determined according to an occupation cost function that maps a cost value indicating the extent to which one or more of the first objects in the first arrangement occupy space that should not be occupied in the arrangement space. For example, the space that should not be occupied may include space for placing additional objects or immovable or fixed objects of the first arrangement. As another example, the space that should not be occupied may include space that should not be placed, for example, as specified by a user. As another example, the space that should not be occupied may include space that cannot support an object. The time cost value may provide for incorporating realities and constraints of the operation of the arrangement and / or robot into the determination of the second arrangement. For example, the time cost value may be determined according to a time cost function that maps a cost value indicating the time it takes for the robot to interact with one or more of the first objects in the first arrangement in the arrangement space. For example, the interaction may include touching (including, for example, moving to and / or physically contacting) one or more of the first objects, engaging (e.g., grabbing or picking up) one or more objects so that the one or more objects can be moved, and / or placing one or more objects in a certain position of a certain arrangement. In some examples, the time cost function may be a minimum value for a minimum time. In some examples, the time cost function may reflect a time budget. For example, a robot may be given a certain amount of time to complete a task, and, for example, an arrangement that will require the robot to establish more than a certain amount of time may be given a high time cost value. Other additional cost functions may be used. In any case, the use of additional cost functions may provide a flexible and practical determination of the second arrangement.

[0013] Optionally, determining the second arrangement includes determining a second arrangement having a second combination cost value, the second combination cost value indicating that the second arrangement differs less from the best arrangement and the corresponding one or more further best combinations than the first arrangement. For example, this can be determined by, for example, gradient descent and / or by sampling a combination cost function (i.e., a combination of the cost function and one or more further cost functions) to identify such a second arrangement. This can allow the second arrangement to be an arrangement with a lower combination cost value than the first (initial) arrangement. Therefore, this can allow a second arrangement to be provided that is closer to the best value of the combination cost function.

[0014] Optionally, determining the second arrangement may include determining the gradient of a combined cost function at a first combined cost value, the combined cost function being a combination of the cost function and one or more additional cost functions. For example, after the determined gradient descent may allow for automatic determination of a second arrangement having a lower combined cost value than the first arrangement. In some examples, the gradient descent may be applied only once or several times to determine a second arrangement having a combined cost value that is closer to the minimum of the combined cost function than the first arrangement. In some examples, the gradient descent may be applied iteratively until a second arrangement is determined at or near the minimum of the combined cost function. In any case, determining the gradient may allow for determining the second arrangement in a relatively fast and resource-efficient manner. For example, this may provide a relatively fast and computationally efficient method for considering multiple costs when determining a second arrangement of objects to be sorted.

[0015] Optionally, determining the second arrangement may include determining a minimum of a combined cost function that is a combination of the cost function and one or more additional cost functions. For example, the second arrangement may be determined as an arrangement whose combined cost value is at or near a minimum (e.g., a local or global minimum) of the combined cost function. For example, this may be useful to allow identification of arrangements that are particularly optimized in terms of multiple contributing costs.

[0016] Optionally, the obtained first data includes first graphical data representing a graph representing the first arrangement of objects in the scene, the graph including nodes and edges connecting the nodes, wherein each node represents a corresponding object, and each edge represents a relative posture between two objects represented by two nodes connected by the edge, wherein the trained machine learning model is a trained graphical neural network, the graphical neural network has been trained to provide a cost function, the cost function is based on the input of graphical data representing the graph of the arrangement of objects, and outputs a cost value indicating the degree of difference between the arrangement and the optimal arrangement. The inventors have realized that graphical neural networks can be well suited for learning object-object relationships, and therefore, providing the obtained first data as first graphical data and using the graphical neural network as a machine learning model can provide efficient and / or reliable determination of the second arrangement.

[0017] Optionally, the first graph data includes: for each node of the graph, a semantic vector representing the corresponding object; and / or for each edge of the graph, a relative posture vector representing the relative posture between two objects represented by the two nodes connected by the edge. For example, having each node include a semantic vector representing an object can allow an arrangement to be determined based on the type or property of the object. Alternatively or additionally, this can allow 'new' objects (where 'new' here refers to objects that the machine learning model has not yet been trained on, but are, for example, semantically similar to objects that the model has already been trained on) to be appropriately arranged. This can allow the second arrangement to be determined flexibly (e.g., flexible with respect to the types of objects to which the method can be applied) and / or in a fine-grained manner (e.g., fine-grained with respect to the types or properties of the objects in the arrangement).

[0018] Optionally, the method comprises: obtaining image data representing an image of an object of a scene in a first arrangement; and generating the first data based on the obtained image data. This may allow the second arrangement to be determined based on, for example, a single image of the scene. This may allow the second arrangement to be determined in a resource efficient manner.

[0019] Optionally, generating the first graphic data includes: generating a semantic vector for each of the one or more objects; and / or generating a posture vector for each of the one or more objects, the posture vector representing the posture of each of the objects, and determining the relative posture vector of each edge based on the posture vectors of the two objects of the two corresponding nodes connected by the edge. This can allow the semantic vector and / or posture vector to be generated from the acquired image. This can help to allow autonomous implementation of the method.

[0020] Optionally, the method includes training the machine learning model to provide a trained machine learning model, and wherein the training includes: obtaining a training data set, the training data set including a plurality of data sets, each data set representing an arrangement of objects in a scene and including data representing the objects and data representing relative poses between the objects, wherein each data set representing the arrangement of the objects is associated with a cost value label, the cost value label indicating the degree of difference between the arrangement and the optimal arrangement of the objects; and training the machine learning model based on the training data set to provide a cost function, the cost function outputting a cost value indicating the degree of difference between the arrangement and the optimal arrangement based on the input of the data representing the arrangement of the objects. This may allow a trained machine learning model to be provided. For example, training the machine learning model may include optimizing parameters of the model to minimize the loss between the cost values ​​predicted by the machine learning model for the arrangement of the training data set and the cost value labels of these arrangements. In some examples, optimizing the parameters may include using maximum likelihood estimation. In some examples, the training data set may be generated from a set of images showing neat arrangements. For example, the image set may be obtained by searching for neat arrangements, for example, on the Internet. For example, in the case where the objects are tableware and crockery for a dining table, the image set may be obtained by searching for images, for example, on the Internet with the search term 'dining table layout'. In some examples, this step may also be performed autonomously by a computer. Thus, in some examples, the training of the machine learning model may be autonomous or semi-autonomous. The cost value label may be continuous, or in some examples may be one of a plurality of discrete values, such as a binary value. For example, a training data set representing a neat arrangement of objects (thus providing a 'positive' training example) may have a cost value label of 0, while a training data set representing an irregular arrangement of objects (thus providing a 'negative' training example) may have a cost value label of 1. In some examples, the obtained training data set may not include such 'negative' examples, and may, for example, only include 'positive' examples. In some examples, the cost value label may be implicit, because including a certain arrangement in a set of 'positive' examples may indicate that the associated cost value label is low (e.g., 0). In some examples, 'negative' or other 'background' examples may be generated during training. For example, negative arrangements or background arrangements may be generated by modifying the relative poses between objects in a given 'positive' example arrangement, and these negative arrangements or background arrangements may be assigned relatively large cost value labels (e.g., 1) and included in the obtained training data set. Thus, in some examples, the method may include determining data representing an arrangement of objects that is different from one or more of the object arrangements in the obtained training dataset (thereby providing a 'negative' or 'background' arrangement); and including the determined data and an associated cost value label into the training dataset, the cost value label indicating that the object arrangement represented by the data is different from the optimal arrangement.As described above, the machine learning model may be a graph neural network, and the training dataset may accordingly include multiple graph datasets. Having the training dataset include permuted graph data may allow such 'negative' or 'background' examples to be generated efficiently.

[0021] According to a second aspect of the present invention, there is provided a method for training a machine learning model to provide a cost function for determining an arrangement of objects, the method comprising: obtaining a training data set, the training data set comprising a plurality of data sets, each data set representing an arrangement of objects in a scene and comprising data representing the objects and data representing the relative postures between the objects, wherein each data set representing the arrangement of the objects is associated with a cost value label, the cost value label indicating the degree of difference between the arrangement and the optimal arrangement of the objects; and training a machine learning model based on the training data set to learn a cost function, the cost function outputting a cost value indicating the degree of difference between the arrangement and the optimal arrangement based on the input of the data representing the arrangement of the objects. This may allow a trained machine learning model to be provided.

[0022] Optionally, the method may include determining data representing an arrangement of objects that is different from one or more of the arrangements of objects in the obtained training data set; and including the determined data and an associated cost value label into the training data set, the cost value label indicating that the arrangement of objects represented by the data is different from the optimal arrangement. For example, this may allow the generation of 'negative' or other 'background' examples during training.

[0023] Optionally, the training data set includes a plurality of graph data sets, each graph data set representing a graph representing an arrangement of objects in a scene, the graph comprising nodes and edges connecting the nodes, wherein each node represents a corresponding object, and each edge represents a relative pose between two objects represented by two nodes connected by the edge, and wherein the machine learning model is a graph neural network. As described above, graph neural networks can be very suitable for learning object-object relationships, and therefore this can provide a resource-efficient and / or reliable way to train machine learning models. In addition, the graph data can provide a resource-efficient method to generate 'negative' examples for machine learning model training. For example, as described above, negative training examples can be provided by generating counter-example graphs (e.g., by modifying one or more graphs of the provided positive example arrangements), which can be more resource-efficient than generating real images of irregular scenes. Alternatively or additionally, this can help allow training to be performed without human supervision or environmental interaction, such as in an autonomous and / or semi-autonomous manner.

[0024] According to a sixth aspect of the present invention, there is provided a device configured to execute the method according to the first aspect and / or the second aspect.

[0025] Optionally, the device is a robot configured to move one or more of the objects in the scene. This may help provide a robot that can autonomously or semi-autonomously organize scene objects.

[0026] According to a fourth aspect of the present invention, there is provided a computer program comprising instructions, which, when executed by a computer, causes the computer to perform a method according to the first aspect and / or the second aspect. In some examples, the computer program may be stored on a non-transitory computer-readable medium. Therefore, according to a fifth aspect of the present invention, there is provided a computer-readable storage medium storing instructions, which, when executed by a computer, causes the computer to perform a method according to the first aspect and / or the second aspect. For example, the computer may be part of a robot, or may be a remote server or other computing device, such as communicating with the robot.

[0027] Further features will become apparent from the following description taken with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart illustrating a method according to an example;

[0029] Figure 2 is a schematic diagram showing a first arrangement of objects of a scene according to an example;

[0030] Figure 3 is a schematic diagram showing a graphical representation of an arrangement of objects according to an example;

[0031] Figure 4 is a schematic diagram showing a flow between functional blocks according to an example;

[0032] Figure 5 is a schematic diagram showing a flow between functional blocks according to an example;

[0033] Figure 6 is a graph showing a plot of a projection of a cost function according to an example;

[0034] Figure 7 is a schematic diagram showing a second arrangement of objects of a scene according to an example;

[0035] Figure 8 is a schematic diagram showing a flow between functional blocks according to an example;

[0036] Fig. 9A is a schematic diagram illustrating a plot of an example cost function for an example set of objects;

[0037] Fig. 9B is a schematic diagram showing a plot of an example additional cost function;

[0038] Fig. 9Cis a schematic diagram of a plot showing a combination of an example cost function and an example additional cost function;

[0039] Fig.10 is a flow chart illustrating a method according to an example.

[0040] Fig.11 is a schematic block diagram illustrating a device according to an example. DETAILED DESCRIPTION

[0041] refer to Figure 1 , an example of a method for determining an arrangement of objects is shown. In general, and with reference to Figure 2 An example arrangement of objects depicted in the method includes:

[0042] - In step 102, first data representing a first arrangement 210 of first objects 204 in a scene is obtained, the first data comprising data representing the first objects and data representing relative postures between the first objects;

[0043] - in step 104, inputting the obtained first data into a trained machine learning model to determine a first cost value of the first arrangement, the trained machine learning model having been trained to provide a cost function that outputs a cost value indicating how much the arrangement differs from an optimal arrangement based on an input of data representing an arrangement of objects, wherein the first cost value indicates how much the first arrangement differs from an optimal arrangement of the first object according to the cost function; and

[0044] - In step 106, a second arrangement of the first object is determined based on the first cost value.

[0045] Determining a second arrangement of the first objects (e.g., an arrangement into which the robot is to arrange the first objects) based on a cost value indicating how different the first arrangement is from an optimal arrangement (e.g., an output of a learned 'neatness' cost function from a machine learning model) can allow the second arrangement to be determined flexibly.

[0046] refer to Figure 2 , an example scene 202 is shown. The scene 202 includes a first arrangement 210 of first objects 204, 206, 208. In this example, the objects are knife 208, fork 204, and plate 206. As humans, we can see the scene 202 and instinctively recognize that the first arrangement 210 of objects is not 'correct' or 'tidy'. The arrangement 210 is not 'tidy' because the plate 206 is not aligned with the fork 204 and the knife 208. Furthermore, we can instinctively recognize that in order to form a neat or correct arrangement, the plate should be placed as Figure 2204 and 206. As shown, the arrangement 210 is moved upward and placed on the centerline between the knife 208 and the fork 204. However, determining whether the arrangement 210 is 'correct' or 'tidy' or the degree to which the arrangement 210 is not 'correct' or 'tidy' and determining what constitutes a more correct or neater arrangement of the objects 204-208 is challenging for a robot or computer because it may depend on a complex mixture of factors. As a first step, data representing a first arrangement 210 of the objects 204-208 in the scene 202 is obtained, and calculations can be performed based thereon.

[0047] As described above, the first data obtained includes data representing the first objects 204-208 and data representing the relative posture between the first objects 204-208. In some examples, the data representing the first objects may include an identifier of each object 204-208 and / or information describing the objects 204-208. In some examples, the relative posture may include the relative distance between each object 204-208 and each other object 204-208 in the first arrangement. In some examples, the relative posture may include the relative orientation of each object 204-208 and each other object 204-208 in the first arrangement 210. The first data can be used to appropriately and accurately represent the first arrangement 210.

[0048] In some examples, the first data obtained includes first graphic data representing a graphic that in turn represents a first arrangement of objects in a scene. Figure 3 , a graph 300 representing a first arrangement 210 of objects in a scene 202 is shown. The graph includes nodes 304, 306, 308 and edges 310, 312, 314 connecting the nodes 304, 306, 308. Each node 304-308 represents a corresponding object 204-208, and each edge 310-314 represents a relative pose between two objects represented by the two nodes connected by the edge. For example, a first edge 310 connects a node 304 representing a fork 204 with a node 308 representing a knife 206, and represents a relative pose between them; a second edge 312 connects a node 308 representing a knife with a node 306 representing a plate 206, and represents a relative pose between them, and a third edge connects a node 306 representing a plate 206 with a node 304 representing a fork 304, and represents a relative pose between them. The inventors have appreciated that such a graph representation can be an efficient way of representing an arrangement of objects, particularly for the task of determining a neater or otherwise more 'correct' arrangement of those objects. The first graph data may include, for example, data representing a graph and suitable for input to a graph neural network. For example, the first graph data may include a node list and an edge list of the graph 300.

[0049] As described in more detail below, the trained machine learning model can be a trained graph neural network. The inventors have appreciated that graph neural networks can be well suited for learning object-object relationships, and therefore, providing the obtained first data as first graph data and using the graph neural network as a machine learning model can provide an efficient and / or reliable determination of the second arrangement. In some examples, the first graph data can include: for each node of the graph, a semantic vector representing the corresponding object; and / or for each edge of the graph, a relative posture vector representing the relative posture between two objects represented by the two nodes connected by the edge.

[0050] In some examples, the method may include generating first data. For example, in some examples, the method may include obtaining image data representing an image of objects 204-206 of a scene 202 in a first arrangement 210; and generating first data based on the obtained image data. This may allow a more optimized arrangement to be determined based on, for example, a single image of a scene. This may allow an arrangement to be determined in a resource-efficient manner. In some examples, the method may include generating first graphical data, which may include: generating a semantic vector for each of one or more objects; and / or generating a posture vector for each of one or more objects, the posture vector representing a posture of each of the objects, and determining a relative posture vector of each edge based on the posture vectors of the two objects of the two corresponding nodes connected by the edge. This may allow a semantic vector and / or a posture vector to be generated from the obtained image. This may help allow the method to be implemented autonomously.

[0051] refer to Figure 4 According to an example, an example process between functional blocks for determining first data is shown. In some examples, one or more of the steps performed by one or more of the functional blocks may form the above reference Figure 1 The method described herein is part of Figure 4 , an image 480 of the scene 202, for example, captured by a camera (not shown), is obtained. The image 480 is input to an object detector 482. For example, the object detector 482 can be configured to detect each of the objects 204-208 in the image 480 of the scene 202, and output a category of the object for each object (e.g., for the fork 204, the output category can be "fork"). For example, the object detector 482 can also output a segmentation mask (e.g., a mask consisting of pixels of the image 480 that determine the detected object) and a bounding box (e.g., the coordinates of the box that bounds the detected object) for each detected object. For example, the object detector 482 can include a pre-trained mask region-based convolutional neural network (R-CNN). Portions of the output 484 of the object detector 482 are then provided to the pose estimator 486 and the semantic embedding generator 488.

[0052] The pose estimator 486 estimates a pose 490 for each detected object 204-208. For example, for each detected object 204-208, the pose estimator 486 takes a segmentation mask as input. The pose estimator 486 estimates the position of the object by determining the centroid of all pixels of the segmentation mask for the object. The pose estimator 486 estimates the orientation of the object by: (1) using the segmentation mask to determine the longest direction of the object, and therefore the main axis of the object; and (2) using the tilt of the pixels in the segmentation mask to determine which direction along the main axis points to the head of the detected object. For each object, the pose estimator 486 can output the concatenation of the vector defining the determined position of the object and the vector defining the determined orientation of the object as the pose 490 of the detected object.

[0053] The semantic embedding generator 488 generates a semantic embedding for each detected object 204-208. For example, for each detected object 204-208, the semantic embedding generator 488 takes as input the pixels included in a bounding box for each object. For example, for each object, the semantic embedding generator 488 (1) crops the captured image 480 to include only the pixels within the bounding box for the object; and (2) passes the cropped image through a pre-trained model to determine the semantic embedding of the object. For example, the pre-trained model may include a trained neural network configured to output a semantic vector representing the position of the object in a semantic vector space, such that, for example, objects of similar type, nature, or category are located in similar regions of the semantic vector space. For example, the pre-trained model may be a contrastive language-image pre-trained model that may represent images as vectors by training on images with captions. Other pre-trained models may be used to determine the semantic embedding (vector) of an object. In some examples, before passing the cropped image of the object through the pre-trained model, the semantic embedding generator 488 can correct the cropped image to a fixed orientation (e.g., pointing up), such as the object orientation determined using the pose estimator 486. This can provide that the object is given the same semantic embedding regardless of the pose of the object. For each object, the semantic embedding generator 488 can concatenate the semantic vector determined for the object with the coordinates of the bounding box of the object to preserve information about the size of the object. The concatenation can be output as the semantic embedding 492 of the object.

[0054] The semantic embedding 492 and pose estimate 490 for each object are then provided to a graph generator 494 to generate a graph representing an arrangement of objects of the scene of the image 480. For example, as described above, the nodes of the graph can be the respective semantic embeddings of each object, and each edge of the graph can be the relative pose between the objects represented by the nodes connected by the edge. The graph generator 494 can output first data 440 representing the first arrangement. For example, the graph generator 494 can output first graph data representing the generator graph. For example, the graph generator 494 can output a list of all nodes of the graph (e.g., a list of semantic vectors for each object of the scene) and a list of all edges of the graph (e.g., a list of relative pose vectors for each pair of nodes).

[0055] It should be understood that the method of obtaining the first data 440 representing the first arrangement 210 of the objects 204-208 of the scene 202 is an example, and other methods may be used in other examples. Figure 1 Step 102 of the described method may or may not include the example method of obtaining the first data 440 described with reference to 4. In any case, in step 102, first data 440 representing a first arrangement is obtained.

[0056] As mentioned above, refer to Figure 1 In step 104 of the method, the first cost value indicates the degree of difference between the first arrangement 210 and the optimal arrangement of the first object according to the learned cost function.

[0057] refer to Figure 6 , shows an example of the projection of the cost function 658 on a two-dimensional plane. It is understood that the cost function can be a multidimensional function, for example, depending on the number of variables representing the first data of the arrangement. Figure 6 In the example of , the projection shows the cost value C as a function of the variable D, which may, for example, represent the position of the plate 206 relative to the fork 204 and the knife 208. In this example, the cost function 658 has a minimum 654 in the projection, and thus the cost value is lowest at the minimum 654. The minimum 654 may, for example, represent the learned optimal position of the plate 206 relative to the fork 204 and the knife 208 in the projection. For example, the minimum 654 may represent an optimal arrangement (at least in the projection) in which the plate 206 is centered between the fork 204 and the knife 208. It will be appreciated that the cost function may be multi-dimensional, and that according to the learned cost function, there may be a global minimum between these multiple dimensions that corresponds to the global optimal arrangement of the objects. Therefore, the cost value of any given arrangement may indicate how different the arrangement is from the optimal arrangement.

[0058] As described above, the cost value for a given permutation indicates how different the given permutation is from the best permutation according to the learned cost function. How the best permutation (e.g., an aligned permutation) may differ from other permutations (e.g., non-aligned permutations) may be learned during training of the machine learning model, for example, from example training permutations provided during training. Figure 6 In the example of , the first arrangement 652 differs from the optimal arrangement 654 in the position of the plate 206. However, it should be understood that in other examples, any number of differences can be learned. For example, the scene may include a cup, a kitchen table, and a kitchen counter, where in the first arrangement, the cup is placed on the kitchen table. The machine learning model may have been trained based on training data, including example arrangements in which the cup is placed on the table or on the kitchen counter (but not, for example, in mid-air between the table and the counter), and the optimal arrangement according to the learned cost function may be an arrangement in which the cup is placed on the kitchen counter. In this case, the machine learning model may learn (and the provided cost function may reflect) that the degree to which the arrangement of the cup in mid-air differs from the optimal arrangement of the cup on the counter is greater than the arrangement of the cup on the table, for example, because when the cup is in mid-air, the cup is not placed on / supported by the surface. Therefore, according to the learned cost function, the cost value output for the arrangement in which the cup is located in mid-air between the table and the counter can be relatively large, indicating that the degree to which the arrangement differs from the optimal arrangement is relatively large. In other words, the cost value of a given arrangement can indicate the likelihood (or its inverse) of the given arrangement under the training arrangement distribution (which is represented by the learned cost function). Permutations that are similar to the positive training permutations may be relatively 'likely' under the distribution, and therefore have low cost values, while permutations that are different from the positive training permutations may be relatively 'unlikely' under the distribution, and therefore have high cost values. Thus, it should be appreciated that the lower the cost value, the closer the permutation corresponds to the optimal permutation, according to the cost function. In some examples, the machine learning model may be an energy-based model, and thus the cost value for a given permutation may be considered to be the energy of the given permutation relative to the energies of other permutations.

[0059] In some examples, the trained machine learning model can be a trained neural network. For example, the trained neural network can implicitly represent or approximate a cost function. For example, the trained neural network can take as input a first data representing a first arrangement and output a cost value of the arrangement at its output layer according to the implicitly represented cost function. In some examples, the trained neural network can include a multilayer perceptron, such as a neural network having multiple hidden layers between an input layer and an output layer.

[0060] In some examples, as described above, the trained machine learning model can be a trained graph neural network. For example, the trained graph neural network can have been trained to provide a cost function that, based on an input of graph data representing a graph representing an arrangement of objects, outputs a cost value indicating how different the arrangement is from an optimal arrangement.

[0061] As is well known, graph neural networks implement optimizable transformations on properties of a graph, such as its nodes and edges, while preserving the symmetries of the graph. For example, the input to a graph neural network may include a list of semantic embeddings of nodes and a list of relative pose estimates of edges. The graph neural network may perform operations on these lists to provide updated node vectors and updated edge vectors. For example, at each layer of the graph neural network, a given node vector may be updated based on the neighboring node vectors and the edge vectors of the edges connecting them. For example, a given node i, j may have a corresponding feature vector x i , x j , and the edge connecting them can have an edge vector e ij A given layer of a graph neural network can compute an output feature vector x for each node i ',x j '. For example, the output feature vector x calculated for node i i ' can be provided by:

[0062]

[0063] in is the eigenvector x i 、x j and e ij Links, and is the function that the layer applies to the connection. According to equation (1), this function is applied to this connection for every other object j in the graph, and the results of these functions are added together to obtain the output vector x for node i i'. This can be done for each node of the graph. Thus, as the graph is passed through the layers of the graph neural network, each node vector and each edge vector is updated to reflect the properties of all other nodes in the graph and their connections. The last one or more layers of the graph neural network may include, for example, one or more layers that map a vector including a concatenation, sum, or other aggregation of all resulting node and edge vectors to a cost value (e.g., a scalar). For example, after passing through the layers of the graph neural network, there may be a node vector at each node of the graph. All of these node vectors may be added together into a single vector (which may be referred to as a graph encoding vector). The graph encoding vector may have the same dimensions regardless of how many nodes there are in the input graph. The graph encoding vector may then be passed through a multilayer perceptron neural network, which outputs a scalar, i.e., a cost value. Graph encoding vectors having the same dimensions regardless of how many nodes there are in the input graph may allow the graph neural network to be used to determine a cost value that is independent of the number of nodes in the input graph. This may allow for flexible determination of the cost value. The graph neural network may have been trained (e.g., its parameters and / or functions may have been optimized) so that the graph neural network implicitly represents a cost function that, based on an input of graph data representing a graph representing an arrangement of objects, outputs a cost value indicating how much the arrangement differs from an optimal arrangement according to the cost function. Thus, based on an input of first graph data, the trained graph neural network may output a first cost value indicating how much the first arrangement differs from the optimal arrangement according to the cost function.

[0064] In some examples, the trained machine learning model may be pre-trained and obtained, for example, from storage. In some examples, the method may include training the machine learning model to provide a trained machine learning model. In either case, in some examples, the training may include obtaining a training data set and training the machine learning model based on the training data set. For example, the training data set may include multiple (e.g., tens or hundreds or thousands) of data sets. Each data set may represent an arrangement of objects in a scene and include data representing the objects and data representing relative postures between objects. For example, each data set may include data similar to the first data representing the first arrangement as described above and / or may be obtained using a process similar to that described above for the first data. For example, in some examples (such as where the machine learning model includes a graph neural network), each data set may include graphical data representing a graph representing the arrangement, for example, similar to that described above for the first graphical data. In any case, each data set representing an arrangement of objects is associated with a cost value label indicating the degree of difference between the arrangement and the optimal arrangement of the object.

[0065] In some examples, the cost value label may be continuous, or in some examples may be one of a plurality of discrete values, such as a binary value. For example, a training data set representing neat arrangements of objects (thereby providing 'positive' training examples) may have a cost value label of 0, while a training data set representing irregular arrangements of objects (thereby providing 'negative' training examples) may have a cost value label of 1. In some examples, the obtained training data set may not include such 'negative' examples, and, for example, training may include training a machine learning model to output a relatively large cost value for arrangements different from the 'positive' examples provided in the training data set. For example, in some examples, 'negative' examples may be generated during training. For example, arrangements of 'negative' examples may be generated by modifying the relative poses between objects in a given 'positive' example arrangement, and the arrangements in these 'negative' examples may be assigned relatively large cost value labels (e.g., 1) and included in the obtained training data set. Thus, in some examples, the method may include determining data representing an arrangement of objects that is different from one or more of the arrangements of objects in the obtained training dataset (thereby providing 'negative' examples); and including the determined data and an associated cost value label into the training dataset, the cost value label indicating that the arrangement of objects represented by the data is different from the optimal arrangement.

[0066] In some examples, the cost value labels can be obtained by applying a labeling process to each data set (or the images on which each corresponding data set is based). In some examples, such a labeling process may not be performed explicitly. For example, in some examples, the training data set can be generated from a set of images showing neat arrangements of objects. For example, each of the images showing neat arrangements can be implicitly associated with a relatively low cost value label (e.g., 0). For example, the images can be obtained by performing an image search (e.g., on the Internet) for neat arrangements. For example, in the case where the objects are tableware and crockery for a dining table, the set of images can be obtained, for example, by performing an image search on the Internet with the search term 'dining table layout'. In some examples, this step can be performed autonomously by a computer. Therefore, in some examples, the training of the machine learning model can be autonomous or semi-autonomous.

[0067] The machine learning model can be trained based on the training data set to provide a cost function that outputs a cost value indicating the degree of difference between the arrangement and the optimal arrangement based on the input of the data representing the arrangement of the object. This can allow a trained machine learning model to be provided. For example, training the machine learning model can include optimizing the parameters of the model to minimize the loss between the cost values ​​predicted by the machine learning model for the arrangement of the training data set and the cost value labels of these arrangements (for example, through a loss function). As described above, the machine learning model can be a graph neural network. In this case, the training data set can accordingly include multiple graph data sets. During training, the parameters of the graph neural network (for example, the parameters of its function) can be optimized to minimize the loss between the cost value predicted by the graph neural network for each of the multiple (for example, tens or hundreds or thousands) training graphs representing the example arrangement of the object and the cost value label associated with each. In an example of generating a 'negative' training example based on a 'positive' example in the obtained training data, a training data set including the arranged graph data can allow the 'negative' example to be generated efficiently. For example, changing the edge vectors of the graph data of the positive example to generate a 'negative' (or other 'background') example can be more resource efficient than generating a real image of an irregular scene.

[0068] As described above, in some examples, the training data set may not include 'negative' examples, and may include 'positive' examples (e.g., only positive examples). In these examples, the cost value labels may be implicit, i.e., by including these permutations in the 'positive' example training data, these permutations are labeled as having a relatively low cost value (e.g., 0). In these examples, the loss function (on which the machine learning model may be optimized) may be based on maximum likelihood estimation. For example, as described above, the trained machine learning model may output a cost value that may be equal to the energy E of the input permutation x. θ (x). In some examples, this energy can be converted into a probability p θ (x), represents the probability that the input permutation x corresponds to the best (e.g., neat) permutation. For example, the probability p θ (x) can be given by the following equation:

[0069]

[0070] Where Z θ is the normalization term and is given by the following equation:

[0071]

[0072] This can be calculated directly or estimated, for example, by sampling some of the permutations. In these examples, the loss function L can be derived based on maximum likelihood estimation. The loss function can be used during training of the machine learning model, for example by optimizing the parameters of the machine learning model so that positive training examples (e.g., indexed by i) have high probability under the learning distribution of the model. Maximizing this probability is equivalent to minimizing the negative log-likelihood. Therefore, in some examples, the loss function L to be minimized during training of the machine learning model can be provided by the following equation:

[0073]

[0074] This loss function can help encourage permutations of positive examples to have high probabilities (and thus low cost values), while all other permutations have lower probabilities (and thus higher cost values). Thus, in some examples, the cost value for a given permutation can indicate how likely the given permutation is under the distribution of training permutations. For example, neat permutations can be "likely" under this distribution (because all training examples are neat), while random permutations can be "unlikely" because they are different from all examples.

[0075] In any case, a trained machine learning model may be obtained. As described above, the obtained first data 440 is input into the trained machine learning model to determine the first cost value of the first arrangement 210.

[0076] Back to Figure 1 As described above, in step 106, the method includes determining a second arrangement of the first object based on the first cost value. The second arrangement can be determined based on the first cost value in a variety of different ways, examples of which are described below.

[0077] In some examples, determining the second arrangement includes determining a second arrangement of the first objects 204-208 having a second cost value indicating that the second arrangement differs less from the optimal arrangement than the first arrangement 210. This can be determined by, for example, gradient descent and / or by sampling the cost function 658 to identify such a second arrangement. For example, again referring to Figure 6 , wherein a cost value 653 for one such second arrangement and a cost value 654 for another such second arrangement are indicated. In each case, the cost values ​​653, 654 are lower than the cost value 652 for the first arrangement. Thus, for example, the second arrangements may each correspond to an arrangement of the first items that is more orderly than the first arrangement 210.

[0078] In some examples, determining the second arrangement may include determining the gradient of the cost function 658 at the first cost value 652. For example, a trained machine learning model (e.g., a trained neural network, such as a trained graph neural network) may be differentiable. Thus, the gradient of the cost function provided by the trained machine learning model may be determined, for example, with respect to multiple variables of the cost function. For example, the gradient of the cost function 658 at the first cost value 652 may be determined, and the direction of the gradient descent of the cost function 658 at the first cost value 652 (such as the maximum negative gradient) may be identified. The gradient indicates a change in the variable of the cost function (e.g., a change in the arrangement of the objects, such as their relative pose) that will result in the maximum reduction in the cost value. Thus, determining the second arrangement may include determining the arrangement of the objects resulting from the indicated change. For example, this may include applying the indicated change (or the indicated change if multiple steps of gradient descent are used) to the first graph, thereby generating second graphic data representing a second graph representing a second arrangement of the objects. The second graph may be used, for example, to generate control instructions so that the robot moves one or more of the objects to be in the second arrangement represented by the second graph.

[0079] In some examples, gradient descent may be applied only once or a few times to determine a second arrangement whose cost value 653 is closer to the minimum of the cost function than the first arrangement. In some examples, gradient descent 656 may be applied iteratively until a second arrangement is determined whose cost value 654 is at or near the minimum of the cost function.

[0080] In some examples, gradient descent can include noise added at each gradient descent step. For example, in some examples, gradient descent can be performed using Langevin dynamics. For example, the permutation at the current step of the gradient descent process It can be given by the following equation:

[0081]

[0082] in is the permutation in the previous step of the gradient descent process, is the parameter of gradient descent, is the cost value or energy permuted by the cost function in the previous step The gradient at is the noise term. It can help ensure that the gradient descent process ends at the global minimum of the cost function rather than a local minimum.

[0083] In some examples, determining the second arrangement may include determining a minimum of the cost function. For example, the second arrangement may be determined as an arrangement whose cost value is at or near a minimum (e.g., a local or global minimum) of the cost function. For example, the minimum may be identified by applying a gradient descent process until the change in arrangement between consecutive steps is below a threshold.

[0084] In some examples, the second arrangement can be determined by sampling the cost function 658 to identify a second arrangement with a lower cost value than the first arrangement 210. For example, this can include changing the first data representing the first arrangement so that the object has a changed arrangement, such as a different relative posture. The changed first data can be input into a trained machine learning model to determine the cost value of the changed arrangement. This can be done once or repeated several times. The changed arrangement with the lowest cost value (and the cost value is lower than the first cost value) can be determined as the second arrangement.

[0085] In some examples, determining the second arrangement can include fixing the pose of one or more of the first objects of the first arrangement. For example, the objects can be fixed by constraining the pose of the objects to remain unchanged in the second arrangement. For example, where determining the second arrangement includes determining the gradient of the cost function at the first cost value, fixing the pose of one or more of the first objects can include constraining the gradient determination to not include determination of gradients with respect to changes in pose of the one or more fixed objects. In other words, fixing the pose can include constraining the gradient determination to only include determination of gradients with respect to changes in pose of objects whose poses have not been fixed). As another example, where determining the second arrangement includes sampling the cost function, changes to the first arrangement can be constrained so as not to change the pose of the one or more objects that have been fixed.

[0086] In any case, Figure 1 In step 106, a second arrangement of the first object is determined based on the first cost value. Figure 5 , schematically illustrating the process between functional blocks 500. In some examples, one or more steps performed in the functional blocks may form the above reference Figure 1 The method described herein is part of Figure 5, first data 440 representing the first arrangement 210 is input into a trained machine learning model 542, which outputs a first cost value 544. The first cost value 544 is provided to a second arrangement determiner 546, which determines a second arrangement 548 of the objects based on the first cost value 544. For example, the second arrangement determiner 546 interacts with the trained machine learning model 542 to determine the second arrangement (e.g., by determining the gradient of the cost function provided thereby or sampling the cost function, such as described above). The second arrangement determiner 546 outputs the determined second arrangement 548. For example, the output 548 can take the form of second data representing the second arrangement of the objects. For example, the output 548 can take the form of second graphic data representing a second graphic representing the second arrangement of the objects.

[0087] As mentioned above, in Figure 1 In step 106, a second arrangement of the first objects is determined based on the first cost value. In some examples, the method may also include generating a control instruction configured to cause the robot to move at least one of the first objects 204-208 to a posture that at least one of the first objects 204-208 has in the second arrangement 548. In some examples, the method may include providing a control instruction to the robot so that the robot moves at least one of the first objects 204-208 to a posture that at least one of the first objects 204-208 has in the second arrangement. For example, the control instruction may be derived from a difference between the first arrangement (represented by the first data) and the determined second arrangement (represented by the second data). For example, if a particular object 206 has a different posture (e.g., position) in the second arrangement than in the first arrangement, the control instruction may include an instruction to cause the robot to move the particular object 206 to or toward a posture (e.g., position) according to the second arrangement.

[0088] As an example, for Figure 2 Referring to the first arrangement 210 of objects 204-208 shown, the method may determine that the arrangement in which the plate 206 is centered between the knife 208 and the fork 204 has a lower cost value than the determined first cost value of the first arrangement 210, and may therefore be determined as the second arrangement. Figure 7 , showing Figure 2 A second arrangement 710 of objects 204-208 of scene 202. Figure 7 In the figure, the dotted circle 762 indicates that the plate is Figure 2In this example, the robot 760 has moved the plate 206 so that the objects 204-208 are in the determined second arrangement 710, so that the plate 206 is centered between the knife 208 and the fork 204. Thus, the objects 204-208 of the scene 202 can be automatically manipulated into a neat arrangement.

[0089] In some of the above examples, the second arrangement 710 is determined based solely on the cost function 658 provided by the trained machine learning model. However, this need not be the case, and in fact, determining the second arrangement based on the first cost value may allow for more flexibility in determining the second arrangement (such as allowing for additional cost functions to be considered in determining the second arrangement), for example, compared to directly predicting the best arrangement for the first object.

[0090] Therefore, in some examples, Figure 1 Step 106 of determining the second arrangement based on the first cost value may include combining the first cost value 652 with one or more further cost values ​​for the first arrangement to generate a first combined cost value. The one or more further cost values ​​may each be determined according to a respective further cost function, and may indicate a respective cost of the first arrangement compared to a respective further optimum value according to the respective further cost function. The method may then include determining the second arrangement based on the first combined cost value. This may provide a balance of a cost function (e.g. a 'neatness' cost function) with other costs, such as the time taken to produce a given arrangement and / or the extent to which certain arrangements may be impossible because certain objects may not be placed in certain spaces. This may provide a flexible and practical determination of the second arrangement, and may therefore, for example, provide a flexible and practical implementation of a tidying robot.

[0091] As an example, one or more additional cost values ​​may include an occupancy cost value indicating the extent to which one or more of the first objects in the first arrangement occupy space that should not be occupied. The occupancy cost value may provide for incorporating the physical reality or constraints of objects 204-208 and / or scene 202 into the determination of the second arrangement. For example, the occupancy cost value may be determined based on an occupancy cost function that maps a cost value indicating the extent to which one or more of the first objects in the first arrangement occupy space that should not be occupied in the arrangement space. For example, the space that should not be occupied may include space for placing additional objects or immovable or fixed objects of the first arrangement. As another example, the space that should not be occupied may include space that should not be placed, for example, as specified by a user. As another example, the space that should not be occupied may include space that cannot support an object.

[0092] As another example, one or more additional cost values ​​may include a time cost value indicating an estimate of the time it takes for the robot to interact with one or more of the first objects 204-206 in the first arrangement 201. The time cost value may provide for incorporating the actual conditions and constraints of the operation of the arrangement 210, 710 and / or the robot 760 into the determination of the second arrangement. For example, the time cost value may be determined according to a time cost function that maps a cost value indicating the time it takes for the robot 760 to interact with one or more of the first objects 204-206 in the first arrangement 210 in the arrangement space. For example, the interaction may include touching (including, for example, moving to and / or physically contacting) one or more of the first objects 204-208, engaging (e.g., grabbing or picking up) one or more objects 204-208 so that one or more objects can be moved, and / or placing one or more objects 204-208 at a certain position in a certain arrangement. In some examples, the time cost function may be a minimum value for a minimum time. In some examples, the time cost function may reflect a time budget. For example, a robot may be given a certain amount of time to complete a task, and a permutation that would require the robot more than the certain amount of time to build may be given a high time cost value, for example. Other additional cost functions may be used.

[0093] refer to Figure 8 , shows the flow between functional blocks 800 in the case where the second arrangement is determined based on the combination cost value. Figure 8 One or more of the steps performed in the functional blocks in the above reference may form Figure 1 The method described herein is part of Figure 8 , and the above reference Figure 5 Similarly, the obtained first data 440 representing the first arrangement 210 is input into the trained machine learning model 542, which outputs a first cost value 544 for the first arrangement 210. However, unlike Figure 5 The difference is that in Figure 8In the example of , the obtained first data 440 representing the first arrangement 210 is also input into the additional model 860. In some examples, the additional model 860 can take the form of a trained machine learning model that provides an implicit representation cost function. In other examples, the additional model 860 can take the form of the additional cost function itself. In any case, the additional model 860 provides an additional cost function and is configured to output an additional cost value 862 based on the input of the data representing the arrangement, which indicates the cost of the first arrangement 210 compared to the additional best according to the corresponding additional cost function. For example, in the case where the additional cost function is a time cost function representing the time taken by the robot 760 to place a given object 206 of the first arrangement 210 at a certain position, the cost function can be a minimum for the current position of the given object 206, and can, for example, increase with the increase of the distance from the current position of the given object 206. The first cost value 544 and the additional cost value 862 may be provided to a second arrangement determiner 846, which may be configured to combine the first cost value 544 and the additional cost value 862 to generate a first combined cost value. As an example, combining the first cost value 544 and the additional cost value 862 may include adding them together, and may, for example, involve performing a weighted sum of the first cost value 544 and the additional cost value 862. The second arrangement determiner 846 may then determine a second arrangement of the first objects 204-206 based on the first combined cost value. The determined second arrangement may be output 848 in the form of second data representing the second arrangement (e.g., second graphic data representing a graphic representing the second arrangement).

[0094] In some examples, determining the second arrangement may include determining a second arrangement having a second combined cost value indicating that the second arrangement differs less from the best arrangement and the corresponding one or more other best combinations than the first arrangement 210. For example, similar to what is described above with respect to the cost function, the second arrangement may be determined by, for example, gradient descent and / or by sampling a combined cost function (e.g., a combination of the cost function with one or more other cost functions, such as a sum or a weighted sum) to identify such a second arrangement.

[0095] In some examples, determining the second arrangement may include determining the gradient of a combined cost function at a first combined cost value, the combined cost function being a combination of the cost function and one or more additional cost functions. For example, the second arrangement determiner 846 may interact with both the trained machine learning model 542 and the additional model 860 to determine the gradient of the cost function at the first cost value and the gradient of the additional cost function at the additional cost value. For example, the corresponding gradients may be determined in a manner similar to that described above. In the case where the cost function and the additional cost function are independent of each other (and both depend on the first data), the two gradients may then be added together (e.g., using a weighted sum) to determine the gradient of the combined cost function at the first combined value.

[0096] In some examples, determining the second arrangement may include determining a minimum of a combined cost function that is a combination of the cost function and one or more additional cost functions. For example, the second arrangement may be determined as an arrangement whose combined cost value is at or near a minimum (e.g., a local or global minimum) of the combined cost function. The minimum may be identified in a manner similar to that described above for the cost function, such as using gradient descent (e.g., using Langevin dynamics).

[0097] 9A to 9C Combinations of cost functions and further cost functions according to specific examples are shown. Fig. 9A shows a plot of cost value (intensity) as a function of the x,y position of an object according to an example (and thus provides a visual representation of the cost function); Fig. 9B showing a plot of further cost value (intensity) as a function of the x,y position of the object (and thus providing a visual representation of a further cost function); and Fig. 9C A plot of the combination of the first cost value and the further cost value as a function of the x,y position of the object is shown (and thus providing a visual representation of the combined cost function).

[0098] In this specific example, and referring to Fig. 9A , each arrangement includes seven points (representing objects) arranged on a 2D plane and located at the vertices of two squares, where the two squares overlap at one vertex so that only one point is placed at that vertex. In each arrangement, the positions of the seven points are fixed. Each arrangement also includes a cross or 'x' (representing an object) on the 2D plane. Fig. 9A The intensity (in this case, darkness) of each pixel in the plot of represents the cost value of the output of the trained machine learning model for the input arrangement of dots and crosses, where the cross is placed at the x, y position corresponding to that pixel. Therefore, Fig. 9AThe plot of shows visually the cost function provided by the trained machine learning model as a function of the x,y position of the fork. The machine learning model has been trained on training data where positive examples correspond to arrangements where the fork is inside one of the two squares. In other words, in this example, a 'tidy' scenario is one where the object represented by the fork is inside one of the two squares where the object represented by the dot is. Thus, Fig. 9A As can be seen from the plot, the cost value of the arrangement shown (i.e., the cost value of the first and current arrangement of the dot and the cross) is relatively high (because it is outside the two squares), while the cost value of the arrangement where the cross is inside one of the two squares is the lowest.

[0099] refer to Fig. 9B , Fig. 9B The intensity (in this case, darkness) of each pixel in the plot of represents an additional cost value output by an additional cost function for the input arrangement of dots and crosses, where the cross is placed at the x, y position corresponding to that pixel. Thus, Fig. 9B The plot of visualizes an alternative cost function provided by the alternative model as a function of the x,y position of the fork. In this example, the alternative cost function is a time cost function, and the alternative cost value for a given pixel indicates the time it takes the robot to move the fork from its current x,y position (as shown in the figure) to the x,y position of the given pixel. Thus, Fig. 9B The plot of shows that the additional cost value of the first arrangement is at a minimum (because the fork is already positioned there), and the cost value increases as the distance of the fork from its initial position in the x,y plane increases. It is worth noting that the additional cost value in the lower left square is lower than the additional cost value in the upper right square because the current position of the fork is closer to the lower left square.

[0100] refer to Fig. 9C , Fig. 9C The intensity (in this case, darkness) of each pixel in the plot of represents a combination (e.g., a weighted sum) of the cost value for the input arrangement of dots and crosses placed at the x, y position corresponding to that pixel and an additional cost value. Thus, Fig. 9C The plot of intuitively shows the combination of the cost function and the additional cost function provided as a function of the x,y position of the fork. Fig. 9CAs can be seen from the drawing, the combined cost value of the first arrangement is relatively high (due to the influence of the cost value); the combined cost value of the arrangement in which the fork is located in the upper right square is also relatively high (due to the influence of another cost value); but the combined cost value of the arrangement in which the fork is located in the lower left square is relatively low (because the cost value and the other cost value here are both relatively low). Therefore, for example, using the method described above, such a gradient descent is performed from the combined cost value of the first arrangement, and a second arrangement can be determined in which the fork is located in the lower left square. For example, a control instruction can be generated to control the robot to position the object represented by the fork to be located in the lower left square of the object represented by the dot. Other additional cost functions can be used. In addition, the cost function can be combined with multiple additional cost functions. The use of additional cost functions can provide a flexible and practical determination of the second arrangement. Determining the second arrangement based on the first cost value can allow such a flexible and practical determination to be made (for example, allowing other costs to be incorporated into the determination), for example compared to directly predicting the optimal arrangement of the first object.

[0101] As described above, in some examples, a machine learning model can be trained. Fig.10 , a method for training a machine learning model to provide a cost function for determining an arrangement of objects is shown. The method includes obtaining a training data set in step 1002, the training data set including a plurality of data sets, each data set representing an arrangement of objects in a scene and including data representing objects and data representing relative postures between objects, wherein each data set representing an arrangement of objects is associated with a cost value label, the cost value label indicating the degree of difference between the arrangement and an optimal arrangement of the objects. The method includes, in step 1004, training a machine learning model based on the training data set to provide a cost function, the cost function outputting a cost value indicating the degree of difference between the arrangement and the optimal arrangement based on data input representing the arrangement of objects. In some examples, the training can be combined with the above reference Figure 1 The training of any of the examples described in FIG. 9 is the same or similar. In some examples, the trained machine learning model can be used as the above reference Figure 1 A trained machine learning model from the described method.

[0102] refer to Fig.11 , shows a device 1100 according to an example. The device 1100 may be configured to perform the above reference Figures 1 to 10 For example, the device 1100 includes a processor 1102 (e.g., including one or more graphics processing units (GPUs)), a memory 1108, an input interface 1104, and an output interface 1106. The memory 1108 may store a computer program including instructions that, when executed by the processor, cause the processor to perform the above-referenced Figures 1 to 10For example, the input interface 1104 can be used to receive training data, and the processor 1102 (e.g., including one or more GPUs) can be configured to train a machine learning model based on the training data, such as according to the above reference Figures 1 to 10 As another example, the input interface 1104 may be used to receive image data of a scene image, for example from a camera (not shown), and the processor 1102 may be configured to generate first data representing a first arrangement, for example according to the above reference Figures 1 to 10 The processor may be configured to determine a first cost value for the first arrangement, and to determine a second arrangement of the objects based on the first cost value, for example according to the above reference Figures 1 to 10 The output interface 1106 can be used, for example, to output data representing the second arrangement, or, for example, to output control instructions to cause the robot to move at least one of the objects to its pose in the second arrangement, for example, according to the above reference Figures 1 to 10 Any of the examples described. A non-transitory computer-readable medium storing instructions may be provided, which, when executed by the computer 1100, causes the computer 1100 to perform the above reference Figures 1 to 10 A method of any of the examples described.

[0103] In some examples, device 1100 may be a computer. In some examples, the device may be a remote server 1101 or a part of a remote server 1101. For example, remote server 1101 may be remote from robot 760 but may be communicatively coupled to robot 760 via wired or wireless means. In other examples, device 1100 may be robot 1101 or may be a part of robot 1101. For example, in some examples, device 1101 may be a robot (e.g., as described above with reference to Figure 7 The robot 760 described herein or a portion of a robot configured to move one or more of the objects in the scene. This can help provide a robot that can autonomously or semi-autonomously organize the scene objects.

[0104] The above examples should be understood as illustrative examples. It should be understood that any feature described with respect to any example can be used alone or in combination with other features described, and can also be used in combination with one or more features of any other example, or in combination with any combination of any other example. In addition, equivalents and modifications not described above may also be adopted within the scope of the appended claims.

Claims

1. A method for determining an arrangement of an object, the method comprising: obtaining first data representing a first arrangement of first objects in a scene, the first data comprising data representing the first objects and data representing relative postures between the first objects; inputting the obtained first data into a trained machine learning model to determine a first cost value of the first arrangement, the trained machine learning model having been trained to provide a cost function, the cost function outputting a cost value indicating the degree of difference between the arrangement and an optimal arrangement based on an input of data representing an arrangement of objects, wherein the first cost value indicates the degree of difference between the first arrangement and the optimal arrangement of the first object according to the cost function; and A second arrangement of the first objects is determined based on the first cost value.

2. The method according to claim 1, wherein: The method comprises: A control instruction is generated, the control instruction being configured to cause the robot to move at least one of the first objects to a posture that the at least one first object has in the second arrangement.

3. The method according to claim 2, wherein: The method comprises: The control instruction is provided to the robot so that the robot moves at least one of the first objects to a posture that the at least one first object has in the second arrangement.

4. A method according to any one of the preceding claims, wherein: Determining the second arrangement includes determining a second arrangement of the first object having a second cost value indicating that the second arrangement differs less from the optimal arrangement than the first arrangement.

5. A method according to any one of the preceding claims, wherein: Determining the second arrangement includes determining a gradient of the cost function at the first cost value.

6. A method according to any one of the preceding claims, wherein: Determining the second arrangement includes determining a minimum value of the cost function.

7. A method according to any one of the preceding claims, wherein: Determining the second arrangement includes fixing the pose of one or more of the first objects of the first arrangement.

8. A method according to any one of the preceding claims, wherein: Determining the second arrangement based on the first cost value includes: combining the first cost value with one or more further cost values ​​for the first arrangement, each of the one or more further cost values ​​being determined according to a respective further cost function and indicating a respective cost of the first arrangement compared to a respective further best according to the respective further cost function, thereby generating a first combined cost value; and The second arrangement is determined based on the first combined cost value.

9. The method according to claim 8, wherein: The one or more additional cost values ​​include one or more of the following: an occupancy cost value indicating the extent to which one or more of the first objects in the first arrangement occupy space that should not be occupied; as well as A time cost value indicating an estimate of time taken by a robot to interact with one or more of the first objects in the first arrangement.

10. The method according to claim 8 or claim 9, wherein: Determining the second arrangement includes: determining a second arrangement having a second combination cost value indicating that the second arrangement differs less from the best arrangement and the corresponding one or more other best combinations than the first arrangement; and / or determining a gradient of a combined cost function at the first combined cost value, the combined cost function being a combination of the cost function and the one or more further cost functions; and / or A minimum value of a combined cost function is determined, the combined cost function being a combination of the cost function and the one or more further cost functions.

11. A method according to any one of the preceding claims, wherein: The obtained first data includes first graphical data representing a graph representing the first arrangement of objects in the scene, the graph including nodes and edges connecting the nodes, wherein each node represents a corresponding object and each edge represents a relative posture between two objects represented by two nodes connected by the edge, wherein the trained machine learning model is a trained graphical neural network that has been trained to provide a cost function that outputs a cost value indicating the degree of difference between the arrangement and the optimal arrangement based on the input of the graphical data representing the graph representing the arrangement of objects.

12. The method according to claim 11, wherein: The first graphic data includes: For each node in the graph, a semantic vector representing the corresponding object; and / or For each edge of the graph, a relative pose vector is provided that represents the relative pose between two objects represented by two nodes connected by the edge.

13. A method according to any one of the preceding claims, wherein: The method comprises: obtaining image data representing an image of the object of the scene in the first arrangement; and The first data is generated according to the acquired image data.

14. A method according to claim 13 when dependent on claim 12, wherein: Generating the first graphic data includes: generating the semantic vector for each of the one or more objects; and / or A pose vector is generated for each of the one or more objects, the pose vector representing a pose of each of the objects, and the relative pose vector of each edge is determined based on the pose vectors of the two objects of the two corresponding nodes connected by the edge.

15. A method according to any one of the preceding claims, wherein: The method comprises training a machine learning model to provide the trained machine learning model, and wherein the training comprises: obtaining a training data set, the training data set comprising a plurality of data sets, each data set representing an arrangement of objects in a scene and comprising data representing the objects and data representing the relative poses between the objects, wherein each data set representing an arrangement of objects is associated with a cost value label indicating a degree of difference between the arrangement and an optimal arrangement of the objects; and The machine learning model is trained based on the training data set to provide a cost function that, based on an input of data representing an arrangement of objects, outputs a cost value indicating how different the arrangement is from an optimal arrangement.

16. A method of training a machine learning model to provide a cost function for determining an arrangement of objects, the method comprising: obtaining a training data set, the training data set comprising a plurality of data sets, each data set representing an arrangement of objects in a scene and comprising data representing the objects and data representing relative poses between the objects, wherein each data set representing an arrangement of objects is associated with a cost value label indicating a degree of difference between the arrangement and an optimal arrangement of the objects; and The machine learning model is trained based on the training data set to provide a cost function that, based on an input of data representing an arrangement of objects, outputs a cost value indicating how different the arrangement is from an optimal arrangement.

17. The method according to claim 16, wherein: The training dataset comprises a plurality of graph datasets, each graph dataset representing a graph representing an arrangement of objects in a scene, the graph comprising nodes and edges connecting the nodes, wherein each node represents a corresponding object and each edge represents a relative posture between two objects represented by two nodes connected by the edge, and wherein the machine learning model is a graph neural network.

18. The method according to claim 16 or claim 17, wherein: The method comprises: determining data representing an arrangement of objects that is different from one or more of the arrangements of objects in the obtained training data set; and The determined data and an associated cost value label are included in the training data set, the cost value label indicating that the arrangement of objects represented by the data is different from the optimal arrangement.

19. A device configured to perform the method according to any one of claims 1 to 18.

20. The apparatus of claim 19, wherein: The device is a robot configured to move one or more of the objects in the scene.

21. A computer-readable medium storing instructions, which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 18.

Images