Method for updating scene representation model

Abstract

A computer-implemented method for updating a scene representation model is disclosed. The method includes: obtaining a scene representation model representing a scene having one or more objects, the scene representation model configured to predict a value of a physical property of one or more of the objects; obtaining a value of the physical property of at least one of the objects, the obtained value derived from physical contact of the robot with the at least one object; and updating the scene representation model based on the obtained value. The invention further discloses equipment.

Description

Technical Field

[0001] The present invention relates to a method for updating a scene representation model. Background Art

[0002] In the field of computer vision and robotics, a scene representation model can be constructed to represent a scene that exists in real space in a virtual space. For example, in order to navigate through a real-world environment or scene, a robot can construct and use an internal scene representation model of that real-world environment or scene. The scene representation model can be generated using simultaneous localization and mapping (commonly referred to as "SLAM"), where a representation or mapping of a scene and a location within that scene can be determined simultaneously.

[0003] Edgar Sucar et al., arXiv number arXiv:2103.12352v2, "iMAP: Implicit Mapping and Positioning in Real-Time", discloses a real-time SLAM system including a scene representation model that predicts the geometry and color of objects of a scene. The scene representation model is optimized by minimizing the loss between a captured scene depth image and a volume rendering of the scene representation model. Thus, the optimized scene representation model can represent the geometry and color of objects of a scene.

[0004] The paper "iLabel: Interactive Neural Scene Labelling" by Shuaifeng Zhi et al., with arXiv number arXiv:2111.14637v2, discloses an extension of the real-time SLAM system in the "iMAP" paper. Specifically, the scene representation model further predicts the semantic categories of the objects of the scene. A graphical user interface is provided to allow the user to provide semantic category labels (e.g., "wall" or "book", etc.) to the captured depth image of the scene. The scene representation model is optimized by minimizing the loss between the rendered RGB (red-green-blue) values ​​of the scene representation model and the true RGB values ​​of the captured image, between the rendered depth values ​​of the scene representation model and the true depth values ​​of the scene from the camera, and between the rendered semantic category labels of the scene representation model and the annotations of the semantic category labels provided by the user. In this way, the optimized scene representation model can represent the geometry, color, and user-specified semantic categories of the objects of the scene.

[0005] However, the geometry and color of an object and / or the user-specified semantic category of an object (such as "book") only partially describes the actual nature of the object. Thus, known scene representation models are limited in how accurately they reflect real-world scenes. This in turn may limit the types of tasks that a robot can autonomously perform using known scene representation models. Moreover, user-specified semantic categories of objects require user input, which may limit the extent to which accurate scene representation models can be autonomously generated.

[0006] It would be desirable to mitigate at least some of these disadvantages. Summary of the invention

[0007] According to a first aspect of the present invention, there is provided a computer-implemented method for updating a scene representation model, the method comprising: obtaining a scene representation model representing a scene having one or more objects, the scene representation model being configured to predict values ​​of physical properties of one or more of the objects; obtaining values ​​of the physical properties of at least one of the objects, the obtained values ​​being derived from physical contact between a robot and the at least one object; and updating the scene representation model based on the obtained values.

[0008] This may allow the scene representation model to include physical properties of the object derived from the robot's physical contact with the object. For example, this may allow the updated scene representation model to accurately predict physical properties that are difficult or possible to determine from images alone. This in turn may allow the updated scene representation model to more completely represent the essence of the object. This in turn may allow the scene representation model to more accurately reflect most real-world scenes. Thus, an improved scene representation model may be provided. Alternatively or in addition, this may allow, for example, a robot to operate based on the updated scene representation to perform a wider range of tasks and / or to perform tasks more accurately. For example, this may allow a robot to perform tasks based on the physical properties of the objects of the scene (e.g., classifying boxes of the same geometry and color but different masses based on the mass of the objects). Alternatively or in addition, the obtained values ​​derived from the robot's physical contact with the at least one object may allow an accurate scene representation to be obtained autonomously, for example without necessarily requiring input from a user.

[0009] Optionally, the physical contact of the robot with the at least one object of the scene comprises the robot physically moving or attempting to physically move the at least one object of the scene. The obtained value is then derived from the physical movement or the attempt. A robot interacting with a scene may physically move an object (or at least attempt to move an object), and this does not necessarily require a specific measurement probe. Therefore, deriving the value of the physical property based on the movement or attempted movement may allow the value of a physical property of an object of the scene (e.g., mass, coefficient of friction, hardness, or a lower bound of these) to be determined in a cost-effective manner.

[0010] Optionally, the physical contact may include one or more of a top-down stabbing of the at least one object, a lateral pushing of the at least one object, and a lifting of the at least one object. Different types of physical contact may determine different physical properties in a cost-effective manner. In some examples, in each case of stabbing, pushing, or lifting, the distance that the robot moves upon contact may be relatively short, such as in the range of a few millimeters, which may be sufficient to sample the physical properties of the object, but not sufficient to substantially damage or change the position of the object in the scene. In other examples, the at least one object may move and / or change position due to the physical contact.

[0011] Optionally, the value of the physical property indicates one or more of the flexibility or hardness of the at least one object, the coefficient of friction of the at least one object, and the mass of the at least one object. These properties can be determined by the robot moving the object (or attempting to move the object), and therefore no specialized measurement probe is required to contact the object, which may be cost-effective. In one example, the coefficient of friction can be estimated based on the sound made by the robot when moving the object.

[0012] Optionally, the physical contact of the robot with the at least one object comprises physical contact of a measurement probe of the robot with the at least one object, wherein the obtained value is derived based on the output of the measurement probe when contacting the at least one object. This may allow a wider range of physical properties to be determined, such as spectral characteristics, thermal conductivity, material porosity and material type.

[0013] Optionally, the method comprises: selecting the at least one object from the one or more objects based on the uncertainty of the predicted value of the physical property of each of the plurality of objects; controlling the robot to make physical contact with the selected object; and deriving the value of the physical property of the selected object from the physical contact, thereby obtaining the value. This may allow the robot to autonomously select (e.g., without input from a user) the object to make physical contact, which may result in the greatest reduction in uncertainty in the scene representation model and therefore provide the greatest gain in accuracy and / or reliability of the scene representation model. This may provide an accurate and / or reliable scene representation model to be created autonomously.

[0014] Optionally, selecting the at least one object comprises: determining a kinematic cost and / or kinematic feasibility of the physical contact of the robot with each of the plurality of objects; and the at least one object is selected based additionally on the determined kinematic feasibility for each of the plurality of objects. This may allow the robot to avoid or reduce attempts to obtain values ​​of the objects that the robot cannot contact or that are kinematically costly to contact. This may allow the scene representation model to be updated in an efficient manner.

[0015] Optionally, the method includes: in response to determining that the physical contact of the robot with a given object of the plurality of objects is kinematically not feasible, adding the given object to a selection mask to prevent the given object from being selected in further selections of objects for which the value of the physical property is to be obtained. This may allow for objects for which physical contact is kinematically not feasible to not require the kinematic cost and / or the kinematic feasibility to be determined multiple times over multiple object selections. This may further improve the efficiency with which the scene representation model is updated.

[0016] Optionally, the scene representation model provides an implicit scene representation of the scene. For example, the implicit scene representation may implicitly represent the scene by providing a mapping function between spatial coordinates and scene properties. This may allow the scene to be represented (and its properties to be queried) in a resource efficient manner. For example, this may be compared to an explicit scene representation (such as a point cloud or mesh of the scene), which may require relatively large resource usage to store, maintain and interact with. The scene representation model that provides an implicit scene representation may in turn increase the speed at which the scene representation model can learn, adapt and / or query. This in turn may help to allow real-time application of the method.

[0017] Optionally, the scene representation model comprises a multi-layer perceptron having a semantic head, wherein for a coordinate of the scene representation input to the multi-layer perceptron, the semantic head outputs the prediction of the value of the physical property at that coordinate. Such an implementation may provide a particularly resource efficient way to provide predictions of the value of a physical property of an object. This in turn may help to allow real-time application of the method.

[0018] Optionally, the multilayer perceptron further has a volume density head and / or a photometric head, wherein for the coordinate of the scene representation input into the multilayer perceptron, the volume density head outputs a prediction of the volume density at that coordinate, and / or the photometric head outputs a prediction of the photometric value at that coordinate. This may provide that the scene representation model also predicts photometric values ​​(e.g., color, brightness) and geometry (e.g., shape) in a resource efficient manner, which may help to allow the method to be used in real-time applications such as real-time SLAM. Alternatively or in addition, the semantic head and the volume density and / or photometric head sharing the same multilayer perceptron backbone may provide that improved predictions of physical property values ​​obtained for a portion of the scene (e.g., a portion of an object) may be automatically propagated to other similar portions (e.g., other portions of the same object). This may increase the efficiency with which the scene representation model is updated.

[0019] Optionally, updating the scene representation model comprises: optimizing the scene representation model so as to minimize the loss between the obtained value and the predicted value of the physical property of the at least one object. This may provide that the predictions of the physical property values ​​of all the objects of the scene representation model are updated based on the obtained value. This may provide a more complete scene representation for a given number of obtained values ​​and / or for the scene representation to be updated efficiently.

[0020] Optionally, the obtained value and the predicted value each represent one of a plurality of discrete values, and optimizing the scene representation model comprises minimizing a cross entropy loss between the predicted value and the measured value; or the obtained value and the predicted value each represent one of a continuous value, and optimizing the scene representation model comprises minimizing a mean square error loss between the predicted value and the measured value. This may allow the scene model to be updated, wherein the value is a discrete value (e.g., a category) or a continuous value (e.g., a hardness).

[0021] Optionally, updating the scene representation comprises: marking a portion of a captured image of the scene with the obtained value of the object represented by the portion; obtaining a virtual image of the scene rendered from the scene representation model, one or more portions of the virtual image being marked with the corresponding predicted values ​​of the corresponding object represented by the corresponding portion; determining a loss between the obtained value of the marked portion of the captured image and the predicted value of the corresponding portion of the virtual image; and optimizing the scene representation model so as to minimize the loss. This may provide that the scene representation model may be updated efficiently. This may provide that the entire scene representation is updated based on one (or more) captured (and marked) images of the scene. This in turn may provide an efficient method of updating the model.

[0022] Optionally, one or more portions of the captured image are each labeled with an obtained depth value from a camera that captured the image, the obtained depth value indicating the depth of a portion of the scene represented by the portion; one or more portions of the virtual image are each labeled with a predicted depth value from a virtual camera from which the virtual image is rendered, the predicted depth value indicating the depth of a portion of the scene representation represented by the portion; and updating the scene representation model comprises: determining a geometry loss between the obtained depth values ​​of the one or more portions of the captured image and the predicted depth values ​​of one or more corresponding portions of the virtual image; and optimizing the scene representation model so as to minimize the geometry loss. This may allow both the physical property values ​​and the geometry of the scene representation to be jointly optimized based on one (or more) captured images. This may provide an efficient means of updating the scene representation model. Alternatively or in addition, this may help provide for propagating the obtained physical property values ​​to other portions of the same object and / or similar objects.

[0023] Optionally, one or more portions of the captured image are each labeled with an acquired photometric value under a camera that captured the image, the acquired photometric value indicating a photometric property of a portion of the scene represented by the portion; one or more portions of the virtual image are each labeled with a predicted photometric value under a virtual camera to which the virtual image is rendered, the predicted photometric value indicating a predicted photometric property of a portion of the scene representation represented by the portion; and wherein updating the scene representation model comprises: determining a photometric loss between the acquired photometric values ​​of the one or more portions of the captured image and the predicted photometric values ​​of one or more corresponding portions of the virtual image; and optimizing the scene representation model so as to minimize the photometric loss. This may allow the physical property values ​​and the photometric values ​​(e.g., color, lighting) (and geometry) of the scene representation to be jointly optimized based on one (or more) captured images. This may provide an efficient means of updating the scene representation model. Alternatively or additionally, this may help to provide for propagation of the obtained physical property values ​​to other parts of the same object and / or other similar objects.

[0024] Optionally, the one or more portions of the captured image are sampled from the captured image, and the predicted depth values ​​and / or the predicted photometric values ​​are predicted for corresponding samples of the one or more portions of the virtual image based on output from the scene representation model. Thus, the update need not be based on the entire captured image, but rather on samples thereof. This may help to increase the speed at which the scene representation model can be updated. This in turn may allow real-time applications of the method, such as real-time SLAM.

[0025] Optionally, the method comprises: estimating the pose of a camera that captured the image when the captured image was captured; and wherein the virtual image is rendered under a virtual camera having the estimated pose. This may provide that the camera that captured the image does not have to have a fixed position and / or orientation, but may be moved. This may provide flexibility in the deployment of the method.

[0026] Optionally, the pose of the camera is estimated based at least in part on data indicative of a configuration of a device for positioning the camera. This may facilitate efficient estimation of the camera pose, for example where the camera is attached to a robotic arm, the base of which is fixed and whose joint configuration is known (and from which the camera pose may be derived).

[0027] Optionally, the pose of the camera is estimated based at least in part on an output of a pose estimation module configured to estimate the pose of the camera, wherein optimizing the scene representation model comprises jointly optimizing the pose estimation module and the scene representation model to minimize the loss. This may help estimate the camera pose even when the camera is freely moveable (e.g., by a human user or by a robot). Alternatively or additionally, this may help fine-tune the camera pose estimate from an initial estimate (e.g., based on a joint configuration of a robotic arm).

[0028] Optionally, the obtained scene representation model is pre-trained by optimizing the scene representation model so as to minimize a loss between a provided estimate of the value of the physical property of at least one object of the scene and the predicted value of the physical property of the at least one object. This may help to provide a relatively accurate scene representation relatively quickly, for example compared to starting with no information about the value of the physical property of any part of the scene. These estimates may be based on, for example, a typical hardness range for household objects.

[0029] Optionally, the estimate is provided by: applying a pre-trained object detector to the captured image to identify the at least one object; and inferring the estimate from the identity of the at least one object. This may provide a more accurate initial estimate of the physical property. This in turn may reduce the time (and interactions) spent generating a relatively accurate scene representation model. For example, object recognition may first be applied to the captured image to identify a portion as a chair, and the estimate of the physical property value may be a typical mass of a chair.

[0030] Optionally, the method comprises: controlling the robot (or another robotic device) to perform tasks based on the updated scene representation model. This may allow the robot to complete a wider range of tasks and / or to complete tasks more accurately than if the model had not been updated. For example, this may allow the robot to perform tasks defined based on physical properties of objects of the scene, such as sorting boxes of the same appearance but different masses based on the mass of the objects.

[0031] According to a second aspect of the invention, there is provided a device configured to perform the method according to the first aspect. In some examples, the device may be a robot, such as the robot of the first aspect. In some examples, the device may include a computer configured to perform the method. For example, the computer may be a remote server, such as a remote server from the robot.

[0032] According to a third aspect of the present invention, there is provided a computer program comprising a set of instructions, which, when executed by a computer, causes the computer to perform a method according to the first aspect. In some examples, the computer may be part of the robot. In some examples, the computer may be part of a remote server that is remote from the robot but is communicatively connected to the robot, for example, via a wired or wireless manner. The computer may include a processor and a memory, the memory storing instructions, which, when executed by the processor, causes the processor to perform a method according to the first aspect. According to a fourth aspect of the present invention, there is provided a non-transitory computer-readable medium having instructions stored thereon, which, when executed by a computer, causes the computer to perform the method of the first aspect.

[0033] Further features will become apparent from the following description taken with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart illustrating a method according to one example;

[0035] Figure 2 is a schematic diagram showing a scene in a real space according to an example;

[0036] Figure 3 is a schematic diagram of a representation of a scene in a virtual space according to an example;

[0037] Figure 4 is a block diagram illustrating a scene representation model according to an example;

[0038] Figure 5 is a block diagram showing the flow between functional blocks according to one example; and

[0039] Figure 6 is a block diagram illustrating a device according to one example. DETAILED DESCRIPTION

[0040] refer to Figure 1 , a computer-implemented method for updating a scene representation model is shown. In summary, the method comprises:

[0041] - in step 102, a scene representation model representing a scene having one or more objects is obtained, the scene representation model being configured to predict a value of a physical property of one or more of the objects;

[0042] - in step 104, obtaining a value of said physical property of at least one of said objects, said obtained value being derived from physical contact of the robot with said at least one object; and

[0043] - In step 106, the scene representation model is updated based on the obtained values.

[0044] This may allow the scene representation model to include physical properties of the object derived from the robot's physical contact with the object. For example, this may allow the updated scene representation model to accurately predict physical properties that are difficult or impossible to determine from the image alone. This in turn may allow the updated scene representation model to more completely represent the essence of the object. This in turn may allow the scene representation model to more accurately reflect most real-world scenes. Thus, an improved scene representation model may be provided. Alternatively or in addition, this may allow, for example, a robot to operate based on the updated scene representation to perform a wider range of tasks and / or perform tasks more accurately. For example, this may allow a robot to perform tasks based on the physical properties of the objects of the scene (e.g., sorting boxes of the same geometry and color but different masses based on the mass of the objects).

[0045] As described above, step 102 includes obtaining a scene representation model representing a scene having one or more objects.

[0046] refer to Figure 2 , a scene 202 with one or more objects 204, 206 according to an example is shown. The scene 202 is in real space (i.e., the scene 202 is a real-world scene). Positions within the scene 202 are defined by coordinates x, y, z. In this example, the scene 202 has two objects 204, 206. In this example, each object 204, 206 is a box, and the boxes have the same appearance and geometry. The scene 202 is a real-world environment of a robot 208. In this example, the robot 208 has a camera 210 and a measuring probe 212. In this example, both the camera 210 and the measuring probe 212 can move in the scene / environment 202. Specifically, in this example, the robot 208 takes the form of a robotic arm 208. The robotic arm 208 includes a base 214 and a plurality of movable sections 216 connected by joints 215. The camera 210 and the measuring probe 212 are located on the terminal section of the arm 208. The joints 215 of the robotic arm 208 may be configured to position the camera 210 and / or the measurement probe 212 at any particular location and orientation within the scene 202. Figure 2 In the example of , joint 215 is configured so that camera 210 points to objects 204, 206. In this example, objects 204, 206 are movable objects, but it can be appreciated that in some examples, the objects of the represented scene can be any objects, including, for example, immovable or virtually immovable objects, such as the walls or floor of a room.

[0047] The obtained scene representation model represents the scene 202. In some examples, the obtained scene representation model may provide an explicit representation of the scene 202 (e.g., using a point cloud or mesh, etc.). However, in other examples, the obtained scene representation model may provide an implicit representation of the scene. For example, the obtained scene representation model may provide a mapping function between spatial coordinates and scene properties (such as volume density and color). If desired, an explicit representation of the scene 202 in the virtual space may be constructed from the implicit representation, such as by determining, for each coordinate in the virtual space, a scene property output from the implicit representation. In any case, the obtained scene representation model represents the scene 202.

[0048] refer to Figure 3 , showing Figure 2 For ease of visualization, Figure 3 The scene representation 202' in is explicit, but it should be understood that in some examples, the obtained scene representation model can provide an implicit scene representation, as described above. The scene representation 202' is in a virtual space. The position within the scene representation 202' is defined by virtual coordinates x', y', z'. Through appropriate coordinate transformations, the virtual coordinates x', y', z' can be converted into real-world coordinates x, y, z, and vice versa. The scene representation 202' includes object representations 204', 206' representing objects 204, 206 of the scene 202. The scene representation 202' also includes a camera representation 210' representing the position and orientation (hereinafter referred to as "pose") of the camera 210 in real space.

[0049] As described above, the scene representation model is configured to predict values ​​of physical properties of one or more of the objects 204, 206. For example, the scene representation model may be configured to output a prediction of a physical property (such as mass, stiffness, etc.) for each object 204', 206'. For example, the scene representation model may be configured to determine, for each coordinate x', y', z' of the virtual space, a prediction of a physical property of the object or portion of the object occupying that coordinate.

[0050] refer to Figure 4, an example of a scene representation model 432 is shown. In this example, the scene representation model 432 provides an implicit scene representation. Specifically, in this example, the scene representation model 432 includes a multilayer perceptron 433. The multilayer perceptron can be a neural network having multiple fully connected layers. For example, the multilayer perceptron 433 can be, for example, a 4-layer MLP including four fully connected layers. The multilayer perceptron 433 has a semantic head 438. For the coordinates 430 of the scene representation 202' input to the multilayer perceptron 433, the semantic head 438 outputs a prediction 439 of the value of the physical property at that coordinate. In this example, the multilayer perceptron 433 further has a volume density head 436. For the coordinates 430 of the scene representation 202' input to the multilayer perceptron 433, the volume density head 436 outputs a prediction 437 of the volume density at that coordinate. In this example, the multilayer perceptron further has a photometric head 434. For a coordinate 430 of the scene representation 202' input to the multi-layer perceptron 433, the photometric head 434 outputs a prediction 435 of the photometric value (such as radiometry) at that coordinate. This can provide that the scene representation model 432 can predict physical properties (such as mass, hardness, material type, etc.) as well as photometric values ​​(e.g. radiometry) and geometry (e.g. shape) in a resource efficient manner. The semantic head 438 and volume density head 436 and / or photometric head 434 sharing the same multi-layer perceptron 433 backbone can provide that updates to the prediction of a physical property value for one portion of the scene representation (such as a portion of an object) can be automatically propagated to other portions of the same geometric (or photometric) contiguous region of the scene representation (such as other portions of the same object). This can increase the efficiency with which the scene representation model is updated.

[0051] Scene representation model 432 may be trained to represent scene 202. That is, parameters of scene representation model 432 may be optimized so that the scene representation model accurately predicts the geometry and appearance of scene 202. This may be achieved, for example, by minimizing the loss between obtained properties of the scene (such as physical properties, geometric properties, and photometric properties) and the prediction of those properties by scene representation model 432. For example, this may include: obtaining a labeled captured image (or a sampled portion thereof) and a rendered virtual image (or a corresponding sampled portion thereof) of the scene from scene representation model 432; and optimizing the scene representation model so as to minimize the loss between the captured image and the rendered image.

[0052] For example, the scene representation model 432 may be formulated as:

[0053]

[0054] in is a multilayer perceptron 433 parameterized by θ; c, s and The 3D positions are = radiance, physical properties, and volume density at (x, y, z). A virtual image of a scene may include volume rendering of color, physical properties, and depth. For example, again briefly referring to Figure 3 , the virtual image 216' may be constructed by projecting a ray 218 with a pose T from the virtual camera 210' into the virtual space 202' for each pixel [u, v] of the virtual image 216'. The radiance, physical properties, and volume density values ​​at each of a plurality of sampling points 220 (indexed as i) along the ray 218 may then be obtained by assigning the 3D positions of those sampling points to the radiance, physical properties, and volume density values. is input into equation (1) (also called the query MLP) to determine the color of the pixel [u, v] , physical properties and depth The volume rendering can then be performed by synthesizing the radiosity c, physical property s, and volume density value of the sample point i along the back-projected ray 218 of the pixel [u, v] according to the following equation To calculate:

[0055]

[0056] in is the sample i at depth d i The probability of ray termination along the ray at ; is the occupancy activation function; and is the distance between samples.

[0057] The geometry and appearance of the scene representation can then be optimized by minimizing the difference or loss between the rendered depth image 216' of the virtual scene 202' and the captured depth image of the scene 202 (e.g., captured by the camera 210). For example, such optimization can be based on sparsely sampled pixels from the captured depth image and corresponding pixel samples from the rendered depth image (e.g., volume rendering can be performed only on the sampled pixels). This can help reduce the resource burden and therefore increase the speed of optimizing the scene representation model. In some examples, this process can be performed for a set of W captured depth images. For example, these W captured depth images can be key frames from a video stream, such as key frames selected to include new information or new perspectives of the scene 202. However, it will be appreciated that in some examples, there may be only one captured image, i.e., W may be equal to 1. In addition, in some examples, this optimization process may also include optimization of the pose T of the virtual camera 210'. However, it will be appreciated that in some examples, the pose T of the virtual camera 210' may be known, estimated, or derived using different means, and may be taken as a given or fixed parameter in the optimization process.

[0058] Minimizing the loss between the captured image and the rendered image may include minimizing the selected number of rendered pixels s i For example, for M pixel samples, the photometric loss L p Can be used as the L1 norm between the rendered color value and the captured color value :

[0059]

[0060] Similarly, the geometry loss L g Can measure rendered depth values ​​vs. captured depth values The difference between the two, and the depth variance along the ray can be used As a normalization factor to mitigate the loss in uncertain areas (such as object boundaries):

[0061]

[0062] An optimizer (such as the ADAM optimizer) can then be applied to the weighted sum of the two losses and factor To adjust the importance of photometric losses:

[0063]

[0064] Thus, the obtained scene representation model 432 may represent the scene 202. As described above, the scene representation model 432 is configured to predict values ​​of physical properties of one or more of the objects. However, so far, in the above optimization process, the scene representation model 432 has not necessarily been targeted at physical properties. Therefore, the predicted physical properties may not be accurate. However, according to Figure 1 In steps 104 and 106 of the method, the accuracy of the physical property prediction can be improved by: obtaining a value of the physical property of at least one of the objects 204, the obtained value being derived from the physical contact of the robot 214 with the at least one object 204; and updating the scene representation model 432 based on the obtained value. Examples of deriving and obtaining values ​​of physical properties are described in more detail below. However, in order to maintain continuity with the above-mentioned optimization examples, an example of updating the scene representation model 432 based on the obtained value of the physical property of at least one of the objects 204 will now be described.

[0065] For example, the robot may physically contact the object 204 and derive therefrom the value of the physical property of the object 204. The captured depth image of the scene 202 shows that a portion of the object 204 may be marked with the obtained physical property value S. For example, one or more (hereinafter K) pixels of the captured depth image corresponding to the position in the scene 202 where the physical property value of the object is measured may be marked with the physical property value. As another example, K pixels of the captured depth image corresponding to the position of the object 204 where the physical property value is obtained may be marked with the physical property value S. As described above, in the virtual image 216', the physical property of the pixel [u, v] is The volume rendering of may be calculated according to equation (2). Minimizing the loss between the captured image and the rendered image may include minimizing the number of K pixels with physical property value labels Physical property loss L s .

[0066] For example, in some cases, the physical property value may be continuous, that is, the physical property value may be one of a continuous value. In these examples, the physical property loss L s For example, the rendered physical property values ​​of K pixels can be measured against the captured physical property values As another example, in some cases, the physical property value may be one of a set of C categories. For example, in the case where the physical property is mass, the value may be one of three categories: light, heavy, very heavy. In these examples, the volume rendering of the physical property It may first be activated by softmax For example, the loss of physical properties L s The activated rendered physical property values ​​of K pixels and d physical property value labels may be measured, In either case, the physical property loss L s can be given by the following equation:

[0067]

[0068] The scene representation model 432 may then be updated / optimized by minimizing the physical property loss (with respect to the parameters of the scene representation model). In some examples, the scene representation model 432 may be updated only by minimizing the physical property loss. In some examples, the scene representation model 432 may be optimized so that for K pixels with physical property value labels , jointly minimizing the physical property loss and one or both of the geometric shape loss and the photometric loss. For example, the scene representation model 432 can be optimized by minimizing the following objective function:

[0069]

[0070] in is a factor that adjusts the importance of photometric losses, and is a factor that adjusts the importance of the physical property loss. Therefore, the scene representation model 432 is updated based on the obtained value, and as a result, the scene representation model 432 can more accurately represent the physical properties of the objects 204 of the scene, and thus can more accurately represent the scene.

[0071] refer to Figure 5 , shows the flow between functional blocks according to an example. In some examples, by Figure 5 One or more of the steps or functions performed by one or more functional blocks may form Figure 1 part of the method.

[0072] Reference Figure 5 In some examples, as shown, the process begins with a pose estimation module 550, which is configured to estimate a pose T of the camera 210. In some examples, the pose of the camera 210 does not need to be estimated. For example, the pose of the camera 210 may be known, for example because the pose of the camera 210 is fixed or predetermined, or because the pose is derived from a known configuration of the joint 215 of the robotic arm 206 to which the camera 210 is fixed, for example. In these examples, the pose estimator 550 may simply take the pose as a known pose, or may provide the known pose directly to the virtual image renderer 552 (described in more detail below) without using the pose estimator 550. Nevertheless, in the example shown, the pose estimation module 550 estimates the pose T of the camera 210.

[0073] In some examples, as shown, the pose estimation module 550 may receive an initial estimate of the pose of the camera 210 from the initial estimator 548. For example, the initial estimator 548 may provide an initial estimate of the pose of the camera based at least in part on data indicating the configuration of the device 206 for positioning the camera 210. For example, the initial pose estimate may be derived based on a known configuration of the joint 215 of the mechanical arm 206 on which the camera 210 is fixed. As described above, in some examples, the pose estimator 550 may be removed, and the initial estimator 548 may provide the initial estimate of the pose of the camera 210 directly to the virtual image renderer 552. This may help allow efficient estimation of the camera pose. Using the pose estimation module 550 may help estimate the camera pose even when the camera pose is not necessarily known in advance, and improve flexibility. Alternatively or in addition, this may help fine-tune the camera pose estimate from the initial estimate.

[0074] In any case, in these examples, the pose of the camera 210, or an estimate of the pose, is provided to a virtual image renderer 552. The virtual image renderer 552 includes a volume renderer 552 and a scene representation model 432. The volume renderer 552 may use the provided pose of the camera 210 in the real space 202 (or an estimate of the pose) to determine the pose of the virtual camera 210' in the virtual space 202'. For example, this may be achieved by a suitable coordinate transformation from the real space 202 to the virtual space 202'. The virtual image renderer 552 renders a virtual image 216' of the scene representation model 432 based on the determined pose of the virtual camera 210'. Similar to what is described above with reference to equations (1) and (2), one or more portions of the virtual image 216' (e.g., one or more selected pixels s of the virtual image 216') are rendered as part of the scene representation model 432. i or ) can be used to respectively use the predicted depth value , the predicted depth value represents the depth of the portion of the scene representation from the virtual camera 210' to which the virtual image 216' is rendered. In addition, one or more portions of the virtual image 216' (e.g., one or more selected pixels s of the virtual image 216') i or ) can be used to predict the photometric values (e.g., color), the predicted photometric value representing the predicted photometric property of the portion of the scene representation under the virtual camera 210' to which the virtual image is rendered. In addition, one or more portions of the virtual image 216' (e.g., one or more selected pixels s of the virtual image 216') i or ) can use the corresponding predicted physical property values ​​of the corresponding objects represented by the corresponding parts Thus, a virtual image 216 ′ is generated that indicates predicted depths, predicted photometric values ​​(eg, colors), and predicted physical property values ​​of the objects 204 , 206 of the scene 202 .

[0075] The virtual image 216' is provided to the optimizer 560. The optimizer 560 is also provided with a captured depth image 558 of the scene 202 captured by the camera 210. The camera 210 may be a depth camera, such as an RGB-D camera. The optimizer 560 is configured to optimize the scene representation model 432 so as to minimize the loss between the captured image 558 and the virtual image 216'. As shown, in an example, where the pose of the camera 210 is estimated by the pose estimation module 550, the optimizer may be configured to jointly optimize the scene representation model 432 and the pose estimation module 550 by minimizing the loss between the captured image 558 and the virtual image 216'.

[0076] For example, one or more portions of captured image 558 (eg, one or more selected pixels s of captured image 558) may be selected to represent the image 558. i or ) can use the obtained depth values 558, the obtained depth value represents the depth of the portion of the scene 202 represented by the camera 210 that captured the image 558. In addition, one or more portions of the captured image 558 (e.g., one or more selected pixels s of the captured image 558) i or ) can be used to obtain photometric values ​​(such as color) , the photometric value represents a photometric property of a portion of scene 202 as represented by the camera that captured image 558.

[0077] Before obtaining the physical property values In the example of , the captured image 558 may not have been labeled with physical property values ​​derived from the physical contact of the robot 208 with the object 204. In this case, in some examples, the captured image 558 may not have been labeled with any physical property values. In this case, the optimizer 562 can optimize the scene representation model 432 (and in some examples, the pose estimation module 550) based on the depth and photometric value labels of the captured image 558 (e.g., as described above with reference to equations (1) to (5)).

[0078] In some examples, captured image 558 may be labeled with estimates of physical property values ​​of objects of the scene shown in captured image 558. For example, a pre-trained object detector (not shown) may be applied to captured image 558 to identify at least one object 204, and the estimate of the physical property value may be inferred from the identity of at least one object 204. For example, if object 204 is identified as a chair, then an estimate of the mass of object 204 may be provided as, for example, the average mass of a household chair. In these cases, one or more portions of captured image 558 (e.g., one or more selected pixels s of captured image 558) may be labeled with an estimate of the physical property value of the object of the scene shown in captured image 558. i or ) may each be labeled with an estimate of the value of the physical property of the object 204 represented by the portion. In these cases, the optimizer 560 may optimize the scene representation model 432 based on the estimated physical property value labels of the captured image 558 (and in some examples, also based on the pose estimation module 550), and in some examples, may also optimize the scene representation model based on the depth and / or photometric value labels of the captured image 558 (e.g., similar to what is described above for equations (6) and (7)). In this way, the scene representation model 432 is pre-trained by optimizing the scene representation model 432 so as to minimize the loss between the provided estimate of the value of the physical property of at least one object 204 of the scene 202 and the predicted value of the physical property of the at least one object. This may help provide a relatively accurate scene representation relatively quickly, for example, compared to starting with no information about the value of the physical property of any portion of the scene.

[0079] In examples where physical property values ​​derived from physical contact between the robot 208 and the object 204 have been obtained (e.g., as described in more detail below), one or more portions of the captured image 558 (e.g., one or more selected pixels of the captured image 558) may be ) can each utilize the obtained physical property value of the object 204 represented by the portion In these cases, optimizer 560 may optimize scene representation model 432 based on the obtained physical property value labels of captured image 558 (and in some examples, also based on pose estimation module 550), and in some examples, may also optimize the scene representation model based on depth and / or photometric value labels of captured image 558 (e.g., as described above with reference to equations (6) and (7)).

[0080] As mentioned above, Figure 1 Step 104 of the method involves obtaining a value of a physical property of at least one of the objects 204 , the obtained value being derived from physical contact of the robot 208 with the one object 204 .

[0081] In some examples, the method may include selecting at least one object 204 from among a plurality of objects 204, 206 in the one or more objects 204, 206. For example, the method may include determining which of the objects 204, 206 in the scene 202 the robot 208 is controlled to make physical contact with, and thus determining which of the objects 204, 206 obtains the value of the physical property. For example, the method may include: controlling the robot 208 to make physical contact with the selected object 204; and deriving the value of the physical property of the selected object 204 from the physical contact, thereby obtaining the value.

[0082] In some examples, the selection can be based on the uncertainty of the predicted values ​​of the physical property of each of the plurality of objects 204, 206. This can allow the robot to autonomously select the object 204 to make physical contact with, which can result in the greatest reduction in uncertainty in the scene representation model 432, and thus provide the greatest gain in accuracy and / or reliability of the scene representation model 432.

[0083] For example, when generating the virtual image 216', the uncertainty of the predicted physical property value may be calculated for each pixel of the virtual image 216'. For example, the uncertainty of the predicted value of the physical property may be given by one or more of softmax entropy, minimum confidence, and marginal sampling. For example, when the predicted value is a predicted classification or category among C classifications or categories, the softmax entropy may be defined as:

[0084]

[0085] This can be determined for each pixel of the virtual image 216'. As a result, an uncertainty distribution map 555 can be generated. Each pixel value of the uncertainty distribution map 555 indicates the uncertainty of the scene representation model 432 predicting the physical property value of the object 204 (or part of the object) shown in that pixel. The method may include selecting one or more pixels based on the uncertainty value. For example, it can be selected from pixels with uncertainty values ​​within the first X% of all uncertainty values ​​of the distribution map 555. For example, it can be selected from a sample (e.g., a random sample) of pixels from the uncertainty distribution map that are within the first X% of all uncertainty values ​​of the distribution map 555. For example, X% can be 0.5%. For example, the selection can be one or more pixels with the highest uncertainty value. In any case, one or more selected pixels can be projected into the virtual space 202' (e.g., using the posture of the virtual camera 210') to determine the position in the virtual space 202' to which the one or more pixels correspond. Thus, it is possible to determine the object 204' or a part of the object 204' in the virtual space 202' at this position. The object 204 that the robot is to physically contact can then be determined to be the object 204 corresponding to the determined virtual object 204'. For example, the robot 208 can be controlled to physically contact the object 204 corresponding to the determined virtual object 204', and the value of the physical property of the object 204 is derived from the physical contact, thereby obtaining the value. The obtained value can be used to mark the associated pixels of the captured image 558, for example as described above, and the scene representation model 432 can be updated / optimized, for example as described above. As a result, the uncertainty of the prediction of the physical property value of the object 204 by the scene representation model 432 can be reduced. In addition, as a result, a new uncertainty distribution map 555 reflecting the reduction in uncertainty of the object 204 can be generated. In some examples, this process can be repeated until the uncertainty values ​​of all pixels of the uncertainty distribution map 555 (or, for example, all pixels corresponding to the positions with which the robot 208 can interact kinematically, see below) are below a certain threshold. For example, at this time, it can be determined that the scene representation model 202' sufficiently accurately reflects the scene 202.

[0086] In some examples, the selection of the object 204 to contact is based on further criteria. For example, in some examples, selecting at least one object 204 may include determining a kinematic cost and / or kinematic feasibility of physical contact of the robot 208 with each of the plurality of objects 204, 206; and the at least one object 204 may be selected based on the kinematic feasibility determined for each of the plurality of objects 204, 206. For example, the kinematic feasibility may indicate whether the robot 208 is able to move and / or configure itself so as to physically contact the object 204, 206 (or to physically contact in a manner that allows the desired physical property value to be obtained), or to what extent it is able to move and / or configure itself. The kinematic cost may indicate the cost (e.g., in terms of energy or time) of the robot 208 moving and / or configuring itself so as to contact the object 204, 206. For example, as described above, in some examples, the selection of one or more pixels from the uncertainty distribution map 555 may be from a random sample of pixels from the uncertainty distribution map, the uncertainty value of the pixel being within the top 0.5% of all uncertainty values ​​of the distribution map 555. The method may include iterating through the random sample from the highest uncertainty value to the lowest uncertainty value, checking the kinematic feasibility of each pixel until a feasible pixel is found. The object selected by the control robot 208 can then be determined based on this feasible pixel, such as described above. For example, a feasible pixel can be a pixel corresponding to a location in 3D space that the robot 208 can reach or physically contact or physically contact in a manner that allows the desired physical property value to be obtained.

[0087] In some examples, the method may include, in response to determining that physical contact between the robot 208 and a given object in the plurality of objects 204, 206 is not kinematically feasible, adding the given object 204, 206 to a selection mask to prevent the given object from being selected in further selections of the objects 204, 206 for which the value of the physical property is to be obtained. For example, as described above, the method may include traversing a random sample of pixels from the uncertainty distribution map 555 from the highest uncertainty value to the lowest uncertainty value, checking the kinematic feasibility of each pixel until a feasible pixel is found. Pixels determined to be kinematically infeasible during this process may be added to a mask applied to the uncertainty distribution map 555 to prevent those pixels from being selected in further pixel selections.

[0088] In some examples, updating scene representation model 432 based on the obtained values ​​does not necessarily involve comparing the captured image and the virtual image of the scene, as per some of the examples described above. For example, in some examples, scene representation model 432 can be optimized by minimizing the loss between the obtained values ​​and the values ​​of the physical properties directly predicted by scene representation model 432.

[0089] For example, a three-dimensional position (x, y, z) at which a physical property value is to be obtained may be determined. For example, this position may correspond to an object 204 for which a physical property is to be obtained. For example, the object 204 and / or the position may be determined based on the entropy distribution map 555 as described above. The robot 208 may accordingly physically contact the object 24 at that position and derive the physical property value of the object 204 from that physical contact. For example, as described in more detail below, the robot 208 may place a spectrometer probe at that position. The spectrometer probe may output an N-dimensional vector representing the spectral characteristics of the material of the object 204 at that position. The measured physical property value (e.g., N-dimensional vector) may be input into a pre-trained classifier that has been trained to output a category or classification of physical property values ​​among a set of classifications C. For example, the N-dimensional vector output by the spectrometer may be input into a pre-trained classifier that may output a material type classification or classification of the N-dimensional value / vector. The output class or classification may be one-hot encoded among other classes among a set of C classes (e.g., the output classification may be encoded as a vector having C elements, where the element corresponding to the output classification is assigned a value of 1 and all other elements are assigned a value of 0). This may serve as ground truth for the classification of the physical property of the object 204 at the 3D location and may be used as the obtained value of the physical property of the object 204.

[0090] In these examples, predicted values ​​of physical properties of objects can be provided by inputting corresponding 3D positions in virtual space (x', y', z') into scene representation model 432; and obtaining direct predictions of physical property values ​​at that position from physical property head 438. Physical property head 438 can be configured to output predictions of physical property values ​​as predicted classifications or categories among a set of classifications C (e.g., material types). For example, similar to the above example, physical property head 428 can be configured to output predicted categories in the form of one-hot encodings among classifications C (e.g., material types).

[0091] In these examples, the physical property loss may, for example, measure the cross entropy loss between the predicted physical property values ​​and the obtained physical property values. For example, the physical property loss may measure the cross entropy loss between the one-hot encoded predicted classification output from the scene representation model 432 and the one-hot encoded ground truth classification provided via a pre-trained classifier. In this case, updating the scene representation model 432 may include optimizing the scene representation model 432 (e.g., adjusting its parameters) so as to minimize this physical property loss. Other examples of updating the scene representation model 432 based on the obtained values ​​may be used.

[0092] As described above, the obtained values ​​(based on which the scene representation model 432 is updated) are derived from the physical contact of the robot 208 with the at least one object 204. Therefore, in some examples, and as described above, the method may include controlling the robot 208 to physically contact the at least one object 204 and / or deriving the obtained values ​​from the physical contact.

[0093] In some examples, the physical contact of the robot 208 with the at least one object 204 of the scene 202 may include the robot 208 physically moving or attempting to physically move the at least one object 204 of the scene 202, and the obtained value may be derived from the physical movement or attempt. The robot 208 interacting with the scene 202 may physically move the object (or at least attempt to move the object), and this may not necessarily require a specific measurement probe, which in turn may allow the physical properties to be determined in a cost-effective manner.

[0094] For example, the physical contact may include one or more of a top-down poking of the at least one object 204 , a lateral pushing of the at least one object 204 , and a lifting of the at least one object 204 .

[0095] For example, the value of the physical property may indicate one or more of the flexibility or hardness of the at least one object 204, the coefficient of friction of the at least one object 204, and the mass of the at least one object 204. These properties may be determined by the robot 208 moving the object (or attempting to move the object), and thus, may not require a specialized measurement probe to contact the object, which may be cost-effective.

[0096] For example, in one example, the physical contact may include poking from top to bottom. For example, this may include poking or pushing a finger or other portion of the robotic arm 208 vertically downward onto the object 204. The force sensor may measure the force applied by the robotic arm 208 to the object 204. For example, the force sensor may be included on the tip of the finger or other portion of the robotic arm 208 that is in contact with the object 204. As another example, one or more force sensors (e.g., torque sensors) may be included in one or more of the joints 215 of the robotic arm, and the output of these sensors may be converted into a force applied to the object 204. The force applied by the robotic arm 208 to the object 204 may be measured in conjunction with the distance traveled by the finger or other portion of the robotic arm when contacting the object 204. This may allow a rough estimate or determination of the hardness of the object 204 or the material of the object 204 to be made. For example, stiffness may be defined as k = F / δ, where F is the top-down force applied to object 204, and δ is the displacement produced by the force in the direction of the force (i.e., the distance traveled by a finger when contacting object 204 when force F is applied). The flexibility of object 204 or the material of object 204 may be roughly estimated as the inverse of stiffness. From the physical property of stiffness / flexibility, other physical properties may be derived, such as the material type or nature of the object (e.g., soft decorative items versus hard table or floor surfaces).

[0097] As another example, the physical contact may include a lateral push on the object 204. For example, this may include pushing a finger or other part of the robotic arm laterally or horizontally against the object 204. Similar to the above, one or more force sensors may measure the force applied by the robotic arm 208 to the object 204 during the lateral push. To cause the object to move, the force required for the lateral push may be proportional to the coefficient of friction between the object 204 and the surface on which the object 204 is placed, and also proportional to the normal force of the object 204 (e.g., in the case of a flat horizontal floor, the normal force is proportional to the mass of the object 204). Therefore, if the mass of the object 204 is known or estimated, then an estimate of the coefficient of friction can be obtained from the force of the lateral push. Conversely, if the coefficient of friction is known or estimated, then an estimate of the mass of the object 214 can be obtained from the force of the lateral push. In any case, the force of the lateral push required to move the object may indicate a physical property of the movability of the object 204. For example, if a high lateral thrust is required to move the object 204, the object's mobility may be determined to be low, while if a low lateral thrust is required to move the object 204, the object's mobility may be determined to be high. Similarly, if a high lateral thrust does not result in the movement of the object 204, the object may be determined to be immovable, such as in the case where the object is fixed to the floor. As another example, one or more audio sensors (such as microphones) may measure the volume or intensity or other characteristics of the sound that occurs during the lateral push of the object 204. The measured characteristics of the sensed sound may be used as a representative of the coefficient of friction between the object 204 and the surface, and / or, for example, may be used as a measure of the extent to which the object 204 may scratch the surface when moving. For example, a high amplitude sound may indicate a high coefficient of friction and / or a high degree of scratching may occur when the object is moved laterally across the surface on which the object 204 is supported. In some examples, the lateral push of the object may be performed at or below the estimated center of gravity of the object 204. This may help avoid the collapse of the object.

[0098] As another example, the physical contact may include lifting of the object 204. For example, in some examples, the lifting may be a small lift, i.e., a vertical upward force is applied to the object 204, but the object 204 does not break contact with the ground or only breaks away to a small extent. This may allow the lifting to be performed, for example, without the robot 206 having to stably grasp the object 204. For example, this may include using a finger or other part of the robotic arm 208 to apply a vertical upward force on the object 204. Similar to the above, one or more force sensors may measure the force applied by the robotic arm 208 to the object 204 during the small lift. The measured force may provide an estimate of the mass of the object 204 (or an estimate of a lower limit of the mass of the object 204). In some examples, the small lift may be performed at multiple locations on the object, and this may be used to derive an estimate of the center of mass of the object. For example, the location of the maximum force required to lift the object 204 may be used to derive the center of mass of the object 204, or may be used as a rough estimate of the center of mass of the object 204. In some examples, lifting can be used to estimate the center of mass of an object, and as described above, this estimate of the center of mass can be used to determine where on the object 204 to perform a lateral push. In some examples, physical contact can include grasping or grabbing the object 204 and performing the lift. For example, with a stable grasp or grab, the force required to vertically lift the object 204 can provide an estimate of the mass of the object 204. For example, in each case of stabbing, pushing, or lifting, the distance that the robot moves upon contact may be relatively short, such as in the range of a few millimeters, which is sufficient to sample the physical properties of the object, but not enough to substantially damage or change the position of the object in the scene. Other physical contacts and derived physical properties are also possible.

[0099] Updating the scene representation model 432 based on the mass or estimate of the mass of the object 204 can be helpful in helping the robot 208 perform a task. For example, a task can be defined to sort empty cups. However, a cup filled with water may look exactly the same as an empty cup. But the mass of a full cup is greater than the mass of an empty cup. Therefore, the robot 208 can physically contact the cup (e.g., by performing a lift or a small lateral push) to derive an estimate of the mass of the cup, and update the scene representation model 432 based on the derived mass of the cup. The robot can then perform the defined task based on the updated scene representation model 432. For example, if the mass of the cup is low, the robot 208 can infer that the cup is empty and therefore sort the cup, but if the mass of the cup is high, the robot 208 can infer that the cup is full and therefore does not sort the cup. As another example, similarly, a task can be defined to sort boxes that look the same but have different masses by mass. Thus, the robot may push or lift each box laterally to determine an estimate of the mass, update the scene representation model 432 based on the mass, and perform the task based on the updated scene representation model 432 .

[0100] In some examples, the physical contact of the robot with the at least one object 204 may include physical contact of the measurement probe 212 of the robot 208 with the at least one object 204 , and the obtained value may be derived based on the output of the measurement probe when contacting the at least one object 204 .

[0101] For example, the measurement probe 212 may include a single-pixel multi-band spectrometer. This may allow for dense spectral rendering of the object 204 (or objects constituting the scene 202). For example, the single-pixel multi-band spectrometer may measure the spectral fingerprint of the object 204. In some examples, this spectral fingerprint may be compared with the spectral fingerprints of known materials in a database. Thus, by matching the obtained spectral fingerprint of the object 204 with the spectral fingerprints of known materials or material types, the material or material type of the object 204 may be determined or estimated. In some examples, as described above, the output of the spectrometer may be an N-dimensional vector (where N>1), for example, indicating the spectral fingerprint of the object 204. This N-dimensional vector may be input to a pre-trained classifier (e.g., implemented by a trained neural network or other machine learning model), which maps the N-dimensional input vector to a material or material type. Thus, the material or material type of the object 204 may be determined or at least estimated. Other measurements performed by the measurement device may be used, and may be combined, for example, to improve the accuracy or reliability of the determination of the material or material type of the object 204. For example, a thermal conductivity sensor may measure the thermal conductivity of the object 204 (or its material). As another example, a porosity sensor may measure the porosity of the material. The material or material type of the object may be determined or estimated from any one or more of the derived physical properties. This may be useful for tasks such as sorting the garbage objects 204 into recyclable material objects and non-recyclable material objects and, for example, the type of recyclable material contained in the object within the recyclable material object. As another example, this may be useful for tasks such as washing objects into wool objects and synthetic objects.

[0102] In some examples, the scene representation model 432 may be configured to predict the value of each of the multiple physical properties of the objects 204, 208 of the scene. For example, the physical property head 438 may be configured to output multiple values ​​corresponding to the multiple physical properties. Thus, multiple values ​​of the multiple physical properties of one or more objects 204 of the scene may be derived from the physical contact of the robot 208 with the one or more objects 204, and the scene representation model 432 may be updated based on these multiple values. This may allow a more complete and / or accurate representation of the scene 202 to be provided.

[0103] As described above, in some examples, the method may include controlling the robot 208 (or another robotic device, not shown) to perform a task based on the updated scene representation model 432.

[0104] In some examples, a collision mesh constructed from a depth rendering of scene representation model 432 may be used to avoid unintended collisions of robot 208 with scene 202 .

[0105] As described above, in some examples, the optimization process may be performed for a set of W captured depth images (where W ≥ 1). For example, these W captured depth images may be key frames from a video stream, such as key frames selected to include new information or new perspectives of the scene 202. As described above, in some examples, there may be only one captured image, that is, W may be equal to 1. In some examples, the physical contact of the robot 208 with the scene 202 may cause the position of one or more of the objects 204, 206 to change, and thus the optimization based on the past key frames may no longer be accurate. However, in such examples, the optimization may be performed based only on the latest key frames. In this way, as the objects move, the scene representation model 432 may be updated accordingly. Moreover, this may allow the robot to perform continuous exploration tasks, such as when objects are removed from the scene 202. For example, if the robot 208 is tasked with dismantling a pile of unknown objects, then as each object is removed from the pile and from the scene, the scene representation model 432 may be updated based on the newly captured images to continue the task.

[0106] refer to Figure 6 , shows a device 600 according to an example. The device 600 may be configured to perform the above reference Figures 1 to 5 In some examples, the device 600 may be a robot or may be part of a robot, such as described above with reference to Figures 1 to 5 The robot 208 of any of the examples described. In some examples, the device 600 may include a computer configured to perform the above reference Figures 1 to 5 A method of describing any of the examples. For example, the computer may be a remote server, such as a server that is remote from the robot 208 but is communicatively connected to the robot 208, such as by wire or wireless means. In the illustrated example, the device 600 includes a processor 602, a memory 604, an input interface 604, and an output interface 606. The input interface 604 may be configured to, for example, obtain a scene representation model 432 (which may be stored in the memory 604) and / or, for example, obtain a value of a physical property of at least one object. The output interface 606 may be used, for example, to output control instructions to cause the robot 208 to physically contact the at least one object 204 and / or to cause the robot 208 to perform a task based on, for example, an updated scene representation model 438. The memory 604 may store a computer program, the computer program comprising a set of instructions that, when executed by the processor 602, cause the processor 602 to perform the tasks described above with reference to the preceding claims. Figures 1 to 5 In some examples, a computer program may be stored on a non-transitory computer-readable medium. In some examples, a non-transitory computer-readable medium having instructions stored thereon may be provided, which, when executed by a computer, causes the computer to perform the operations described above. Figures 1 to 5 A method of any of the described examples.

[0107] The above examples should be understood as illustrative examples. It should be understood that any feature described with respect to any example can be used alone or in combination with other features described, and can also be used in combination with one or more features of any other example, or in combination with any combination of any other example. In addition, equivalents and modifications not described above can also be adopted within the scope of the appended claims.

Claims

1. A computer-implemented method for updating a scene representation model, the method comprising: obtaining a scene representation model representing a scene having one or more objects, the scene representation model being configured to predict a value of a physical property of one or more of the objects; obtaining a value of the physical property of at least one of the objects, the obtained value being derived from physical contact of the robot with the at least one object; and The scene representation model is updated based on the obtained value.

2. The method according to claim 1, wherein: The physical contact of the robot with the at least one object of the scene comprises the robot physically moving or attempting to physically move the at least one object of the scene, wherein the obtained value is derived from the physical movement or the attempt.

3. The method according to claim 1 or 2, wherein: The physical contact includes one or more of a top-down poking of the at least one object, a lateral pushing of the at least one object, and a lifting of the at least one object.

4. A method according to any one of the preceding claims, wherein: The value of the physical property indicates one or more of flexibility or hardness of the at least one object, a coefficient of friction of the at least one object, and a mass of the at least one object.

5. A method according to any one of the preceding claims, wherein: The physical contact of the robot with the at least one object comprises physical contact of a measurement probe of the robot with the at least one object, wherein the obtained value is derived based on an output of the measurement probe when contacting the at least one object.

6. The method according to any one of the preceding claims, wherein the method comprises: selecting the at least one object from among the one or more objects based on an uncertainty in the predicted value of the physical property of each of the plurality of objects; controlling the robot to physically contact the selected object; and The value of the physical property of the selected object is derived from the physical contact, thereby obtaining the value.

7. The method according to claim 6, wherein: Selecting the at least one object comprises: determining a kinematic cost and / or a kinematic feasibility of the physical contact of the robot with each of the plurality of objects; and Therein, the at least one object is additionally selected based on the determined kinematic feasibility for each of the plurality of objects.

8. The method according to claim 7, wherein: The method comprises: In response to determining that the physical contact of the robot with a given object of the plurality of objects is not kinematically feasible, adding the given object to a selection mask to prevent the given object from being selected in further selections of objects for which to obtain values ​​of the physical property.

9. A method according to any one of the preceding claims, wherein: The scene representation model provides an implicit scene representation of the scene.

10. The method according to claim 9, wherein: The scene representation model comprises a multi-layer perceptron having a semantic head, wherein, for a coordinate of the scene representation input to the multi-layer perceptron, the semantic head outputs the prediction of the value of the physical property at the coordinate.

11. The method according to claim 10, wherein: The multilayer perceptron further has a volume density head and / or a photometric head, wherein, for the coordinate of the scene representation input into the multilayer perceptron, the volume density head outputs a prediction of the volume density at the coordinate and / or the photometric head outputs a prediction of the photometric value at the coordinate.

12. A method according to any one of the preceding claims, wherein: Updating the scene representation model includes: The scene representation model is optimized so as to minimize a loss between the obtained value and a predicted value of the physical property of the at least one object.

13. The method according to claim 12, wherein: The obtained value and the predicted value each represent one of a plurality of discrete values, and wherein optimizing the scenario representation model comprises minimizing a cross entropy loss between the predicted value and the measured value; or wherein the obtained value and the predicted value each represent one of a plurality of continuous values, and wherein optimizing the scenario representation model comprises minimizing a mean square error loss between the predicted value and the measured value.

14. A method according to any one of the preceding claims, wherein: Updating the scene representation includes: marking a portion of a captured image of the scene with the obtained value of the object represented by the portion; obtaining a virtual image of the scene rendered from the scene representation model, one or more portions of the virtual image being labeled with respective predicted values ​​of respective objects represented by respective portions; determining a loss between the obtained values ​​of the marked portions of the captured image and the predicted values ​​of corresponding portions of the virtual image; and The scene representation model is optimized to minimize the loss.

15. The method according to claim 14, wherein: One or more portions of the captured image are each labeled with an obtained depth value indicating a depth of a portion of the scene represented by the portion from a camera that captured the image; one or more portions of the virtual image are each labeled with a predicted depth value indicating a depth of a portion of the scene representation represented by the portion from a virtual camera from which the virtual image is rendered; and Wherein, updating the scene representation model comprises: determining a geometric loss between the obtained depth values ​​of the one or more portions of the captured image and the predicted depth values ​​of one or more corresponding portions of the virtual image; and The scene representation model is optimized to minimize the geometry loss.

16. The method according to claim 14 or 15, wherein: one or more portions of the captured image are each labeled with an obtained photometric value indicating a photometric property of a portion of the scene represented by the portion under a camera that captured the image; one or more portions of the virtual image are each labeled with a predicted photometric value indicating a predicted photometric property of a portion of the scene representation represented by the portion under a virtual camera from which the virtual image is rendered; and Wherein, updating the scene representation model comprises: determining a photometric loss between the obtained photometric values ​​of the one or more portions of the captured image and the predicted photometric values ​​of one or more corresponding portions of the virtual image; and The scene representation model is optimized so as to minimize the photometric loss.

17. The method according to claim 15 or 16, wherein: The one or more portions of the captured image are sampled from the captured image, and predicted depth values ​​and / or predicted photometry values ​​are predicted for corresponding samples of the one or more portions of the virtual image based on output from the scene representation model.

18. The method according to any one of claims 14 to 17, wherein: The method comprises: estimating a pose of a camera that captured the image when the captured image was captured; and Wherein, the virtual image is rendered under a virtual camera with an estimated pose.

19. The method according to claim 18, wherein: The pose of the camera is estimated based at least in part on data indicative of a configuration of a device for positioning the camera.

20. The method according to claim 18 or 19, wherein: The pose of the camera is estimated based at least in part on an output of a pose estimation module configured to estimate the pose of the camera, wherein optimizing the scene representation model comprises jointly optimizing the pose estimation module and the scene representation model to minimize the loss.

21. A method according to any one of the preceding claims, wherein: The obtained scene representation model is pre-trained by optimizing the scene representation model so as to minimize a loss between a provided estimate of a value of the physical property of at least one object of the scene and a predicted value of the physical property of the at least one object.

22. The method according to claim 21, wherein: The estimate is provided by: applying a pre-trained object detector to the captured image to identify the at least one object; and inferring the estimate from the identity of the at least one object.

23. A method according to any one of the preceding claims, wherein: The method comprises: The robot is controlled to perform a task based on the updated scene representation model.

24. An apparatus configured to perform the method according to any one of claims 1 to 23.

25. A computer program comprising a set of instructions which, when executed by a computer, causes the computer to perform the method according to any one of claims 1 to 23.

Images