Apparatus and method for controlling a robot

Abstract

To provide a device and a method for controlling a robot.SOLUTION: A method for controlling a robot includes: performing demonstrations (301); supplying descriptor images for the demonstrations from a point of view of the robot with respect to an object (302); selecting a set of multiple feature points that are selected by searching for an optimum condition of an objective function which rewards selected feature points being visible in the descriptor images (303); training a robot control model using the demonstrations (304); obtaining the descriptor image of the object (306); locating the selected set of multiple feature points in the descriptor image of the object (307); determining Euclidean coordinates of the located feature points (308); estimating a pose from the determined Euclidean coordinates (309); and controlling the robot to handle the object by means of the robot control model with the estimated pose (310), thereby controlling the robot for a control scene including the object (305).SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] Prior Art The present disclosure relates to an apparatus and method for controlling a robot.

Background Art

[0002] Through an approach called learning from demonstration (LfD), the skills of a robot can be programmed, where the nominal plan of the skill is learned by the robot based on the demonstration. The main idea of LfD is to parameterize a Gaussian distribution by a camera monitoring the robot's workspace and the pose of the target object to be processed by the robot. For this purpose, and also to utilize the resulting robot control model, it is necessary to perform an estimation of the pose of an object in a given scene. One possibility is pose estimation techniques based on RGB images. However, these can sometimes lack reliability.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Therefore, an approach that enables reliable pose estimation in the context of LfD is desired.

Means for Solving the Problems

[0004] Disclosure of the Invention According to various embodiments, a method for controlling a robot is provided, the method comprising the steps of: performing a demonstration, each demonstration of actually processing an object; for each demonstration, supplying at least one descriptor image from the robot's viewpoint of the object, wherein the descriptor image specifies feature points relating to the location on the object; selecting a set of multiple feature points from the specified feature points, wherein these feature points are selected by searching for the optimal condition of an objective function that rewards the selected feature points that are visible in the descriptor image; and demonstrating A robot control model is trained using a robot control model configured to output control information depending on the input object pose; and a robot is controlled with respect to a control scene containing an object, the robot being controlled by obtaining a descriptor image of the object from the robot's viewpoint, placing a selected set of feature points in the descriptor image of the object, obtaining the Euclidean coordinates of the placed feature points with respect to the control scene, estimating the pose from the obtained Euclidean coordinates, the estimated pose being supplied to the robot control model as input, and the robot being controlled using the robot control model to process the object.

[0005] Thus, automatic selection of feature points used for pose estimation of a given object is performed such that the selected feature points are clearly visible (for example, in most of the descriptor image). Therefore, the selection of feature points can be considered to be based on the following assumption: that the features observed during the demonstration (for example, images taken to record the demonstration) are clearly visible in this sense, and that they are usually found in scenes where individual manipulative skills should be performed by the robot.

[0006] The robot's viewpoint could be, for example, the viewpoint of a camera mounted on the robot, or, for example, the viewpoint of the robot's end effector.

[0007] Therefore, the method described above makes it possible to accurately and reliably estimate the orientation of a single object (for example, its 6D orientation).

[0008] Various examples are given below.

[0009] Example 1 is a method for controlling the robot described above.

[0010] Example 2 is a method of the method of Example 1, wherein the objective function further rewards one or more of the following: the selected feature points are spaced apart in descriptor space; the locations on the object corresponding to the selected feature points are spaced apart in Euclidean space; and the detection error for the selected feature points relative to the object is small.

[0011] These conditions guarantee that feature points can be reliably detected with respect to an object, and that the object's orientation can be reliably determined when those feature points are detected.

[0012] Example 3 is a method according to Example 1 or 2, further comprising the steps of: matching a plane to an object; selecting feature points such that the feature points define a coordinate frame on this plane; and estimating the orientation from the obtained Euclidean coordinates of the placed feature points and information about the orientation of the matched plane.

[0013] This ensures reliable pose estimation even if the object is flat and / or has little variation in one spatial direction (i.e., it is not a "real" 3D object).

[0014] Example 4 is a method according to Example 3, wherein the step of estimating the orientation from the obtained Euclidean coordinates includes the step of projecting the Euclidean coordinates of the placed feature points onto a matched hyperplane.

[0015] Therefore, even if a feature point is detected in a way that its Euclidean coordinates do not lie on the plane, a planar coordinate frame (within this plane) can still be derived.

[0016] Example 5 is a method according to the method of Example 3 or 4, wherein the plane is matched to an object, feature points are selected so as to define one coordinate frame on the plane, and the orientation is estimated from the obtained Euclidean coordinates of the placed feature points and information about the orientation of the plane, provided that the change of the object in one spatial direction is below a predetermined threshold.

[0017] Thus, a highly reliable pose determination scheme can be trained for both real and non-real 3D objects. In particular, since (hyper)planar fitting is only applied when necessary, pose detection for real 3D objects remains efficient.

[0018] Example 6 is a method according to any one of Examples 1 to 5, which includes the step of finding a coordinate frame derivation rule from the Euclidean coordinates of selected feature points, and the step of estimating the pose from the obtained Euclidean coordinates includes the step of applying the derivation rule to the selected feature points, the derivation rule being obtained by searching for the minimum dependency of the coordinate frame from noise in the Euclidean coordinates.

[0019] This ensures reliable pose detection, even in cases of feature point detection errors.

[0020] Example 7 is a method including, in the method described in any one of Examples 1 to 6, the step of training a robot control model using a demonstration, which includes, for each demonstration, the step of placing a selected set of a plurality of feature points within a descriptor image of an object in the demonstration, the step of obtaining Euclidean coordinates of the feature points placed for the demonstration, and the step of estimating a pose from the Euclidean coordinates obtained for the demonstration.

[0021] What this means is that the selected feature points can be used for pose estimation for the training of the robot control model itself.

[0022] Example 8 is a method including, in the method described in any one of Examples 1 to 7, the step of obtaining a descriptor image of an object from a camera image of the object by a Dense Object Net.

[0023] A Dense Object Net (DON) can be trained to uniquely identify points on a certain object. Even if the configurations of the objects are different from each other, such points can be reliably tracked.

[0024] Example nine is a robot controller configured to implement the method described in any one of Examples 1 to 8.

[0025] Example 10 is a computer program including instructions for causing the computer to implement the method described in any one of Examples 1 to 8 when executed by the computer. 1]

[0026] Example 11 is a computer-readable medium including instructions for causing the computer to implement the method described in any one of Examples 1 to 8 when executed by the computer.

[0027] In the drawings, similar reference numerals generally refer to the same parts across different drawings. The drawings are not necessarily to actual size and proportions, but rather generally focus on illustrating the principles of the invention. In the following description, various aspects will be described with reference to the following drawings. [Brief explanation of the drawing]

[0028] [Figure 1] A diagram showing a robot. [Figure 2] This diagram illustrates a method for selecting feature points, learning a coordinate frame for objects in a training dataset that change very little in one direction, and then applying that coordinate frame during deployment. [Figure 3] This diagram shows a flowchart illustrating how to control a robot. [Modes for carrying out the invention]

[0029] The following detailed description is accompanied by the accompanying drawings, which illustrate specific details and embodiments of the present disclosure that enable the implementation of the present invention. Other embodiments may be used, and structural, logical, or electrical modifications may be made without departing from the scope of the present invention. Since some embodiments of the present disclosure may be combined with one or more other embodiments of the present disclosure to form new embodiments, the various embodiments of the present disclosure are not necessarily mutually exclusive.

[0030] The following describes various embodiments in more detail.

[0031] Figure 1 shows robot 100.

[0032] Robot 100 includes a robotic arm 101, which includes, for example, an industrial robotic arm for processing or assembling a workpiece (or one or more other objects). Robotic arm 101 includes manipulators 102, 103, 104 and a base (or support) 105 that supports these manipulators 102, 103, 104. The term “manipulator” refers to a movable member of robotic arm 101, which, by acting upon this movable member, enables physical interaction with the surroundings to perform, for example, a particular task. For control, robot 100 includes a (robot) controller 106, which is configured to perform interactions with the surroundings according to a control program. The last member of the manipulators 102, 103, 104 (the one furthest from the support 105), 104, is also referred to as an end effector 104, which may include one or more tools, such as a welding torch, a gripping device, or a painting device.

[0033] Other manipulators 102 and 103 (located relatively close to the support 105), together with the end effector 104, can form a positioning device such that a robotic arm 101 with this end effector 104 is provided at its end. The robotic arm 101 is a mechanical arm capable of providing similar functionality to a human arm (possibly having a tool at its end).

[0034] The robot arm 101 may include joint members 107, 108, and 109 that interconnect the manipulators 102, 103, and 104 with each other and with the support 105. The joint members 107, 108, and 109 may have one or more joints, each of which can provide rotational motion (i.e., rotational motion) and / or translational motion (i.e., displacement) relative to the associated manipulators. The motion of the manipulators 102, 103, and 104 can be initiated using actuators controlled by the controller 106.

[0035] The term "actuator" can be understood as a component adapted to act on a mechanism or process in response to being driven. An actuator can produce mechanical motion by carrying out a command (so-called activation) issued by the controller 106. An actuator can be configured, for example, as an electromechanical converter to convert electrical energy into mechanical energy in response to being driven.

[0036] The term "controller" can be understood as any type of logic implementation entity, which may include, for example, a circuit and / or processor capable of executing software, firmware, or a combination thereof stored in a storage medium, and which, for example, in this embodiment, can issue commands to actuators. For example, a controller can be configured by program code (e.g., software) to control the operation of a system, in this embodiment, a robot.

[0037] In this embodiment, the controller 106 includes one or more processors 110 and a memory 111 for storing code and data, and the processors 110 control the robot arm 101 based on this code and data. In various embodiments, the controller 106 controls the robot arm 101 based on a statistical model 112 stored in the memory 111.

[0038] Robot 100 can learn to perform a specific task or collaborate with a human partner by utilizing an approach called demonstration-based learning (LfD). The human demonstration can be encoded by a probabilistic model (also called a statistical model) that represents a nominal plan for the task for the robot. Then, the controller 106 can use the statistical model, also called a robot trajectory model, to generate the desired robot motion, in some cases as a function of the states of both the human partner and the surrounding environment.

[0039] The basic idea behind LfD is to fit a specified skill model, such as GMM, to a handful of demonstrations. Suppose there are M demonstrations, each of which is a dataset of all N = ΣmTm observations.

number

number

number

number

number

number

[0040] Unlike standard GMMs, the aforementioned mixture model cannot be trained independently frame by frame. In fact, the mixing coefficient π k This is shared by all frames, and the k-th component in frame p must be mapped to the corresponding k-th component in the global frame. Expectation maximization (EM) is a well-established method for training such a model.

[0041] Once learned, TP-GMM can be used during execution to reproduce the trajectory of the learned skills.

[0042] Hidden semi-Markov models (HSMMs) extend standard hidden Markov models (HMMs) by embedding time information into the underlying stochastic process. In other words, while in HMMs the underlying hidden process is considered Markov, meaning the probability of transitioning to the next state depends only on the current state, in HSMMs the state process is considered semi-Markov. This means that the transition to the next state depends not only on the current state but also on the time elapsed since entering that state. These parameters can be combined with TP-GMMs and applied to robot skill coding for learning spatiotemporal features of demonstrations, resulting in a task-parameterized HSMM (TP-HSMM) model.

[0043] To apply such a robot control model, it is necessary to estimate the pose of the object 113 to be processed. This can be related to the initial pose of the object 113 to be processed so that the parameters of the TP-GMM can be set appropriately. However, there are also cases where the pose must be estimated for observation. For example, a demonstration is recorded by taking a sequence of multiple images, and the observation corresponds to (or includes) the pose of the object being processed (or part of the robot) extracted from the images. According to various embodiments, an approach is provided for estimating the pose at selected feature points, which allows for the initial estimation of the pose (e.g., 6D pose) of a particular object in a given control scenario (scene), for example, if an RGBD (RGB + depth) image is available during deployment. This pose is then supplied to the TP-HSMM, and the desired trajectory for this control scenario is calculated. The selection of these key feature points is performed before training the TP-HSMM. Once this selection is made, the 6D object pose can then be derived for each demonstration. These 6D poses are used during TP-HSMM training to calculate task parameters within the object frame.

[0044] Therefore, according to various embodiments, demonstration-based learning is combined with a pose estimation approach based on mapping images to descriptor images. An example of a model that can be trained to perform such mapping is a high-density object network.

[0045] A high-density object network provides a means for tracking object-specific features. This network can be trained to be either object-specific or object-class-specific. Thus, a method is provided for detecting object-class-specific features across a wide range of object classes using a single neural network.

[0046] A Dense Object Net (DON) is,

number

number

[0047] To train DON, a loss function based on pixel-level symmetric loss can be minimized in a fully connected network. Pixel-level symmetric loss is a technique for generating a loss from two given images. According to this loss, all pairs of semantically similar pixels in the two images must be mapped to similarity pairs in the descriptor space. Furthermore, all pairs of semantically dissimilar pixels must be mapped to different regions in the descriptor space. By applying pixel-level symmetric loss, a network capable of detecting object (class)-specific features is obtained. This capability is utilized according to various embodiments. In particular, object-specific feature points are detected in images not yet seen (in the training data) (this represents a specific configuration, e.g., an initial configuration that serves as a starting point when controlling the robot arm 101) (if it is an element in the descriptor space, this element is considered a feature point). The Euclidean coordinates of a given descriptor (and therefore a given feature, e.g., object 113) are found by finding the pixels of the image represented in descriptor space using descriptors that are as close as possible to a given (target) descriptor.

number

number

number

[0048] As mentioned above, this is just one example, and other approaches can be used to find matches in a (new) image using a given descriptor. For example, according to an embodiment where the expected value of exponentially weighted squared errors is obtained for greater robustness, a modified version called spatial expectation is used.

[0049] The conversion from image coordinates (pixel coordinates) to world coordinates (e.g., the robot's reference frame coordinates) is a standard technique in robotics and computer vision. Applying this conversion to the "best" pixels found yields a routine for finding Euclidean coordinates for object-specific feature points in descriptor space. These Euclidean coordinates are expressed as the Euclidean coordinates of the feature points. These are scene-dependent (i.e., configuration-dependent).

[0050] The premise here is that the output of a high-density object network is available, and for each descriptor in the current image descriptor image (e.g., showing the initial configuration of the object to be controlled by the robot arm 101), it is known whether it corresponds to a point on the target object (e.g., object 113), and if so, the Euclidean coordinates of the point on the object to which it belongs are also known. Below, we describe an approach that allows us to find a pose estimate applicable to demonstration-based learning. The pose estimate must be invariant with respect to the motion of the object, in the sense that the pose is also transformed when the object is moved. It should be noted here that RGB images are not used for this purpose. In particular, all data points in the training (or learning) dataset

number

[0051] In various embodiments, approaches that enable the discovery of pose estimations applicable to demonstration-based learning include methods based on optimization for finding K descriptors (referred to as feature points) and (optionally) subsets.

number

[0052] Descriptor Selection This involves finding feature points that can be used to parameterize the coordinate frame, in this case.

[0053] According to various embodiments, a set of feature points is selected to satisfy various conditions. Four conditions are listed below, but it should be noted that not all of them must be considered, and weights can be assigned to one condition to give more importance to the others.

[0054] The first condition is that selected feature points must be detected with a small detection error (this includes the visibility of the object features they encode). To quantify this criterion, for each possible descriptor, the matching error is calculated across all data points in the training dataset, and this error is calculated as the expected matching error for unknown data points (since the detection error is normalized, the cumulative error can be used instead of the mean error).

[0055] There are two domains in which matching errors can be calculated: the descriptor domain (the distance between a descriptor and a descriptor image) and the Euclidean domain (the change in the Euclidean coordinates of a feature point). The advantage of calculating matching errors within the descriptor domain is that it does not require any ground truth information about which pixels (semantically) belong to which pixels in other images. Despite the availability of such datasets (training datasets for high-density object networks), using them would compromise generality. However, the Euclidean domain can also be used, as there are objects in which feature points can be tracked very well within descriptor space but poorly within Euclidean space.

[0056] A second condition that a set of feature points must satisfy in order to select it is spatial variation in descriptor space. This condition guarantees that the selected feature points describe various features of an object in Euclidean space from the perspective of a high-density object network. Similar to the spatial separation condition in descriptor space, a third condition can be spatial separation in Euclidean space. If the feature points are too close to each other in Euclidean space (i.e., their Euclidean coordinates are too similar with respect to a single object), the effect of matching errors increases. Unlike matching errors in Euclidean space, the spatial separation condition does not require knowledge of the relationships between various images, and therefore does not compromise generality.

[0057] The fourth condition is that features present in all data points (or as many data points as possible) of the training dataset must be captured. Assuming that each descriptor image in the training dataset is generated by a DON trained on camera images, this means that all descriptors must be visible in all (or as many as possible) images of the training dataset. Since we can assume that configurations that will also exist during deployment are represented by the demonstration, we can expect that the object parts having these feature points are likely to be visible in the images acquired during deployment as well.

[0058] Equation (2) provides an optimization problem for selecting feature points that take into account the four conditions mentioned above.

number

[0059] In this case, K is the number of selected descriptors.

number

number

number

number

number

number

[0060] Detected function l (p k ) is a feature point p in image l. k This indicator function is equal to 0 if it is detected, and ∞ otherwise.

[0061] For any finite set I, the normalization operator is

number

[0062] In equation (3), the function e(.) maps the descriptor to the corresponding matching error in Euclidean space. If a training dataset for a high-density object network is available (or any other dataset consisting of multiple scenes where the position and orientation of objects are static), the following estimation process can be used for e(x): select k images (randomly) for each scene i, and for x in each image, the Euclidean coordinates p ij Infer the average value within a single scene.

number

number

[0063] Parameterization of coordinate frames Given a set of multiple feature points (for example, learned as described above), a coordinate system (or coordinate frame) for space X that depends on the learned feature points is derived (learned). Here, D represents a matrix with columns that are the Euclidean coordinates of the learned feature points, and B(D) represents the desired basis matrix (i.e., its columns are the basis of X). In the following, (without loss of generality)

number

number

number

number

number

number

number

[0064] The point to note here is,

number

[0065] Using the cumulative matching error described above, the weights

number

[0066] For simplicity, the optimization in (4) can be performed over all B that define the following coordinate system, namely, the axes of this coordinate system are either vectors between two points or vectors pointing from one center point to another. To obtain an orthogonal basis, the orthogonalization process can be applied to B(D).

[0067] The average of the Euclidean coordinates of the feature points can be used as the origin.

[0068] supporting hyperplane If the object being trained is a "real" 3D object, the above approach for feature point selection and coordinate frame learning can be applied to that object. However, in many industrial applications, the object under consideration is not a real 3D object in the sense that there are directions in which the object undergoes only slight changes. In such cases, the above approach will result in at least one unreliable coordinate axis. Therefore, in the case of such an object (e.g., a flat object), this direction is eliminated by using orthogonal projection onto a selected appropriate hyperplane.

[0069] Figure 2 illustrates a method for selecting feature points, learning a coordinate frame 204 for an object 201 in a training dataset 202 that is largely unchanged in one direction, and then applying that coordinate frame during deployment.

[0070] This method includes the step of selecting (learning) a hyperplane 203. The hyperplane must approximate the surface on which the points corresponding to the object's (matched) descriptors (referred to as the object's point cloud) are located. If the surface is nearly flat, a standard least squares or singular value decomposition approach can be used. However, this is not always the case. For example, objects in a classic PCB (printed circuit board) have one direction of little variation (a direction perpendicular to the board), and approaches such as least squares or singular value decomposition may not work well due to the size of the electronic components and matching errors in the high-density object net.

[0071] To overcome this problem, according to various embodiments, a modified (or adapted) Theil-Sen estimator is used to calculate the support plane 203.

[0072] Every plane is uniquely defined by two direction vectors and one support point. Moreover, a hyperplane is a function

number

[0073] The slope of f is,

number

[0074] A point y that occurs within the image f,

number

number

[0075] The ratio m1 / m2 can be calculated from y and c2 / c1. In order to learn (i.e., fit) the hyperplane, the slopes of both must be determined simultaneously. Therefore, according to one embodiment, an iterative algorithm is used that alternates between determining m1 / m2 and estimating c1 and c2. The following algorithm 1 shows one example that operates on a point cloud P (where each point has three spatial coordinates).

number

[0076] Hyperplane 203 can be estimated from any one of the data points in the training dataset (i.e., any of the descriptor images accompanied by a transformation to generate the corresponding point cloud).

[0077] If there is a hyperplane 203, the Euclidean coordinates of the feature points can be projected onto that hyperplane. To apply the method for parameterizing the coordinate frame described above, (

number

number

[0078] Thus, during training, a point cloud consisting of multiple points (in Euclidean space) is generated from object 201, which corresponds to descriptors that correspond to locations on object 201. For each descriptor image, the Euclidean coordinates of the descriptor are obtained by individual transformations applied to the descriptor image.

[0079] If the object is not in real 3D, the object's supporting hyperplane 203 can be determined as described above. One possible method for determining whether a hyperplane (and projection onto this hyperplane) is needed is provided by the singular values ​​of the point cloud, which are interpreted as a single matrix. This determination can be performed automatically.

[0080] For each point in the point cloud, the Euclidean distance between its Euclidean coordinates and the hyperplane is calculated. The point with the shortest Euclidean distance to the hyperplane is selected, and the other points are removed, thus excluding outliers from the point cloud.

[0081] Next, the above approach is used to obtain a planar coordinate frame 204 (i.e., a coordinate frame with respect to the hyperplane) that depends on the Euclidean coordinates of the selected set of feature points. If the object is not real 3D, these Euclidean coordinates of the selected set of feature points are Euclidean coordinates projected onto the hyperplane. Therefore, if feature points are selected considering the condition regarding the Euclidean coordinates of the feature points, and the object is not real 3D, this condition applies to the Euclidean coordinates projected onto the hyperplane.

[0082] During deployment (i.e., online estimation), if a point cloud is obtained based on observation 209 (for example, the current image of object 113 in a control scenario not yet seen during training), and the object is not real 3D, then the hyperplane 205 for this point cloud and the positions 206 of the 3D points corresponding to the selected feature points can be determined (by matching the selected feature points with the descriptors of the current image). This means that, given the descriptor image (generated by DON for the image of object 113) and the intrinsic and extrinsic parameters of the camera, the Euclidean coordinates of the point cloud, and especially the selected feature points (up to the possible matching error), can be determined. Using the approach described above, the hyperplane 204 can be determined (i.e., matched to the point cloud) as during training.

[0083] The Euclidean coordinates of the selected feature points (i.e., the points defining the planar frame 204) are projected onto the hyperplane at 207, thereby defining the planar coordinate frame resulting from these projections.

[0084] The combined (3D) coordinate frame 208 is, as described above, a combination of the average value of the point cloud's Euclidean coordinates, the planar coordinate frame, and the transformed world coordinate system. In cases where projection is not required (i.e., in the case of real 3D objects), the combined coordinate frame 208 is determined solely by the positions of selected feature points (found through feature matching between the feature points and the descriptor image of the current image).

[0085] The selection based on the optimization of feature points according to equation (2) can be solved, for example, by the following heuristic approach. Given one dataset, one data point is selected and a set of descriptors located on the object is inferred. In this set, all descriptors that are closer to each other than a given threshold are replaced / approximated by the mean of the coordinate units of the original descriptors. In the second step, the matching error of all descriptors in the compressed set is calculated over all data points in the training dataset. Descriptors with a matching error greater than a given threshold for any data point are removed. In this set, a subset of descriptors with the smallest cumulative matching error (individually for each descriptor) is selected. For each possible combination of K descriptors in this subset, the cost used in the optimization problem of equation (2) is evaluated. It should be noted that since this cost is evaluated based on a finite number of points, equation (3) is applicable. All feature point pairs that have a Euclidean distance less than a given threshold (in descriptor space or Euclidean space) are rejected. The descriptor pair with the lowest total cost is used.

[0086] In summary, various embodiments provide a method as illustrated in Figure 3.

[0087] Figure 3 shows a flowchart 300 illustrating a method for controlling the robot.

[0088] In step 301, a demonstration is performed (for example, by a human user), during which the processing of objects is actually carried out by each demonstration.

[0089] In step 302, for each demonstration, at least one descriptor image is supplied (or acquired) from the robot's viewpoint relative to the object, and this descriptor image specifies feature points related to the location on the object. The descriptor image can be generated by DON, for example, from images captured by a camera attached to the robot during the demonstration.

[0090] In step 303, a set of multiple feature points is (automatically) selected from the specified feature points. These feature points are selected by searching for the optimal condition of the objective function that rewards the selected feature points that are visible within the descriptor image.

[0091] In step 304, the robot control model (i.e., TP-HSMM) is trained using a demonstration, in which case the robot control model is configured to output control information depending on the input object pose.

[0092] In step 305, the robot is controlled in the following steps with respect to a control scene that includes an object. That is, In step 306, obtain a descriptor image of the object from the robot's viewpoint. In step 307, a selected set of multiple feature points is placed within the descriptor image of the object. In step 308, the Euclidean coordinates of the feature points placed with respect to this control scene are obtained. In step 309, the attitude is estimated from the obtained Euclidean coordinates. In step 310, the estimated pose is supplied as input to the robot control model, and the robot control model is used to control the robot to process the object.

[0093] Using the approach shown in Figure 3, control signals can be calculated to control physical systems such as computer-controlled machines like robots, vehicles, home appliances, power tools, manufacturing machinery, personal assistants, or access control systems. According to various embodiments, policies for controlling physical systems can be learned, and then the physical systems can be operated accordingly.

[0094] Various embodiments can receive and use image data (i.e., digital images) from various visual sensors (cameras), such as video, radar, LiDAR, ultrasound, thermal imaging, motion, and sonar, as the basis for descriptor images, for example.

[0095] According to one embodiment, this method is implemented by a computer.

[0096] Although specific embodiments have been illustrated and described herein, as will be obvious to those skilled in the art, various alternative embodiments and / or equivalent embodiments can be used in place of the specific embodiments shown and described herein without departing from the scope of the present invention. This application is intended to cover any adaptation or modification of the specific embodiments discussed herein. Accordingly, the present invention is intended to be limited only by the claims and their equivalents.

Claims

1. A method for controlling a robot, The step of conducting a demonstration, where each demonstration involves actually performing the processing of an object, For each demonstration, the step of supplying at least one descriptor image from the robot's viewpoint relative to the object, wherein the descriptor image specifies feature points relating to the location on the object; A step of selecting a set of multiple feature points from the specified feature points, wherein the feature points are selected by searching for the optimal condition of an objective function that rewards the selected feature points that are visible in the descriptor image. A step of training a robot control model using a demonstration, wherein the robot control model is configured to output control information depending on the input object pose, A step of controlling a robot in a control scene that includes an object, A descriptor image of the object is obtained from the viewpoint of the robot. A selected set consisting of multiple feature points is placed within the descriptor image of the object. The Euclidean coordinates of the feature points arranged in relation to the control scene are determined. The attitude is estimated from the obtained Euclidean coordinates, The estimated posture is supplied as input to the robot control model, and the robot is controlled using the robot control model to process the object. This involves the steps of controlling the robot, Includes, The step of training the robot control model using the aforementioned demonstration is: For each demonstration, the steps include: placing a selected set of multiple feature points within the descriptor image of the object; The steps include determining the Euclidean coordinates of the feature points arranged for the demonstration, The steps include: estimating the attitude from the Euclidean coordinates obtained for the aforementioned demonstration; Methods that include...

2. The objective function further rewards one or more of the following: the selected feature points are spaced apart in descriptor space; the locations on the object corresponding to the selected feature points are spaced apart in Euclidean space; and the detection error for the selected feature points on the object is small. The method according to claim 1.

3. The steps include matching one plane to the object, The steps include selecting the feature points such that the feature points define a coordinate frame on the plane, A step of estimating the orientation from the Euclidean coordinates obtained for the arranged feature points and information regarding the orientation of the matched plane, Further including, The method according to claim 1.

4. The step of estimating the orientation from the obtained Euclidean coordinates includes the step of projecting the Euclidean coordinates of the placed feature points onto a matched hyperplane. The method according to claim 3.

5. The plane is matched to the object, the feature points are selected such that the feature points define one coordinate frame on the plane, and the orientation is estimated from the determined Euclidean coordinates of the positioned feature points and information about the orientation of the plane, provided that the change of the object in one spatial direction is below a predetermined threshold. The method according to claim 3.

6. The process includes the step of determining a rule for deriving a coordinate frame from the Euclidean coordinates of the selected feature points, The step of estimating the attitude from the obtained Euclidean coordinates includes the step of applying the derivation rule to the selected feature points, the derivation rule being obtained by searching for the minimum dependence of the coordinate frame on noise in the Euclidean coordinates. The method according to claim 1.

7. The steps include obtaining the descriptor image of the object from a camera image of the object using a high-density object network. The method according to claim 1.

8. A robot controller configured to carry out the method described in any one of claims 1 to 7.

9. A computer program that, when executed by a computer, includes instructions causing the computer to perform the method according to any one of claims 1 to 7.

10. A computer-readable medium containing instructions for causing a computer to perform the method according to any one of claims 1 to 7 when executed by the computer.

Images