Apparatus and method for determining the location of an object from a camera image of the object.

Abstract

To provide a method for localizing a location of an object from a camera image of the object.SOLUTION: In a method for localizing a location of an object from a camera image of the object: a machine learning model for mapping the camera image onto a descriptor image is trained; a location to be localized on each object is identified by means of a descriptor supplied by the trained machine learning model, with respect to each camera image; and the descriptor used for localizing the location to be localized is updated in the process of the sequence of the camera images.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] This disclosure relates to an apparatus and method for determining the location of an object from a camera image of the object. [Background technology]

[0002] To enable flexible production or processing of objects by robots, it is desirable that robots can handle objects regardless of their orientation when placed within the robot's workspace. Therefore, robots are required to be able to recognize the location of each part of an object, enabling them to grasp objects in the correct position, for example, to attach them to other objects or to weld them to specific locations. This means that robots are required to be able to recognize the object's pose (position and orientation), or even its regions, such as barcodes, from one or more images captured by a camera mounted on the robot. One approach to achieving this is to determine descriptors, i.e., points (vectors) in a given descriptor space, for parts of an object (i.e., pixels of the object represented in the camera image plane). The robot is trained to assign the same descriptor to the same part of an object regardless of its current pose, thereby recognizing the topology of the object in the image, and thus knowing, for example, which corners of an object are located where in the image. If the camera's pose is known, the object's pose or the location of its regions in three-dimensional space can be deduced. Recognizing topology is achievable using a reasonably trained machine learning model. However, difficulties arise even in this procedure. This is because, for example, due to varying lighting conditions (especially reflections) or insufficient training on multiple different camera images, the machine learning model may not always assign descriptors to the same location on an object. This can lead to inaccuracies in pose identification, or in general, the identification of object locations. Therefore, an approach that can achieve higher accuracy, for example, in pose identification, is desirable. [Overview of the project] [Means for solving the problem]

[0003] According to various embodiments, a method for determining the location of an object from a camera image of the object, the method is This defines the location where an object should be located for an object type, and To determine the reference point for the relative orientation of the location to be identified, The process involves training a machine learning model to map camera images, each representing an object of a given object type, to descriptor images, wherein the descriptor image to which the camera image should be mapped, for a given image position, has a descriptor of the object's location at that image position. Setting the descriptor base set to the descriptor initial set, Receiving a temporal sequence of camera images, each representing an object of a specific object type, For each camera image, A trained machine learning model maps camera images to descriptor images. By searching for a base set of descriptors within the descriptor image, we can identify the location where the object should be located relative to the base set of descriptors. This allows for the identification of the location on each object that needs to be located. The method includes, With respect to at least some of the camera images in the sequence of camera images, To determine the relative orientation of a localized location to a reference set, This involves comparing the relative orientation of the location to be located with the relative orientation of the located location relative to the set of references, By searching for the test set descriptor among the descriptors in the descriptor image, we can identify the location where the object should be located relative to the test set of the descriptor, To determine the relative orientation of the localized location to the test set, This involves comparing the relative orientation of the location to be localized with the relative orientation of the localized location relative to the test set, If the agreement between one or more camera images between the reference for the relative orientation of the location to be localized and the relative orientation of the localized location relative to the test set is better than the agreement with respect to the reference set, update the descriptor of the reference set to the descriptor of the test set. A method is provided that includes this.

[0004] The method described above allows for more accurate localization of an object's location when multiple descriptors are used (e.g., tracking), such as to identify a pose or to identify (or track) a region. This is because the descriptors used are improved across the sequence of camera images in terms of better satisfying their theoretical properties, namely, being assigned to the same location in a view-invariant manner. In particular, the method above increases robustness to changes in lighting conditions in the sequence of camera images.

[0005] In this way, the above method enables, for example, reliable picking (e.g., grasping) of an object in any orientation, or enables precise identification of a specific area of ​​an object, such as a barcode.

[0006] Various examples are described below.

[0007] Example 1 is a method for determining the location of an object from its camera image.

[0008] Example 2 is the method of Example 1, wherein the relative orientation has a pairwise distance between a location to be located or a location that has been located in three-dimensional space.

[0009] This allows us to evaluate how accurately the located location is positioned within three-dimensional space. In particular, by using distances in three-dimensional space (not just distances in the camera image plane), it is ensured that detection errors that are only reflected in deviations perpendicular to the camera image plane are taken into account.

[0010] Example 3 is the method of Example 1 or 2, wherein the location to be localized for an object type is defined on a reference camera image of the object of the object type, the reference camera image is supplied to a machine learning model, and the reference set of descriptors is set to the descriptors of the location to be localized in the descriptor image output by the machine learning model for the reference camera image.

[0011] In this way, the locations to be localized and good initial values ​​for the descriptor reference set can be easily defined, because this reference set corresponds to the locations to be localized, at least with respect to the reference camera image. Subsequently, the reference set used to localize the locations to be localized can be improved over the course of the camera image sequence.

[0012] Example 4 is a method according to any one of Examples 1 to 3, wherein the test set is selected within a limited range of the initial set of descriptors.

[0013] For example, the limited range only permits a specific relative deviation between the descriptors of the test set and those of the initial set. Descriptors that deviate significantly from the initial set, and thus are only suitable with a low probability for multiple camera images, are thereby (unnecessarily) tested, or even adopted into the reference set based on one (or a few) camera images, can be avoided. In particular, the unstable behavior of the descriptor matching process can be avoided.

[0014] Example 5 is a method according to any one of Examples 1 to 4, which includes specifying a test set of descriptors by means of the covariance matrix adaptation evolution strategy method.

[0015] This enables efficient optimization of non-closed-form functions, where, when a large number of evaluation points such as those that occur in the process of camera images occur, the mapping of descriptors to a specific intended accuracy of the location of the object becomes possible.

[0016] Example 6 is a method for controlling a robot, the method comprising: locating the location of an object to be processed by the robot according to any one of Examples 1 to 5; specifying the pose of the object from the located location and controlling the robot according to the specified pose, and / or specifying the area of the object (e.g., the area to be scanned, or otherwise the area important for processing or control) from the located location and controlling the robot according to the specified area; and is a method including the above.

[0017] Example 7 is a software or hardware agent, particularly a robot, comprising a camera configured to provide a camera image of an object, and a control device configured to implement the method according to any one of Examples 1 to 6. and is a software or hardware agent having the above.

[0018] Embodiment 8 is the software or hardware agent described in Embodiment 7, wherein the control device has at least one actuator and is configured to control at least one actuator using a localized location.

[0019] Example 9 is a computer program that, when executed by a processor, includes instructions to cause the processor to perform the method described in any one of Examples 1 to 6.

[0020] Example 10 is a computer-readable medium that, when executed by a processor, stores instructions causing the processor to perform the method described in any one of Examples 1 to 6.

[0021] In drawings, generally, the same reference numerals refer to the same parts in various different viewpoints. The drawings are not necessarily to scale, and instead, the focus is generally on illustrating the principles of the present invention. In the following description, various embodiments will be explained with reference to the following drawings. [Brief explanation of the drawing]

[0022] [Figure 1] A diagram showing a robot. [Figure 2] This figure shows the training of a neural network according to one embodiment. [Figure 3] This figure shows the identification of an object pose or grasping pose according to one embodiment. [Figure 4] This diagram shows the fitting of descriptors used to locate objects through a sequence of camera images. [Figure 5] This is a flowchart illustrating a method for determining the location of an object from its camera image. [Modes for carrying out the invention]

[0023] The following detailed description refers to the accompanying drawings which schematically illustrate specific details and embodiments of the present disclosure that may be used to carry out the present invention. Other embodiments may be used, and structural, logical, and electrical modifications may be made without departing from the scope of protection of the present invention. Many embodiments of the present disclosure can be combined with one or more other embodiments of the present disclosure to form new embodiments, so the multiple different embodiments of the present disclosure are not necessarily mutually exclusive.

[0024] The following provides detailed explanations of various examples.

[0025] Figure 1 shows robot 100.

[0026] Robot 100 includes a robotic arm 101, for example, an industrial robotic arm for handling or assembling a workpiece (or one or more other objects). The robotic arm 101 includes manipulators 102, 103, 104 and a base (or support) 105 that supports the manipulators 102, 103, 104. The term "manipulator" refers to a movable component of the robotic arm 101, which enables physical interaction with the environment by manipulating these movable components, for example, to perform a task. For control, robot 100 includes a (robot) control device 106 configured to enable interaction with the environment according to a control program. The last component 104 of the manipulators 102, 103, 104 (the one furthest from the base 105) is also referred to as an end effector 104 and may include one or more tools such as a welding burner, a gripping device, a painting device, or similar.

[0027] Other manipulators 102 and 103 (closer to the base 105) can constitute a positioning device, which in cooperation with the end effector 104 provides a robotic arm 101 having the end effector 104 at its end. The robotic arm 101 is a mechanical arm that can provide functions similar to a human arm (which may have a tool at its end).

[0028] The robot arm 101 may include joint elements 107, 108, and 109, which connect manipulators 102, 103, and 104 to each other and to the base 105. The joint elements 107, 108, and 109 may have one or more joints, each of which can provide rotatable motion (i.e., rotational motion) and / or translational motion (i.e., movement) relative to the associated manipulator. The motion of the manipulators 102, 103, and 104 can be initiated by actuators controlled by the control device 106.

[0029] The term "actuator" can be understood as a component configured to execute a mechanism or process in response to its own drive. An actuator can execute commands (so-called activations) created by the control device 106 in the form of mechanical motion. An actuator, such as an electromechanical transducer, can be configured to convert electrical energy into mechanical energy in response to its own drive.

[0030] The term "control device" can be understood as any type of logical implementation entity that can execute software, firmware, or a combination thereof stored in a storage medium, and that can issue instructions, such as instructions to an actuator in this example, including a circuit and / or processor. For example, a control device can be configured by program code (e.g., software) to control the operation of a system, in this example, a robot.

[0031] In this example, the control device 106 includes one or more processors 110 and a memory 111 for storing code and data, and the processors 110 control the robot arm 101 based on this code and data. According to various embodiments, the control device 106 controls the robot arm 101 based on a machine learning model 112 stored in the memory 111.

[0032] According to various embodiments, the machine learning model 112 is configured and trained so that, for example, a robot 100 can recognize the picking pose of an object 113 placed within the workspace of the robot arm 101 from a camera image, for example, for a robot that should pick objects from a box ("bin-picking"). This means that the robot 100 recognizes how it can pick the object 113, i.e., how it must orient and move its end effector 104 in order to pick (e.g., grasp) the object 113. The picking pose is understood to contain sufficient information for picking, i.e., information about the orientation and position of the object 113, sufficient to determine how the object 113 can be grasped. The picking pose does not necessarily require complete orientation information about the object 113, because, for example, in the case of an object 113 that has a rotationally symmetric part for grasping, it may not matter how that rotationally symmetric part is rotated about its axis of rotation.

[0033] For example, the robot 100 may be equipped with one or more cameras 114 that enable the robot 100 to take images of its workspace. The cameras 114 are attached, for example, to the robot arm 101, so that the robot can obtain images of the object 113 from various different viewpoints by rotating the robot arm 101.

[0034] An example of a machine learning model 112 for object recognition is a high-density object mesh. A high-density object mesh maps an image (e.g., an RGB image provided by a camera 114) to a descriptor space image having a specific selected dimension D. However, other machine learning models 112 may also be used, in particular a machine learning model that does not necessarily generate a "high-density" feature map, but merely assigns descriptor values ​​to specific points of an object (e.g., corners).

[0035] In various embodiments, typically applicable to industrial assembly or processing tasks, an approach is used to recognize an object and its orientation, assuming that the 3D model of the object (e.g., a CAD (computer-aided design) model) is known. Nonlinear dimensionality reduction techniques can be used to compute the optimal target image for training the input image for the neural network. Thus, in various embodiments, supervised training of the neural network is used. Alternatively, an RGBD image (RGB + depth information) of the object can be taken, and the 3D model for the object can be determined from this RGBD image. Alternatively, self-supervised training may be performed such that the machine learning model itself learns descriptors about the object's location.

[0036] In supervised training, according to one embodiment for generating training data to train a machine learning model 112, data collection is performed first. In particular, for example, registered RGB (red-green-blue) images are collected. Registered images as used herein mean RGB images with known intrinsic and extrinsic camera values. In a real-world scenario, for example, a camera 114 attached to the robot (e.g., a camera attached to the robot's wrist) is used to scan objects while the robot (e.g., a robotic arm 101) is moving around. In a simulated scenario, a photorealistically generated RGB image is used using known object poses.

[0037] After collecting RGB images, a target image for the RGB images is rendered for supervised training of the neural network.

[0038] It is assumed that the orientation of each object in world coordinates is known for each collected RGB image. This is straightforward in simulated scenarios, but requires manual adjustment in real-world scenarios, such as positioning objects in their designated locations. RGBD (RGB + depth information) images can also be used to determine object positions.

[0039] Using this information, and for each RGB image (i.e., training input image), a descriptor image (i.e., the training output image, also called the target image or ground truth image) is rendered using, for example, a vertex descriptor calculation technique as described later.

[0040] Once a target image is generated for each RGB image, that is, once pairs of RGB images and target images are formed, these pairs of training input images and their associated target images can be used as training data to train a neural network, as shown in Figure 2.

[0041] Figure 2 shows the training of a neural network 200 according to one embodiment.

[0042] Neural Network 200 is a fully convolutional network that maps an h×w×3 tensor (input image) to an h×w×D tensor (output image).

[0043] The neural network 200 includes multiple stages 204 of convolutional layers to combine the outputs of multiple different layers, followed by a pooling layer, an upsampling layer 205, and skip connections 206.

[0044] For training, the neural network 200 receives a training input image 201 and outputs an output image 202 having pixel values ​​in descriptor space (e.g., color components corresponding to descriptor vector components). The training loss is calculated between the output image 202 and the target image 203 associated with the training input image. This can be done for a batch of training input images, the training loss can be averaged across these training input images, and the weights of the neural network 200 are trained using stochastic gradient descent with the training loss. The training loss calculated between the output image 202 and the target image 203 is, for example, an L2 loss function (to minimize the pixel-wise least-squares error between the target image 203 and the output image 202).

[0045] The training input image 201 shows an object, and the target and output images contain vectors in descriptor space. The vectors in descriptor space can be mapped to colors so that the output image 202 (and target image 203) resembles a heatmap of the object.

[0046] A vector in descriptor space (also called a (high-density) descriptor) is a d-dimensional vector (for example, d is 1, 2, or 3) assigned to each pixel in each image (for example, each pixel in input image 201, assuming that input image 201 and output image 202 have the same dimensions). The high-density descriptor implicitly encodes the surface topology of an object shown in input image 201 in a way that is invariant with respect to the object's orientation or camera position.

[0047] Given a 3D model of an object, it is possible to analytically determine the optimal and unique descriptor for each vertex of the object's 3D model. According to various embodiments, these optimal descriptors (or estimates of these descriptors determined by optimization) are used to generate a target image for a registered RGB image, which leads to fully supervised training of the neural network 200. Furthermore, the descriptor space becomes explainable and optimal regardless of the selected descriptor dimension d.

[0048] If a machine learning model 112, such as a neural network 200, has been trained to map camera images of object 113 to descriptor images, then to identify the picking pose of object 113 in an unknown position, the following steps can be taken:

[0049] First, on the object, there are multiple reference points p on object 113. i ,i=1,···,N are selected and the descriptors for these reference points are identified. These reference points are the locations where the object should be positioned for subsequent ("new") camera images, and the reference point descriptors are the (initial, i.e., initial) reference set of descriptors (which can be adapted in the course of the sequence of camera images, as described later). The above selection takes a camera image of object 113 and reference pixels (u i ,v i) can be implemented by selecting i =Î d (u i , v i ) and mapping the camera image of the neural network 200 to the descriptor image. Thereafter, the descriptor at the position in the descriptor image given by the position of the reference pixel can be regarded as the descriptor of the reference point, that is, the descriptor of the reference point is d d =Î = f(I;θ) is the descriptor image, f is the mapping (from the camera image to the descriptor image) performed by the neural network, I is the camera image, and θ is the weight of the machine learning model 200.

[0050] When the object 113 is in an unknown pose, a new camera image I neu is taken, and the corresponding descriptor image I d neu =Î = f(I neu ;θ) is specified by the machine learning model. In this new descriptor image, the descriptor in the new descriptor image that is as close as possible to the d i descriptor of the reference image is searched for, for example, by

Number

[0051] The position (u d neu )* of the reference point in the descriptor image I neu (and thus the corresponding new camera image I i ) is thus specified or estimated, and the position of the reference point in the three-dimensional space is specified. For example, a depth image is taken together with the camera image I i (or the camera image I neu has a depth channel and is, for example, an RGBD image), and thus, from (u neu , v i )*, the i-th reference point p is obtained by projecting the depth value at this position (u i to the respective working space coordinate systems )* ( i , v i )*i It is possible to determine its three-dimensional position.

[0052] If the positions of multiple reference points in space are known, the picking pose shown in Figure 3 can be identified from these positions.

[0053] For example, the positions of two reference points p1 and p2 on object 300 in space are identified, and these two positions are linearly combined, for example, by taking the average of these two positions, to define an anchor point 304. To define the orientation of gripping, a first axis 301 having directions p1 and p2 is defined, and a second axis 302 is arbitrarily defined through the anchor point 304, for example, in the z-axis direction of camera 114, or in the direction of an axis of the work space coordinate system. A third axis 303 passing through the anchor point 304 can be calculated by the vector product of the direction vector of the first axis 301 and the direction vector of the second axis 302. The three axes 301 to 303 and the anchor point 304 define the picking pose for object 300. The robot can then be controlled to grasp the shaft of object 300 extending in the direction of the first axis. Reference points p1 and p2 are defined, for example, to extend along the shaft as shown in the figure, that is, to extend along the elongated member of an object suitable for gripping.

[0054] Similarly, three or more reference points can be placed on the gripping surface of an object, and thus, from the positions of these reference points, the complete 6D picking pose of the object can be determined, or the orientation of the gripping surface from which the object can be picked (gripped or sucked) can also be determined.

[0055] The gripping portion does not necessarily have to be forceps-shaped, but it should be noted that it may have a suction device for picking up an object by sucking it onto a suitable surface. In this case, it may be desirable to identify a picking pose that indicates the orientation and position of the object surface suitable for suction, for example, in order to guide the suction device into the correct position. This can be done, for example, by identifying an anchor point and the plane normal vector at the anchor point.

[0056] To identify the picking pose, you can also use more than three reference points, for example, to reduce errors by averaging.

[0057] Similar to the picking pose, the control device 106 can also identify a predetermined area on the object 300, for example, the bounding box of a barcode provided on the object 300 to identify the object. The control device 106 can then control the robot arm 101, for example, so that the robot arm 101 holds the object 300 in front of the camera so that the camera can read the barcode.

[0058] As described above, the high-density object network assigns descriptor images to camera images (e.g., RGB) of an object or part thereof, taken from any observation angle, and these descriptor images assign multidimensional descriptors to each pixel (or each position) in the input camera image. Descriptors have the theoretical property that a particular point on the surface of an object is always associated with the same descriptor, regardless of the observation angle. This property is useful for various applications, for example, to recognize a region on the surface of a target object by recognizing multiple descriptors, and for example, to recognize a region or corner point of a picking pose, as described above with reference to Figure 3. Using additional depth information, i.e., using RGBD input data (i.e., RGB image + depth information) instead of just the RGB image, the identified points can be projected into 3D space to fully define such regions.

[0059] However, the mapping of points on an object to view-invariant descriptors is not fundamentally perfect in practice. The quality deteriorates in camera images showing various views (fields of view) of an object due to factors such as a small number of visual features, a small dimension in the descriptor space, insufficient training data for some parts, ambiguity due to symmetrical or similar objects, significantly different observation angles or significantly different external conditions such as ambient lighting, and invalid depth information due to reflective surfaces.

[0060] In various embodiments, robustness against such factors is enhanced not by statically tracking initially selected descriptors, but by optimizing them by (typically slightly) changing them during the sequence of camera images, thereby enabling better identification of locations on objects (and consequently, regions given by poses or locations, e.g., the corners of bounding boxes). In other words, during operation ("online"), the selection of descriptors used is improved over time (i.e., during the temporal sequence of received camera images).

[0061] Descriptor selection can be performed using black-box optimization methods. Examples include Bayesian optimization (BO) and CMA-ES (Covariance Matrix Adaptive Evolutionary Strategy). In one embodiment, CMA-ES is used because of its relatively good performance for a large number of sample values, which is a typically important aspect in this application.

[0062] To improve descriptors in the course of a sequence of camera images, a set of descriptors (for example, three descriptors for pose detection, or four descriptors to define a rectangular area) is first defined as an initial criterion (i.e., an initial value for the set of descriptors). This definition is achieved, for example, by having the user select a corresponding number of locations on objects in the camera image, and then having these locations mapped to descriptors by a machine learning model. This baseline set of descriptors is adapted over time (i.e., in the course of a sequence of camera images). Specifically, the baseline set of descriptors is replaced by a test set of descriptors if that test set provides higher accuracy. That is, starting with the set of descriptors selected as the initial criterion, there is a current baseline set of descriptors for each camera image. According to one embodiment, multiple test sets of descriptors are evaluated in parallel, i.e., based on the same image or on the same partial sequence of multiple images, and the one that provides the best accuracy is selected as the new baseline set.

[0063] Next, the following is performed on each camera image in the sequence of camera images. (A) An identification accuracy is determined, which indicates how well the location of an object to be identified is identified (for example, how well the target area is identified) in relation to the camera image and the (current) set of descriptors. (B) A new set of descriptors (test set) is proposed to improve the accuracy of identifying future camera image sequences.

[0064] In case (A), a quality measure of the set of descriptors is used for identification accuracy. Instead of evaluating the descriptors individually (for example, by calculating the confidence that the identified location is unique), the quality of the set of descriptors is evaluated based on the relative spatial orientation of the locations identified by that set of descriptors on the surface of the object (in 3D space). (A1) For each descriptor, the machine learning model selects the location (pixel) in the camera image that maps to these descriptors. (A2) Each of these positions in the camera image plane is mapped to a corresponding 3D position using depth information (and, for example, intrinsic camera calibration parameters). (A3) A scale is calculated to capture the relative orientation of 3D positions, for example, the pairwise (e.g., Euclidean) distance between 3D positions. (A4) The calculated scale is compared to a reference scale. The result of the comparison (e.g., the average over the difference in pairwise distances between the 3D position and the reference distance) indicates the specificity. The smaller the comparison result, the higher the specificity. For example, the value of specificity is defined as the reciprocal of the comparison result. The reference scale or reference distance can be, for example, identifiable in a first camera image or measurable on an actual object. For example, the distance between corner points (or other prominent locations) of a barcode on an actual object is measured.

[0065] In case (B), an optimization method is performed on an objective function (which is unknown in closed form but can be evaluated by (A)), which takes a set of descriptors as input and outputs a precision measure for camera images (the precision measure may be the average over the precision measures of multiple camera images). To improve the baseline set of descriptors over the course of a sequence of camera images (or to adapt to changes in conditions, for example, changes in lighting), a black-box optimization method can be used as follows: (B1) Select one or more promising test sets of descriptors (for example, test sets that a black-box optimization method proposes according to, for example, an acquisition function). (B2) Evaluate the test set and the reference set by identifying a measure of accuracy for the test set and the reference set (for one camera image or across multiple camera images). (B3) If the accuracy measure for the test set is better than the accuracy measure for the baseline set, the test set becomes the new baseline set (i.e., the previous baseline set is updated to the test set).

[0066] Optimization begins with an initial set of criteria for the descriptors, which serve as a baseline. Instead of fitting the descriptors completely freely, they can only be fitted within a limited range centered around the initial baseline. This ensures that the descriptors do not deviate too far from the initial baseline.

[0067] The control device 106 can perform the above-described method for adapting descriptors completely automatically, without the involvement of a human user (except, in some cases, defining the initial criteria at the start). Optimal performance is achieved by using camera images from the target application instead of a general dataset. Thus, according to one embodiment, the method is used online during the operation of the application (e.g., during the use of a robot to pick objects from a box) to automatically improve the quality of the results over time. For each newly received camera image in a sequence of camera images, the control device 106 can automatically select which set of descriptors yields the best performance and should be considered for that particular image. For example, the control device 106 will use the current reference set if it performs better than the current test set being evaluated for the current camera image, or vice versa.

[0068] Figure 4 illustrates the procedure described above.

[0069] First, descriptor 401 is manually defined as the initial criterion (indicated here by its position in the first descriptor image 402 belonging to the first camera image). In order to determine the precision measure for the (initial) set of criterions according to (A), the pairwise distances 403 of their positions are determined.

[0070] In the second camera image, the positions of the descriptors within the second descriptor image 404 belonging to it are shifted in this example due to detection errors. For example, descriptor 405 is assigned to the wrong location by the machine learning model due to light reflection.

[0071] Accordingly, a new descriptor 406 is selected for the descriptor set according to (B), and according to this new descriptor 406, the pairwise distance between the 3D locations of the places given by the descriptors matches the identified distance 403 better, i.e., the shape of the identified region matches the original shape better. This fit (or test of possible fits) is repeated for new camera images (e.g., for received camera images in a sequence of camera images).

[0072] In summary, various embodiments provide a method as shown in Figure 5.

[0073] Figure 5 shows a flowchart 500 of a method for determining the location of an object from its camera image.

[0074] In 501, the location to which an object should be located is defined for each object type.

[0075] In 502, a standard is determined for the relative orientation of the location to be located.

[0076] In 503, a machine learning model is trained to map camera images representing objects of each object type to descriptor images, and the descriptor image to which the camera image should be mapped for the location of the object indicated by the camera image at a certain image position has a descriptor of the object's location at that image position.

[0077] In 504, the base set of descriptors is set to the initial set of descriptors.

[0078] In 505, a temporal sequence of camera images is received, each representing an object of a specific object type.

[0079] In 506, for each camera image, a trained machine learning model maps the camera image to a descriptor image, and by searching for a reference set of descriptors among the descriptors in the descriptor image, the location where the object should be located relative to the reference set of descriptors is identified, thereby determining the location on each object (for example, as described above in relation to Figure 3, to search for a location within descriptor values ​​that has a descriptor as close as possible to the descriptor of the reference set). This determines the location on the camera image plane, and it can then be located in 3D space, for example, by depth information or by solving a PnP (perspective n points) problem.

[0080] In 507, with respect to at least some of the camera images in the sequence of camera images, To determine the relative orientation of a localized location to a reference set, This involves comparing the relative orientation of the location to be located with the relative orientation of the located location relative to the set of references, By searching for the test set descriptor among the descriptors in the descriptor image, we can identify the location where the object should be located relative to the test set of the descriptor, To determine the relative orientation of the localized location to the test set, This involves comparing the relative orientation of the location to be localized with the relative orientation of the localized location relative to the test set, If the agreement between one or more camera images between the criterion for the relative orientation of the location to be localized and the relative orientation of the localized location identified for the test set is better than the agreement for the criterion set, update the descriptor of the criterion set to the descriptor of the test set. This will be implemented.

[0081] It should be noted that steps 506 and 507 are performed in parallel or alternately, for example, that step 507 is performed for each camera image (or after a predetermined number of camera images) during the localization process of step 506.

[0082] In other words, according to various embodiments, a machine learning model is trained to map camera images to descriptor images, and for each camera image, the descriptors provided by the trained machine learning model identify the locations on each object to be localized. The descriptors used to identify the locations to be localized are updated (i.e., fitted) during the course of the sequence of camera images, for example, to track changing light conditions, or to compensate for errors in the initial set of descriptors or for any lack of optimality in the initial set of descriptors.

[0083] An object is an instance of an object type, meaning that, for example, all objects have the same form specified by the object type. For example, an object is a component having a specific shape. However, differences in shape are possible as long as the topology of the objects is equivalent. For example, if the object type is "shoe" and the location to be specified is a point on the edge of the shoe's tongue, then each object can be a different shoe.

[0084] A machine learning model is, for example, a neural network. However, other machine learning models that are appropriately trained may also be used.

[0085] According to various embodiments, a machine learning model assigns descriptors to the pixels of an object (in the image plane of each camera image). This can be considered an indirect encoding of the object's surface topology. This connection between descriptors and surface topology can be explicitly performed by rendering to map the descriptors to the image plane. It should be noted that descriptors at the faces (i.e., points other than vertices) of an object model can be determined by interpolation. For example, if a face is given by three vertices of an object model, each having descriptors yi, y2, and y3, then at any point on this face, the descriptor y can be calculated as a weighted sum of these values ​​w1·y1+w2·y2+w3·y3. In other words, descriptors are interpolated at the vertices.

[0086] To generate image pairs for training data for a machine learning model, for example, an image of an object (e.g., an RGB image) containing (one or more) objects is mapped to an optimal (high-density) descriptor image, in the sense that it is generated by searching for descriptors that minimize the deviation in geometric properties (particularly the proximity of points of objects) between the object model and its representation (embedding) in descriptor space, given a known 3D (e.g., CAD) model and a known orientation in a (e.g., world) coordinate system. In practical use, since the search is limited to a specific search space, the theoretical optimal solution for minimization is generally not limited. Nevertheless, within the constraints of the actual application (available computational precision, maximum number of iterations, etc.), an estimate of the minimum value is determined.

[0087] A pair of training data images includes a training input image and a target image of an object, the target image being generated by projecting the descriptors of visible vertices in the training input image onto the training input image plane according to the orientation of the object in the training input image. The images and their associated target images are used for supervised training of a machine learning model.

[0088] Therefore, machine learning models are trained to recognize the unique features of (one or more) objects. This information can be used for various applications in robot control by evaluating the machine learning models in real time, for example, to predict the posture for grasping objects for assembly. It should be noted that the supervised training approach allows for the explicit encoding of symmetry information.

[0089] The method shown in Figure 5 can be implemented by one or more computers including one or more data processing units. The expression “data processing unit” can be understood as any type of entity that enables the processing of data or signals. For example, data or signals can be processed according to at least one (i.e., one or more) specific functions performed by the data processing unit. A data processing unit may include, or can be composed of, analog circuits, digital circuits, mixed-signal circuits, logic circuits, microprocessors, microcontrollers, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable gate arrays (FPGAs), integrated circuits, or any combination thereof. Any other methods for implementing each of the functions described in more detail later can also be understood as data processing units or logic circuits. Needless to say, one or more of the method steps described in detail herein can be implemented by a data processing unit via one or more specific functions performed by the data processing unit.

[0090] Various embodiments can, for example, receive and use sensor signals from various sensors such as (e.g., RGB or RDB-D) cameras, video sensors, radar sensors, LiDAR sensors, ultrasonic sensors, and thermal imaging technology sensors to acquire sensor data indicating an object. Embodiments for autonomously controlling a robot, such as a robotic manipulator, can be used to accomplish various operational tasks under various scenarios. In particular, each embodiment can be used, for example, to control and monitor the execution of operational tasks in an assembly line.

[0091] While specific embodiments have been illustrated and described herein, those skilled in the art should recognize that various alternative and / or equivalent implementations may be used in place of the specific embodiments illustrated and described without departing from the scope of protection of the present invention. This application is intended to cover any adapted or modified forms of the specific embodiments discussed herein. Accordingly, the present invention is intended to be limited only by the claims and their equivalents.

Claims

1. A computer-based method for determining the location of an object from a camera image of the object, the method being: This defines the location where an object should be located for an object type, and To determine a standard for the relative orientation of the location to be identified, The method involves training a machine learning model to map camera images, each representing an object of the aforementioned object type, to descriptor images, wherein the descriptor image to which the camera image is mapped, for the location of the object represented by the camera image at a given image position, has a descriptor of the location of the object at that image position. Setting the descriptor base set to the descriptor initial set, Receiving a temporal sequence of camera images, each representing an object of the aforementioned object type, For each camera image, The camera image is mapped to a descriptor image using a trained machine learning model. By searching for the descriptors of the reference set among the descriptors in the descriptor image, the location to be located of the object relative to the reference set of descriptors is identified. By doing so, the location to be located on each of the aforementioned objects is to be located. Includes, The method applies to at least some of the camera images in a sequence of camera images, To determine the relative orientation of the specified location with respect to the aforementioned reference set, Comparing the reference point for the relative orientation of the location to be located with the relative orientation of the located location with respect to the reference set, By searching for the descriptor of the test set among the descriptors in the descriptor image, the location of the object to be located relative to the test set of the descriptor is identified. To determine the relative orientation of the identified location with respect to the test set, Comparing the reference point for the relative orientation of the location to be located with the relative orientation of the located location with respect to the test set, If the agreement between one or more camera images between the reference for the relative orientation of the location to be located and the relative orientation of the located location relative to the test set is better than the agreement with respect to the reference set, the descriptor of the reference set is updated to the descriptor of the test set. Methods that include...

2. The aforementioned relative orientation has a pairwise distance between the location to be located or the location that has been located in three-dimensional space. The method according to claim 1.

3. The location to be located for the object type is defined on a reference camera image of the object of the object type, The aforementioned reference camera image is supplied to the machine learning model. The reference set of descriptors is set to the descriptors of the locations to be localized in the descriptor image output by the machine learning model for the reference camera image. The method according to claim 1 or 2.

4. The test set is selected within a limited range of the initial set of descriptors. The method according to any one of claims 1 to 3.

5. The test set of the aforementioned descriptors is identified by the covariance matrix adaptive evolutionary strategy method. The method according to any one of claims 1 to 4, including the method described in any one of claims 1 to 4.

6. A computer-based method for controlling a robot, the method being: Identifying the location of an object to be processed by a robot, in accordance with any one of claims 1 to 5, Identifying the pose of the object from the location identified and controlling the robot according to the identified pose, and / or identifying the region of the object from the location identified and controlling the robot according to the identified region, Methods that include...

7. A robot, A camera configured to provide a camera image of an object, A control device configured to carry out the method described in any one of claims 1 to 6 A robot that possesses the following features.

8. Having at least one actuator, The control device is configured to control the at least one actuator using the location-specified location. The robot according to claim 7.

9. A computer program that, when executed by a processor, includes instructions for causing the processor to perform the method according to any one of claims 1 to 6.

10. A computer-readable medium storing instructions for causing a processor to perform the method according to any one of claims 1 to 6, when executed by the processor.

Images