Modeling objects using robot-guided cameras

Abstract

Method for modeling an object (1) using a camera (3) guided by a robot (2), comprising the steps: - Aligning (S30) the robot-guided camera with a target point; - Moving (S40) the robot-guided camera into different poses relative to the object while maintaining alignment with the target point; - Taking (S50) pictures of the object with the camera in different poses; - Saving (S50) pose data that depends on the specific pose, and image data that depends on an image taken in that pose; and - Determining (S60, S70) a model of the object based on the stored pose and image data.

Description

[0001] The present invention relates to a method, system and computer program product for modeling an object using a camera guided by a robot.

[0002] DE 10 2017 007 737 A1 relates to a method for capturing an image of a plant with a sensor device, in which the sensor device is guided around the plant with an arm of a robot.

[0003] DE 10 2016 223 665 A1 relates to a method for determining the position of a manipulator, wherein the manipulator has a manipulator arm on which at least one camera is set up for optical recordings, wherein the method is characterized by the steps: moving the camera into the vicinity of at least one reference object by the manipulator, recording at least two images of the reference object by means of the camera from different perspectives, determining the position of the manipulator relative to the reference object on the basis of the at least two recorded images, wherein the camera is part of a smartphone.

[0004] DE 11 2016 002 057 T5 relates to a motorized mobile platform comprising an articulated robot arm and a triangulation scanner for performing three-dimensional measurements, wherein the robot arm and the triangulation scanner are detachably coupled by connecting pieces.

[0005] US 2015 / 0 000 410 A1 relates to a method and a device for scanning an object, wherein two virtual, orthogonal axes are positioned on a surface of the object, a scanning path of a moving probe is controlled as a function of the two virtual, orthogonal axes, the scanning path can comprise a plurality of probe positions which are determined according to a desired coverage of the object, a single probe can be used or optionally a pair of probes or an arrangement of probes can be used, wherein optionally the probes are mounted on a multi-axis movable carrier.

[0006] EP 2 831 539 B1 relates to a detection probe for detecting the size of an object by irradiating the object with light and detecting reflected light, comprising: a detection unit comprising an imaging sensor and a sensor lens arrangement for detecting the reflected light, a light projection unit comprising a light source for generating light and source optics for focusing the light and light plane generation optics for generating a light plane for irradiating the object, and an adjustment mechanism contained in the light projection unit for adjusting the position and / or orientation of the light plane relative to the detection unit so that it coincides with an object-side imaging plane of the sensor lens arrangement, and wherein the adjustment of the light plane is fixable for operational stability, wherein the sensor lens arrangement comprises a plurality of lenses,wherein all of the lenses in the sensor lens assembly are arranged in a fixed, non-adjustable alignment with each other, the detection unit and the light projection unit are attached to a rigid base plate, the sensor lens assembly is rigidly and non-adjustably attached to the rigid base plate, and this rigid base plate is designed for mounting the detection probe on a locator.

[0007] EP 2 010 863 B1 relates to a scanning probe for recording data from a plurality of points on the surface of an object by irradiating the object with a light stripe and detecting the light reflected from the object surface, the scanning probe comprising: (a) a stripe generating means for generating and emitting a light stripe; (b) a camera comprising an image sensor having a pixel array for detecting the light beam reflected from the object surface;(c) means for adjusting the intensity of the light strip during the recording of an individual image section as a function of the intensities detected by the camera, wherein: the camera and / or the strip generating means are arranged such that during the recording of the image section different subgroups of the image sensor pixels detect reflected light from the strip at different times, the different subgroups corresponding to different positions on the detected light strip, the means for adjusting the light intensity being arranged such that the intensity of a light beam for a pixel subgroup is adjusted as a function of the detected intensity of the previous pixel subgroup;

[0008] The object of the present invention is to improve the modeling of objects.

[0009] This object is achieved by a method having the features of claim 1. Claims 8 and 9 protect a system or computer program product for implementing a method described herein. The subclaims relate to advantageous developments.

[0010] According to one embodiment of the present invention, a method for modeling an object using a camera guided by a robot comprises the steps: - Aligning the robot-guided camera (by means of the robot) to a target point, in particular a predetermined and / or object-fixed one; - Moving the robot-guided camera (using the robot) into different poses relative to the object while maintaining the orientation (of the camera) to the target point; - Taking pictures of the object with the camera in these different poses; - Save - of pose data that depend on the respective pose, in one embodiment specify or define the respective pose and / or are determined on the basis of measured joint or axis positions of the robot, and - from image data that depend on an image taken in this or the respective pose, in one embodiment, are determined, in particular calculated, on the basis of the taken image; and - Determining, in one embodiment calculating, a model of the object, in one embodiment two- or three-dimensional, in one embodiment generating a new model or modifying an existing model, on the basis of or depending on these stored poses and image data, wherein in one embodiment image data which depend on an image taken in a pose are or are respectively assigned to the pose data which depend on this pose.

[0011] In one embodiment, this allows a model of the object to be advantageously determined, particularly quickly and / or precisely. A pose in one embodiment comprises a one-, two-, or three-dimensional position and / or a one-, two-, or three-dimensional orientation.

[0012] In one embodiment, the robot-guided camera is moved into one or more of the poses by applying forces to the robot; in another embodiment, this is done manually by an operator. In one embodiment, this allows a model of the object to be advantageously determined, particularly quickly and / or precisely.

[0013] Additionally or alternatively, in one embodiment, the robot-guided camera is automatically moved into one or more of the poses, in one embodiment without external forces exerted on the robot or without movement command inputs from an operator. This allows, in one embodiment, a model of the object to be advantageously determined, particularly quickly and / or precisely.

[0014] In one embodiment, when or for moving the robot-guided camera into one or more of the poses, the robot's drives are controlled by means of or on the basis of the Operational Space Framework and / or on the basis of a predetermined minimum distance of the robot and / or the camera from the target point, in particular in such a way or with the proviso that the robot and / or the camera maintain a predetermined minimum distance from the target point, and / or on the basis of predetermined joint limits of the robot, in particular in such a way or with the proviso that the robot maintains predetermined joint limits. In one embodiment, this allows a model of the object to be advantageously determined, in particular quickly and / or precisely.

[0015] The Operational Space Framework uses an (operational) coordinate system to describe the position and orientation of an operational point in a reference system as a function of a task, with the space of this task being referred to as the operational space. Accordingly, only the coordinates required to describe the task need to be specified in the operational space.

[0016] Tasks can be described in particular in the Cartesian or workspace and / or in the joint coordinate space of the robot.

[0017] Multiple simultaneous tasks can be implemented, in particular, through multiple operational spaces. Additionally or alternatively, tasks in an implementation can be described as operational forces, for example, as forces to move an end effector to a target position and to drive or hold joints away from their joint limits. These operational forces can be mapped to forces, in particular (torsional) moments, in the joint coordinate space using a Jacobian matrix to control the robot. In this context, closed-loop control is also generally referred to as control.

[0018] The dynamic behavior of a point (“operational point”) in the so-called operational space can be determined in particular by ft=Λt(xt)x¨t+μt(xt,x˙t)+pt(xt) with the vector of (virtual) operational forces in the operational space f t , the symmetric inertia or mass matrix Λ t (x t), the vector of centrifugal and Coriolis forces µ t (x t ,ẋ t ) and the vector of gravitational forces p t (x t ) (all in the Operational Space). In addition to the Operational Space Framework, reference is made to Khatib, O., A unified approach for motion and force control of robot manipulators: The operational space formulation, IEEE Journal on Robotics and Automation, 1987, 3(1), pp. 43-53, and the content of this article is incorporated in its entirety into the present disclosure.

[0019] In the joint coordinate space, the dynamic behavior of the robot can be described in particular by: τ=M(q)q¨+c(q,q˙)+g(q) with the inertia or mass matrix M(q), the vector of centrifugal and Coriolis forces c(q,q̇), the vector of gravitational forces g(q), and the generalized forces, especially driving forces, τ in the joint coordinate space. Antiparallel force pairs or (torsion) moments are also referred to as forces in this context.

[0020] Using the Jacobian matrix J t results as a relationship between the operational forces f t and the joint or drive forces, especially torques τ: τ=JtTft

[0021] If the robot is redundant with respect to the task or the dimension of the minimum or joint coordinates q is larger than the dimension of the task coordinates x t is (dim(q) > dim(x t )), movements or (driving) forces or moments in the null space of JtT, be added without affecting the resulting forces at the operational point: τ=JtTft+Ntτ0 with Nt=I−JjTJ¯tT with the projection N t into the null space of the (main) task, where τ0 is an arbitrary generalized joint, in particular driving force, in particular moment vector, which is projected into the null space of the (main) task, and J¯tT the dynamically consistent generalized inverse, which minimizes the instantaneous kinetic energy of the robot: J¯t=M−1JjTΛt

[0022] This results in: Λt=(JtM−1JtT)−1 μt=J¯tTc−ΛtJ˙tq˙ pt=J¯tTg

[0023] The generalized joint, in particular driving force, in particular moment vector τ0 in (4) can be used for a second or subordinate task, which is (only) completely executed in the null space if it does not collide or interfere with the primary task, which is determined by the force vector f t is carried out, i.e., which has a higher priority. This makes it possible to implement a prioritization or prioritized structure for several tasks: τ1=J1Tf1+N1τ2τ2=J1Tf2+N2τ3⋮τk−1=Jk−1Tfk−1+Nk−1τkτk=JkTfk where the index is from the highest priority (Index1) to the lowest priority (Index k ), ie τ1 is the force vector for the task with the highest priority, which is executed if the kinematics or dynamics of the robot allow it, and τ kis the force vector for the lowest priority task, which will only be completed if enough coordinates are available in null space.

[0024] The task of positioning the end effector or a fixed point, in particular a view or alignment point, in particular a focus point of the camera, at the Cartesian (target) position x E = [x,y,z] T can be implemented in particular by a virtual spring-damper system: f*=MΔx¨E+DΔx˙E+KΔxE, with the difference Δx E between the desired or target position and the current Cartesian position of the end effector (x E,d - x E ), the mass matrix M , damping matrix D and stiffness or spring matrix K of the virtual spring-damper system, where M can be set equal to zero, in particular due to noise, so that: fpos*=kp(xE,d−xE)+kd(x˙E,d−x˙E), with the commanded force vector ft*, which corresponds to or represents the task, the spring constant k p and the damping constant k d .

[0025] The corresponding drive forces or moments for commanding or controlling the robot are determined by projection into the joint coordinate space: τpos=JposT(Λposfpos*+μpos+ppos).

[0026] In one embodiment, one or more (target) views of the object to be captured, in one embodiment, one or more (target) views required for a complete capture or mapping of all (currently) visible surfaces of the object are determined, and one or more poses are determined based on these determined view(s). In one embodiment, these determined poses are automatically approached, or the robot-guided camera is automatically moved into these poses. In one embodiment, this allows a model of the object to be advantageously determined, particularly quickly and / or precisely.

[0027] In one embodiment, the target point is determined based on at least one image captured with the camera, in one embodiment based on a center, in another embodiment based on a center of gravity, of a point cloud determined based on the captured image(s). In one embodiment, this allows a model of the object to be advantageously determined, particularly quickly and / or precisely.

[0028] In one embodiment, when determining image data, a background in the captured image is at least partially masked out, in one embodiment by means of segmentation, for example, a RANSAC-based plane fitting. In one embodiment, this allows a model of the object to be advantageously determined, particularly quickly and / or precisely.

[0029] The camera is either a 2D or 3D camera, and the images taken are 2D or 3D images of the object, respectively.

[0030] In one implementation, limited hand guidance can be used, maintaining orientation to the target point or keeping the object in view of the camera. The operator can move the robot manually, and the robot automatically adjusts its orientation, always keeping the object in the center of the camera image and maintaining the minimum distance to the object.

[0031] In one embodiment, movements around the object can be planned and automated in such a way that only one Cartesian point (the flange or the wrist) needs to be programmed and the orientation is automatically adjusted.

[0032] The invention may, in one embodiment, offer one or more of the following advantages: - the operator only has to define the location of the object or the target point, which can also be done automatically using 3D image processing; - Workspace constraints can be automatically or implicitly considered and maintained. This can save programming effort, especially time, by reducing the complexity of planning. - used robot trajectories can be saved and reused, which can make offline programming unnecessary; - the operator does not have to worry about collision-free path planning by setting joint limits.

[0033] The use of the robot can improve the accuracy of the modeling because the camera poses are determined using the very precise forward kinematics and depend only on the camera calibration, which can also be very precise.

[0034] A means within the meaning of the present invention can be designed in hardware and / or software, in particular a processing unit, in particular a microprocessor unit (CPU), graphics card (GPU) or the like, which is preferably connected to a memory and / or bus system in terms of data or signals, and / or can have one or more programs or program modules. The processing unit can be designed to execute instructions implemented as a program stored in a memory system, to detect input signals from a data bus, and / or to output signals to a data bus. A memory system can have one or more, in particular different, storage media, in particular optical, magnetic, solid-state, and / or other non-volatile media. The program can be designed in such a way that it embodies the methods described here oris capable of executing, so that the processing unit can execute steps of such methods and thus, in particular, model an object, in particular, command or move the camera-guided robot accordingly. In one embodiment, a computer program product can have, in particular be, a storage medium, in particular a non-volatile one, for storing a program or with a program stored thereon, wherein executing this program causes a system or a controller, in particular a computer, to execute a method described here or one or more of its steps.

[0035] In one embodiment, one or more, in particular all, steps of the method are carried out completely or partially automatically, in particular by the system or its means.

[0036] In one embodiment, the system comprises the robot.

[0037] Further advantages and features emerge from the subclaims and the exemplary embodiments. The following shows, partly schematically: Fig. 1: a system modeling an object using a camera guided by a robot according to an embodiment of the present invention; and Fig. 2: a method for modeling the object using the camera guided by the robot according to an embodiment of the present invention.

[0038] Fig. Figure 1 shows a system modeling an object 1 using a camera 3 guided by a robot 2 according to an embodiment of the present invention. The robot is controlled by a controller 4, which executes the steps of the method, in particular commands the robot 2 or its drives.

[0039] In Fig. 1 is a robot base-fixed coordinate system 0 through its x- and z-coordinate axes °x, °z and a robot end-effector-fixed coordinate system EE through its y- and z-coordinate axes EE y, EE z indicated.

[0040] When guiding by hand, the operator should be able to change the orientation, whereby a target point in the center of the object 1 is held in the coordinate system EE fixed to the robot end effector in such a way that the robot end effector or the camera 3 attached to it can also be EE can move in the z-direction, ie, is also "free" in this direction. For this purpose, the Jacobian matrix and the Cartesian positions and velocities are described in the robot end-effector-fixed coordinate system, and the row of the Jacobian matrix in the z-direction is deleted:

[0041] If the Jacobian matrix is ​​initially described in the robot-based coordinate system 0, the Jacobian matrix must be rotated into the robot end-effector coordinates. To do this, the following steps are performed: 1. Calculate the rotation 0 R E from the robot base-fixed coordinate system 0 into the robot end-effector-fixed coordinate system EE using the forward kinematics or transformation of the robot. 2. Rotate the Jacobian matrix using the equation: EEJpos=[0RET000RET] 0Jpos 3. Using the first and second lines of EE J pos to define the task. 4. Determine the command vector in the robot end effector-fixed coordinate system: fpos*=kp(EExE,d−EExE)+kd(EEx˙E,d−EEx˙E) 5. Calculate the command torque for the task: τpos=EEJposT(Λposfpos*+μpos+ppos).

[0042] These include EE xE,d , EE x E , EE ẋ E [2x1] vectors in the x and y coordinates of the end effector, EE x E,d is a [2×1]-zero vector and Λ pos , µ pos and p pos are made using only the first two rows of EE J pos calculated.

[0043] A joint damping task with the lowest priority is used to dissipate energy and keep the system passive. This results in the final stack of tasks: τstack=τpos+Nposτ2 τ2=τdamp with τdamp=JdampT(Mfpos*+c+g), where JdampT is an [nxn] identity matrix with the number n of degrees of freedom of the robot.

[0044] Two higher priority tasks are included: • to maintain a specified minimum distance between the camera and the target point or object or not to fall below it; and • to comply with specified joint limits or not to exceed them, preferably to avoid them.

[0045] The highest priority is assigned to the task of maintaining or not exceeding the joint limits, or preferably avoiding them, and is calculated, for example, using saturation in joint space ("Saturation in Joint Space algorithm", SJS). For this purpose, reference is made to Osorio, Juan D. Muñoz; Fiore, Mario D.; Allmendinger, Felix: Operational Space Formulation Under Joint Constraints, ASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers, 2018, pages V05BT07A022-V05BT07A022, and the content of this article is incorporated in its entirety into the present disclosure.

[0046] The task of maintaining the specified minimum distance can be implemented as a Cartesian constraint, for example, using a saturation in Cartesian Space algorithm (SCS) and given the second highest priority (by using the rotated Jacobian matrix in SCS, it is possible to constrain the coordinates along the axis from the camera to the object). Thus, τ stack sent to the SCS algorithm and the torque output of the SCS is sent to the SJS algorithm.

[0047] This allows the operator to manually guide the robot while it performs the main task of maintaining the camera alignment to the target point.

[0048] In the null space of this main task, a further task can be defined: to move to a Cartesian point at a given maximum speed, preferably as described in the above-mentioned article by Khatib. The operational point moves in a straight line at maximum speed to a desired point, except during the acceleration and deceleration sections. The force is calculated as follows: fgoToPoint*=−kd(X˙op−vX˙op,d), where: v=min(1,VmaxX˙op,dTX˙op,d) and X˙op,d=kpkd(Xop,d−Xop)

[0049] The subscript op indicates that these positions and velocities are those of the operational point, which, for example, can be the sixth joint axis in the LBR iiwa. The end effector cannot be used for this task to avoid conflicts with the main task. The commanded torques for this task are: τgoToPoint=JgoToPointT(ΛgoToPointfgoToPoint*+μgoToPoint+pgoToPoint). where J goToPoint expressed or developed to the operational point, and only the rows of position coordinates are used. The new task stack is: τstack=τposNposτ2 τ2=τgoToPoint+NgoToPointτ3 τ3=τdamp

[0050] The operational point now moves to the desired point while the end effector points to a target point or target position. This improves programming because only the desired points, for example, for the sixth joint axis, need to be specified, and the operator does not have to worry about the desired orientation. The end effector or camera always points to the target point or object or is aligned with it. This task stack allows robot movements to be executed or programmed while images are being captured for modeling. The desired Cartesian points can also be predefined or entered using the described hand guidance.

[0051] The procedure for modeling object 1 includes the following steps (see Fig. 2): S10: the object 1 is (or is) placed on a table 5 or another flat surface so that the entire object is visible in the recorded two- or three-dimensional image. S20: Using a segmentation algorithm, e.g., RANSAC-based plane fitting, the table or flat surface is removed from the 3D point cloud so that the data only contains 3D points belonging to object 1. S30: a 3D center of the object is calculated and used as target position or point for the (orientation of) camera 3. S40: an operator (not shown) guides robot 2 manually around the object and attempts to capture its entire (currently visible) surface. S50: At different times, the current point cloud (which is determined from the camera images) and the corresponding or current camera pose are saved in the world or robot base coordinate system. S60: all stored point clouds are aggregated, for example by means of TSDF, as described in Curles, B., Levoy, M.: A Volumetric Method for Building Complex Models from Range Images, 1996, to which reference is made and the content of which is fully incorporated into the present disclosure. S70: A polygon mesh is calculated from the resulting point cloud and, if necessary post-processed, used or provided as a model of object 1, for example saved.

Claims

[1] Method for modelling an object (1) by means of a camera (3) guided by a robot (2), comprising the steps: - Aligning (S30) the robot-guided camera to a target point; - moving (S40) the robot-guided camera into different poses relative to the object while maintaining the orientation to the target point; - Taking (S50) pictures of the object with the camera in the different poses; - storing (S50) pose data that depend on the respective pose and image data that depend on an image taken in this pose; and - Determining (S60, S70) a model of the object based on the stored pose and image data. [2] Method according to claim 1, characterized by that the robot-guided camera is moved into at least one of the poses by applying forces to the robot. [3] Method according to one of the preceding claims, characterized bythat the robot-guided camera is automatically moved into at least one of the poses. [4] Method according to one of the preceding claims, characterized by that when moving the robot-guided camera into at least one of the poses, drives of the robot are controlled on the basis of the Operational Space Framework and / or a predetermined minimum distance of the robot and / or the camera to the target point and / or predetermined joint limits of the robot. [5] Method according to one of the preceding claims, characterized by that at least one view of the object to be captured is determined and at least one of the poses is determined on the basis of this determined view. [6] Method according to one of the preceding claims, characterized by that the target point is determined on the basis of at least one image taken with the camera (S30). [7] Method according to one of the preceding claims, characterized bythat when determining image data, a background in the recorded image is at least partially hidden (S20). [8] System (4) arranged to carry out a method according to one of the preceding claims. [9] Computer program product comprising a program code stored on a computer-readable medium for carrying out a method according to any one of the preceding claims.

Images