Servo system and processing method

Abstract

To provide a technique for improving accuracy or speed of visual servo.SOLUTION: In a servo system which has a robot comprising an end effector and performs visual servo, a processor which determines a servo track concerning the servo system and a start posture of the robot at a start time point of the visual servo has: planning means which determines a plurality of candidates of the start posture, and generates a plurality of tracks of the end effector via any one of the plurality of candidates when a posture of the robot changes; and determination means which determines a candidate with highest evaluation value, from among a plurality of candidates of the servo track corresponding to the plurality of tracks of the end effector, as the servo track, and determines a candidate of the start posture corresponding to the servo track as the start posture.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a processing device, a servo system, a processing method, and a program.

Background Art

[0002] There is a technology called visual servo that controls the operation of a robot based on visual information (images) acquired by a sensor. For example, by executing visual servo, the robot can realize processes such as placing a grasped object grasped by an end effector on a target object.

[0003] Patent Document 1 describes a technique for calculating the trajectory of a robot when performing visual servo based on feature amounts of an image acquired by a camera (sensor).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] On the other hand, in visual servo, the grasped object or the target object may disappear from the image acquired by the sensor, or they may overlap with each other. In this regard, Patent Document 1 only discloses calculating the trajectory of the robot while avoiding the movement prohibited region in the three-dimensional space, without considering the situation where the grasped object or the target object disappears from the image acquired by the sensor or they overlap with each other. For this reason, conventionally, there has been a problem that only a very small part of the grasped object or the target object is included in the image acquired by the sensor, resulting in a decrease in the accuracy and / or speed of visual servo.

[0006] Therefore, an object of the present invention is to provide a technique for improving the accuracy or speed of visual servoing.

Means for Solving the Problems

[0007] To achieve the above object, the present invention employs the following configuration.

[0008] That is, a processing device according to an aspect of the present invention has a sensor that images and detects an object and a robot including an end effector, and in a servo system that performs visual servoing, it is a processing device that determines at least one servo trajectory that is a trajectory related to the servo system and a start pose of the robot at the start time of the visual servoing. The acquisition means for acquiring information on the initial pose of the robot and information on the target pose of the robot at the end time of the visual servoing, 1) determining a plurality of candidates for the start pose, and 2) generating a plurality of trajectories of the end effector when the pose of the robot changes from the initial pose to the target pose via any one of the plurality of candidates. And determining means for determining, as the servo trajectory, the candidate having the highest evaluation value regarding the accuracy or speed of the visual servoing from among a plurality of candidates of the servo trajectories corresponding to each of the plurality of trajectories of the end effector, and determining the candidate for the start pose corresponding to the servo trajectory as the start pose.

[0009] According to such a configuration, the start pose of the robot at the start of visual servoing can be determined without the user manually setting it. That is, the burden on the user in visual servoing can be reduced. Further, a control trajectory having the highest evaluation value is selected as the servo trajectory from among a plurality of candidates for the servo trajectory. Therefore, the servo trajectory can be determined to be a trajectory along which the robot can move easily or / and a trajectory along which the sensor moves to a position where the object can be easily grasped. For this reason, for example, the probability of failure of visual servoing for performing an operation of connecting a gripping object to an object can be reduced. That is, the accuracy of visual servoing can be improved. And the time required for visual servoing can be reduced. ​

[0010] In the above processing device, the planning means may determine a plurality of candidates for the start posture from among the postures between the initial posture and the target posture. The posture between the initial posture and the target posture is, for example, the posture that would be passed through in the change when the planning means generates a change in the posture of the robot according to a predetermined algorithm so that the posture of the robot changes from the initial posture to the target posture. According to such a configuration, since the change from the initial posture to the target posture passing through the start posture can be determined to be a more efficient change, the robot can servo along a trajectory that is more efficiently achievable.

[0011] In the above processing device, it may include at least any one of the trajectory of the sensor, the trajectory of the relative position change between the end effector of the object, and the trajectory of the object in the captured image captured by the sensor.

[0012] In the above processing device, there is further simulation means for generating a plurality of candidates for the servo trajectory corresponding to each of the plurality of trajectories of the end effector, and simulating visual servoing according to each of the plurality of candidates for the servo trajectory. The simulation means corrects each of the plurality of candidates for the trajectory of the servo system based on the simulation result, and the determination means may determine the servo trajectory from among the plurality of candidates for the corrected servo trajectory. According to such a configuration, since the servo trajectory can be determined based on the result of the simulation of the visual servo, the servo trajectory corresponding to the trajectory that the end effector can actually move can be determined. That is, a servo trajectory that satisfies the operating constraints imposed on the end effector can be determined.

[0013] In the above processing device, the servo system has a first controller and a second controller. The first controller determines the position and orientation of the end effector corresponding to each of the plurality of trajectories of the end effector. The second controller controls the posture of the robot that realizes the position and orientation of the end effector determined by the first controller. The simulation means has a third controller that emulates the first controller and a fourth controller that emulates the second controller. The visual servo may be simulated using the third controller and the fourth controller. According to such a configuration, since the servo trajectory can be determined based on the result of the simulation corresponding to the second controller that controls the posture of the robot, a servo trajectory corresponding to a trajectory along which the robot can actually move can be determined. That is, a servo trajectory that satisfies the operating constraints imposed on the robot can be determined.

[0014] In the above processing device, it may further have evaluation means for evaluating each of the plurality of candidates of the servo trajectory.

[0015] In the above processing device, the evaluation means may calculate an evaluation value for each of the plurality of candidates of the servo trajectory based on a task score related to the movement of the robot and a sensing score related to the detection of the object when the visual servo is executed according to each of the plurality of candidates of the servo trajectory. According to such a configuration, a servo trajectory along which the robot can move easily and the entire object can be detected (imaged) can be determined.

[0016] In the above-described processing device, the task score may be a score based on at least one of the time required for the movement of the robot and the difference between the posture of the robot at the end of the visual servo and the target posture. If the task score is a score based on the time required for the movement of the robot, a servo trajectory that allows the robot to move faster can be determined. If the task score is the difference between the posture of the robot and the target posture at the end of the visual servo, a servo trajectory that allows the posture of the robot to change closer to the target posture can be determined.

[0017] In the above-described processing device, the servo system can detect the object, and the sensing score may be a score based on at least one of the total number of expected invalid detections during the execution of the visual servo, the distance between the detectable range, in which the servo system can detect an object, and the object, and the total number of expected detection errors. According to such a configuration, a servo trajectory that can detect (image) the object more accurately can be determined.

[0018] In the above-described processing device, the total number of expected invalid detections may be the number of times when at least one of the following occurs: in the number of times the sensor has performed imaging, less than a first ratio or more of the object is included in the field of view of the sensor or the detectable range, and in the imaging image captured by the sensor, another object overlaps with an area of at least a second ratio or more of the entire object. According to such a configuration, in an imaging image or the like, the number of occurrences of the object being located in a place where it cannot be detected (imaged) by the sensor, or another object overlapping with a predetermined ratio or more in front of the object (occlusion occurring) can be reduced, so that it is possible to suppress the detectable range of the object from becoming narrow.

[0019] In the above-described processing device, the planning means may determine a plurality of candidates for the start posture from among a plurality of postures of the robot corresponding to each of a plurality of position postures of the end effector located within a predetermined range. According to such a configuration, for example, the user can simply specify a predetermined range of an arbitrary size (for example, a spherical range), and the processing device can determine the start posture. Therefore, the processing burden on the user is small, and the processing device can determine an appropriate start posture.

[0020] In the above-described processing device, the planning means selects a plurality of candidates for the start posture from among a plurality of postures of the robot corresponding to each of a plurality of position postures of the sensor in which no other object is disposed between the sensor and the object when the sensor images, and the object is included in the field of view of the sensor. According to such a configuration, it is possible to determine, as the start posture, the posture of the robot corresponding to the position posture of the sensor in which occlusion does not occur and the entire object can be imaged. Therefore, at the start of visual servoing, the sensor can be arranged at a position posture where the object can be detected (imaged) more accurately.

[0021] Furthermore, the present invention may be a servo system characterized by including the above-described processing device, a robot having the end effector, a sensor that images the object, and a controller that operates the robot according to the servo trajectory determined by the processing device and starts the visual servo when the robot changes to the start posture. According to such a configuration, as described above, visual servoing can be executed using the robot and the sensor based on the appropriate servo trajectory determined by the processing device.

[0022] The present invention may be regarded as a device having at least a part of the above-described means, or may be regarded as an electronic device, a control system, an information processing system, an information processing device, a servo device, or a system. Good. Also, the present invention may be regarded as a control method, a processing method, a servo method, or a generation method including at least a part of the above processing. Further, the present invention can also be regarded as a program for realizing such a method and a recording medium (storage medium) in which the program is non-temporarily recorded. Note that each of the above means and processes can be combined with each other as much as possible to constitute the present invention.

Advantages of the Invention

[0023] According to the present invention, the accuracy or speed of visual servoing is improved.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4

Embodiments for Carrying Out the Invention

[0025] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

[0026] <Application Example> Hereinafter, in a servo system 1 that executes visual servo, before executing the visual servo, a technique for determining the start posture of the robot 10 at the start time of executing the visual servo and the appropriate trajectory of the servo system 1 when executing the visual servo (servo trajectory; trajectory of the sensor 20, the object 2, etc.) will be described. Specifically, the servo system 1 generates a plurality of candidates for the start posture. Also, the servo system 1 generates candidates for the appropriate trajectory when moving the robot 10 so as to change from the initial posture to a preset target posture (the posture of the robot 10 that is the target at the end of the visual servo) via any of the plurality of candidates for the start posture. Then, the servo system 1 selects one appropriate trajectory from among the plurality of candidates for the appropriate trajectory based on the evaluation values of these appropriate trajectories. As a result, one appropriate trajectory is selected, and the start posture corresponding to the appropriate trajectory is selected. Here, the evaluation value is an evaluation value related to the accuracy or speed of the visual servo. And the evaluation value is a value based on a task score related to the movement of the robot 10 and a sensing score related to the detection of the object 2.

[0027] According to this, the start posture of the robot 10 at the start of the visual servo can be determined without the user manually setting it. That is, the burden on the user in the visual servo can be reduced. Furthermore, the control trajectory with the highest evaluation value is selected as the appropriate trajectory from among the plurality of candidates for the appropriate trajectory. For this reason, the appropriate trajectory can be determined as a trajectory along which the robot 10 moves easily, or / and a trajectory along which the sensor 20 moves to a position where the object 2 can be easily grasped. For this reason, for example, the probability that a visual servo that performs an operation of connecting the gripping object 3 to the object 2 fails can be reduced. That is, the accuracy of the visual servo can be improved. And the time required for the visual servo can be reduced.

[0028] <Embodiment 1> Hereinafter, the servo system 1 according to Embodiment 1 will be described. The servo system 1 executes visual servo (operation of the robot 10 based on visual information). In Embodiment 1, the servo system 1 controls the posture of the robot 10 so that the gripping object 3 gripped by the robot 10 is connected to the object 2 (target object; predetermined object), while imaging the gripping object 3 and the object 2. Note that hereinafter, the "trajectory" may be a single-line segment when indicating the trajectory along which an object moves, or may be information indicating a plurality of positions and the movement order among the positions on the single-line segment. Also, in the present embodiment, the trajectory of a certain object indicates the position and posture when the object moves. That is, it may be so.

[0029] [Configuration of Servo System] First, with reference to the configuration diagram of FIG. 1, the configuration of the servo system 1 will be described. The servo system 1 includes a robot 10, a sensor 20, a control device 30, and a processing device 40. Also, the servo system 1 can be regarded as having the object 2 and the gripping object 3, or can be regarded as not having the object 2 and the gripping object 3.

[0030] The robot 10 controls the position and posture of the sensor 20 and the end effector 11 by changing its own posture. In the present embodiment, the posture of the robot 10 indicates the angles of the joints of the arm of the robot 10, the rotation angle of the end effector 11, etc. Therefore, in the present embodiment, the posture of the robot 10 and the position and posture of the end effector 11 correspond one-to-one. Also, the posture of the robot 10 is controlled by the control device 30. A sensor 20 is provided at the tip of the arm of the robot 10. Therefore, the robot 10 may be regarded as having the sensor 20. Note that in the present embodiment, the "position and posture" means position and posture, but may be position or posture as long as there is no technical contradiction. Also, the "position and posture" can be regarded as 6D information of a certain object (3D coordinates of a certain object and information representing the three axial directions of the object).

[0031] Robot 10 has an end effector 11 (gripping part) and a base 12. The end effector 11 grips the gripped object 3. The base 12 fixes the position of one end of the robot 10. Hereinafter, the posture of the robot 10 at the start of visual servo execution is referred to as the "start posture". Also, the posture of the robot 10 that is specified by the user and is the target of visual servo is referred to as the "target posture". Since this target posture is arbitrarily set by the user, it does not have to be a posture in which the gripped object 3 is connected to the object 2.

[0032] The sensor 20 captures images of the object 2 and the gripped object 3 to obtain a captured image. In this embodiment, the relative positional relationship between the sensor 20 and the gripped object 3 is fixed. For this reason, the sensor 20 includes the gripped object 3 in the field of view (FOV; Field Of View), which is an imaging range, as long as other objects do not overlap the gripped object 3. The captured image captured by the sensor 20 includes the object 2 and the gripped object 3, for example, as shown in FIG. 2A.

[0033] The control device 30 acquires a captured image from the sensor 20 and acquires information on the current posture of the robot 10 (such as information on the rotation angle of the motor) from the robot 10. Thereby, the control device 30 detects the object 2 and the gripped object 3 (their position postures) arranged in the detectable range 60 within the field of view based on the captured image and the posture of the robot 10 corresponding to the captured image. The detectable range 60 depends not only on the field of view of the sensor 20 but also on the distance from the sensor 20 as shown in FIG. 2B. The control device 30 can detect the position postures of the object 2 and the gripped object 3, for example, based on the difference between the captured images captured by the sensor 20 from two different position postures and the postures of the robot 10 corresponding to the two different position postures. Also, the control device 30 controls the posture of the robot 10 to control the position postures of the end effector 11 and the sensor 20. Thereby, the control device 30 performs control of the robot 10 (performs visual servo) so as to move the gripped object 3 to a desired position in the captured image.

[0034] The processing device 40 determines the start posture of the robot 10 when the servo system 1 executes visual servoing (operation of the robot 10 based on visual information). Further, the processing device 40 determines an appropriate trajectory (servo trajectory) of the servo system 1 that is a set of the trajectory of the sensor 20 (sensor trajectory), the trajectory indicating the relative position change with respect to the end effector 11 of the object 2 (object trajectory), and the trajectory of the visual feature of the object 2 in the captured image (visual trajectory). Here, the visual feature of the object 2 is, for example, one end, one corner, or the center position of the object 2. The servo system 1 starts visual servoing from the start posture of the robot 10 and continues to execute visual servoing while controlling the robot 10 according to the appropriate trajectory. The servo system 1 starts visual servoing from the start posture of the robot 10 and continues to execute visual servoing while controlling the robot 10 according to the appropriate trajectory.

[0035] [Configuration of Control Device and Processing Device] Hereinafter, with reference to FIG. 3, the detailed configurations of the control device 30 and the processing device 40 will be described. The control device 30 includes a simulator 310 and a controller 320. The processing device 40 includes an input / output unit 410, a planning unit 420, a simulation unit 430, and an evaluation unit 440.

[0036] The simulator 310 detects (estimates) the object 2 and the gripped object 3 (position and posture of the object 2, etc.) based on the captured image acquired from the sensor 20 and the posture of the robot 10.

[0037] The controller 320 includes a visual controller that controls the position and posture of the end effector 41 and a robot controller that controls the position and posture of the robot 10. The visual controller determines the posture position of the end effector 11 corresponding to the trajectory of the end effector 11. The robot controller outputs a command for driving the motor of the robot 10 to realize the change in the posture position of the end effector 11 determined by the visual controller.

[0038] Note that the controller 320 generates (outputs) a command for controlling the posture of the robot 10 based on the proper trajectory acquired from the processing device 40. Then, the controller 320 outputs a command to the sensor 20 so that the sensor 20 performs imaging during the period in which the posture of the robot 10 changes from the start posture. Through such processing, the controller 320 can execute visual servoing when the posture of the robot 10 changes to the start posture while moving the robot 10 based on the proper trajectory.

[0039] Note that when the object 2 moves or when the object 2 and the gripped object 3 cannot be connected at the target position and posture set by the user (when the target as visual servoing cannot be achieved), it is necessary to correct the operation of the robot 10. In this case, the controller 320 may generate a command for controlling the robot 10 so as to correct the operation of the robot 10 based on the position and posture of the object 2, the position of the object 2 in the captured image, and the like.

[0040] The input / output unit 410 is a user interface that receives (acquires) user operations (user inputs). The user can input (set) the initial posture and the target posture of the robot 10 via the input / output unit 410. Further, the user can input (set) a candidate range 50 (a predetermined range in a three-dimensional space), which is a range of the position and posture candidates of the end effector 11 when starting the execution of visual servoing as shown in FIG. 2C, via the input / output unit 410. Furthermore, the user can input information such as the parameters of the controller 320 and the virtual controller 431, the three-dimensional models of the object 2 and the gripped object 3, and the motion constraints of the robot 10 via the input / output unit 410. Also, the user can input information on an area (a movement prohibited area) where the robot 10 is prohibited from entering via the input / output unit 410. In the present embodiment, it is assumed that the candidate range 50 is a spherical area, but it may be an area of any shape. Further, the planning unit 420 may determine a sphere with a predetermined radius having, as a center point, the center of the position of the end effector 11 corresponding to the initial posture of the robot 10 and the center of the position of the end effector 11 corresponding to the target posture of the robot 10 as the candidate range.

[0041] Also, when it is assumed that the object 2 moves (for example, when it is assumed that the object 2 is mounted on a conveyor rotating at a constant speed), the user can input the motion information of the object 2 via the input / output unit 410. to the motion information of the object 2.

[0042] Furthermore, the input / output unit 410 outputs information on the proper trajectory (sensor trajectory, object trajectory, and visual trajectory) determined by the processing device 40 and information on the start posture to the control device 30.

[0043] The planning unit 420 determines a plurality of candidate poses of the start pose as a plurality of candidate poses such that the end effector 11 is located inside the candidate range 50. Note that the planning unit 420 may determine, as a candidate pose, a pose of the robot 10 such that the object 2 and the gripped object 3 do not overlap in the captured image captured by the sensor 20 (no other object exists between the sensor 20 and the object 2). Alternatively, the planning unit 420 may determine the candidate pose such that a part of the object 2 and the gripped object 3 is not located outside the visual field range and / or the detectable range 60. That is, the planning unit 420 may select a plurality of candidates for the start pose from among a plurality of poses of the robot 10 corresponding to each of a plurality of position and orientation poses of the sensor 20 in which the object 2 is included in the visual field range of the sensor 20.

[0044] Then, the planning unit 420 generates candidates for a plurality of trajectories (effector trajectories) of the end effector 11 when the pose of the robot 10 changes from the initial pose to the target pose via any one of the plurality of candidate poses. For example, when the planning unit 420 generates 10 candidate poses, the planning unit 420 repeats only 10 times the generation of candidates for the trajectory of the end effector 11 when the pose of the robot 10 changes from the initial pose to the target pose via one candidate pose. Note that any path calculation algorithm can be used to calculate the effector trajectory.

[0045] The simulation unit 430 generates candidates for visual trajectories, object trajectories, and sensor trajectories (these three types of trajectory candidates are referred to as "non-controlled trajectories" as a set) corresponding to each of the plurality of effector trajectory candidates. Then, the simulation unit 430 calculates (simulates) the movement of the robot 10 such that the object 2 passes through a plurality of points in each candidate visual trajectory. And the simulation unit 430 corrects the candidate visual trajectories based on the simulation results. For example, when the object 2 cannot pass through a plurality of points in the candidate visual trajectory due to the movement restriction of the robot 10, the simulation unit 430 corrects the candidate visual trajectories by correcting (shifting) the positions of the plurality of points. The simulation unit 430 also performs such correction for the candidate object trajectories and the candidate sensor trajectories. Hereinafter, the corrected non-controlled trajectories (a set of candidate visual trajectories, candidate object trajectories, and candidate sensor trajectories; candidate proper trajectories) are referred to as "controlled trajectories".

[0046] Further, the simulation unit 430 has a virtual controller 431. The virtual controller 431 is software that realizes the same processing as the controller 320 (an emulator of the controller 320), and the same parameters as those of the controller 320 are set. For this reason, the virtual controller 431 has a virtual visual controller that emulates (imitates) the visual controller of the controller 320 and a virtual robot controller that emulates (imitates) the visual controller of the controller 320. The virtual visual controller determines changes in the posture positions of the end effector 41 corresponding to each of the plurality of effector trajectories. The virtual robot controller controls the posture of the robot 10 to realize the changes in the posture positions of the end effector 41 determined by the virtual visual controller.

[0047] The simulation unit 430 uses a virtual visual controller and a virtual robot controller to simulate the movement of the robot 10 such that the object 2 passes through a plurality of points on each visual trajectory. By using the virtual visual controller and the virtual robot controller in this way, movements that the robot 10 cannot achieve can be excluded. excluded.

[0048] The evaluation unit 440 evaluates each of the plurality of control trajectories generated by the simulation unit 430. Specifically, the evaluation unit 440 calculates an evaluation value based on a task score related to the movement of the robot 10 and a sensing score related to the detection of the object 2 for each of the plurality of control trajectories. For this reason, the evaluation value is a value related to the accuracy or speed of visual servoing. For example, the evaluation value is larger as the task score is smaller and larger as the sensing score is smaller. Note that the task score may be any score as long as it takes a lower value for a control trajectory along which the robot 10 can move more easily, or a lower value for a control trajectory that can change the posture of the robot 10 closer to the target posture. Also, the sensing score may be any score as long as it takes a lower value as the detection accuracy of the object 2 (the degree of coincidence between the actual object 2 and the detection result) becomes higher.

[0049] The task score is, for example, a value based on at least one of an error (final posture error) between the final posture of the robot 10 and the target position posture when simulated and the total time (total movement time) when the robot 10 moves along the control trajectory. The task score is, for example, a value obtained by adding the final posture error and the total movement time with weights, or a value obtained by multiplying the final posture error and the total movement time.

[0050] The sensing score is, for example, a value based on at least one of the total number of expected invalid detections (total invalid detections), the distance between the detectable range 60 and the object 2 (object distance), and the total number of expected detection errors (total detection errors) during execution of visual servoing. Note that the object distance may be the average value, mode, median, minimum value, or maximum value of the distance between the center or outer periphery of the detectable range 60 and the center or outer periphery of the object 2 during execution of visual servoing. The sensing score is, for example, a value obtained by adding the total invalid detections, the object distance, and the total detection errors with a weight, or a value obtained by multiplying the total invalid detections, the object distance, and the total detection errors.

[0051] The total number of invalid detections is, for example, the number of times that the sensor 20 has performed imaging in which either the object 2 is not included in the detectable range 60 or the field of view, or another object is placed between the sensor 20 and the object 2 (occlusion has occurred). The total number of invalid detections may also be the number of times that the sensor 20 has performed imaging in which either a first percentage or more of the object 2 is not included in the detectable range 60 or the field of view, or another object overlaps with an area of ​​a second percentage or more of the entire object 2 in the captured image captured by the sensor 20. For example, if a certain object does not overlap the object 2, then 2 cm of the object 2 is not overlapped. 2 When the object is captured in the image, the object is 2 cm away from the captured image. 2 1.2 cm, which is 60% (the second percentage) of 2 As described above, if an object overlaps with the target object 2, it can be said that an invalid detection has occurred. Note that instead of the total number of invalid detections, the reciprocal of the average value of the ratio of the area included in the detectable range 60 or the field of view range to the entire area of ​​the target object 2 may be used.

[0052] The detection error is, for example, a value based on the variance of visual features corresponding to the captured images at two time points. Specifically, the detection error is a value based on the variance of the positions of the visual features of the images obtained by projecting (projecting) the two captured images onto the surface of the object 2. Note that the detection error is not limited to this, and any value that increases as the detection accuracy of the sensor 20 (the degree of match between the actual object and the detected object) is lower may be used.

[0053] Further, the control device 30 and the processing device 40 can be configured by, for example, a computer including a CPU (processor), a memory, a storage, and the like. In that case, the configuration shown in FIG. 3 is realized by loading a program stored in the storage into the memory and the CPU executing the program in the RAM. Such a computer may be a general-purpose computer such as a personal computer, a server computer, a tablet terminal, or a smartphone, or may be an embedded computer such as an on-board computer. Alternatively, all or part of the configuration shown in FIG. 3 may be configured by an ASIC, an FPGA, or the like. Alternatively, all or part of the configuration shown in FIG. 3 may be realized by cloud computing or distributed computing.

[0054] [Determination Process of Optimal Trajectory] Subsequently, the determination process (determination method) of the optimal trajectory executed by the processing device 40 will be described with reference to the flowchart of FIG. 4.

[0055] In step S1001, the input / output unit 410 acquires user-defined information. The user-defined information includes information on a three-dimensional model (models of the robot 10, the sensor 20, the object 2, the grasped object 3, and other external environments), information on parameters of the controller 320 (visual controller and robot controller), and information on the size and shape of the candidate range 50. Further, the user-defined information includes, for example, information indicating the calculation methods of the above-described task score and sensing score, information on operation constraints (restrictions on the robot 10, the sensor 20, and other external environments), and information on the initial posture and the generator (software) when generating the effector trajectory.

[0056] Here, the operation constraints of the robot 10 are conditions such as the operable angles of the respective joints of the robot 10, the relationship between the angles of the respective joints and the end effector 11, the change speeds and change accelerations of the respective joints, the movable range of the end effector 11, and the torque of the motor of the robot 10. The operation constraints of the sensor 20 are conditions such as the field of view (FOV) of the sensor 20 and the detectable range 60.

[0057] In step S1002, the planning unit 420 determines a plurality of candidates for the start posture as a plurality of candidate postures such that the end effector 11 is positioned inside the candidate range 50 by using a generator for generating the start posture. Note that the planning unit 420 may further determine a plurality of candidate postures such that the end effector 11 is positioned inside the candidate range 50 from among the postures between the initial posture and the target posture of the robot 10. Further, the planning unit 420 determines, for example, based on the information of the three-dimensional model, a posture of the robot 10 such that the object 2 and the gripped object 3 do not overlap in the captured image captured by the sensor 20 (for example, such that no other object is disposed between the sensor 20 and the object 2) as a candidate posture. Alternatively, the planning unit 420 determines a posture of the robot 10 such that all of the object 2 and the gripped object 3 are positioned within the visual field range or the detectable range 60 as a candidate posture. Furthermore, the planning unit 420 may limit the candidate postures to postures that the robot 10 can change based on the operation constraint information, or may limit the candidate postures to postures where the robot 10 is not located in the movement prohibited area.

[0058] In step S1003, the planning unit 420 generates a plurality of candidates for the effector trajectory by using a generator for generating the effector trajectory. Here, each of the plurality of candidates for the effector trajectory is generated so as to correspond to the change in the posture of the robot 10 from the initial posture to the target posture via any one of the plurality of candidate postures. Further, the planning unit 420 generates a plurality of candidates for the effector trajectory, for example, based on the operation constraint information, so that the robot 10 does not move into the movement prohibited area. Note that the planning unit 420 may generate a plurality of effector trajectories corresponding to the change in the posture of the robot 10 from the initial posture to the target posture before generating the plurality of candidate postures. Then, the planning unit 420 may determine, as a plurality of candidates for the start posture (a plurality of candidate postures), postures of the robot 10 such that the end effector 11 is positioned inside the candidate range 50 for each of the plurality of effector trajectories.

[0059] In step S1004, the simulation unit 430 generates a plurality of non-control trajectories corresponding to each of the plurality of effector trajectories based on the three-dimensional model and the information on the constraint conditions. Then, the simulation unit 430 uses the virtual controller 431 to simulate the movement of the robot 10 such that a plurality of points (a plurality of position postures) among each non-control trajectory are passed by the sensor 20 and the object 2.

[0060] In step S1005, the simulation unit 430 generates a plurality of control trajectories. Specifically, for each of the plurality of non-control trajectories, the simulation unit 430 corrects each non-control trajectory so as to correspond to the movement of the robot 10 simulated in S1004, and generates a control trajectory (a candidate for the proper trajectory) corresponding to the non-control trajectory.

[0061] In step S1006, the evaluation unit 440 evaluates each of the plurality of control trajectories. Specifically, the evaluation unit 440 calculates an evaluation value based on the above-described task score and sensing score for each of the plurality of control trajectories.

[0062] In step S1007, the planning unit 420 selects the one with the highest evaluation value among the plurality of control trajectories as the proper trajectory. For example, the planning unit 420 may select, as the proper trajectory, the one with the highest evaluation value among the control trajectories in which the task score is smaller than the first threshold and the sensing score is smaller than the second threshold. According to this, it can be ensured that the task score and the sensing score are smaller than a predetermined value. Further, the planning unit 420 selects the candidate posture corresponding to the proper trajectory (the posture of the robot 10 at the start point of the visual servo when the servo system 1 is executed according to the proper trajectory) as the start posture.

[0063] In this way, the processing device 40 generates (determines) an appropriate trajectory and a start posture. According to this, the start posture of the robot 10 at the start of visual servoing can be determined without the user manually setting it. That is, the burden on the user in visual servoing can be reduced. And the processing device 40 can set the posture where occlusion does not occur, the posture where the object 2 is located in the visual field range or the detectable range 60 as the start posture. That is, the processing device 40 can determine a suitable start posture for performing visual servoing.

[0064] Furthermore, the processing device 40 selects the control trajectory with the highest evaluation value from among a plurality of control trajectories as the appropriate trajectory. And the evaluation value is a value related to the accuracy and / or speed of visual servoing. For this reason, the appropriate trajectory can be a trajectory along which the robot 10 can move easily, or / and a trajectory along which the sensor 20 moves to a position where it is easy to grasp the object 2. That is, the probability that the visual servoing for connecting the gripping object 3 to the object 2 fails can be reduced (the accuracy of visual servoing is improved). And the time required for visual servoing can be reduced.

[0065] Note that in the above, the servo system 1 detects the position and posture of the object 2 and the gripping object 3 based on the captured images between two time points in the control device 30, but this is not the only case. For example, if the sensor 20 is a three-dimensional sensor, the sensor 20 itself may detect the position and posture of the object 2 and the gripping object 3.

[0066] Also, in this embodiment, the effector trajectory, the sensor trajectory, etc. are assumed to be the trajectories of the end effector 11 and the sensor 20 when the posture of the robot 10 changes from the initial posture to the target posture via the start posture. However, the effector trajectory, the sensor trajectory, etc. may be the trajectories of the end effector 11 and the sensor 20 when the posture of the robot 1 0 changes from the start posture to the target posture. This is because visual servoing is not executed during the period when the robot 10 changes from the initial posture to the start posture.

[0067] Also, in the proper trajectory, the control trajectory, and the non-control trajectory, instead of the sensor trajectory, an effector trajectory corresponding to the sensor trajectory may be used. This is because, in the present embodiment, the relative positional relationship between the sensor 20 and the end effector 11 is fixed. Note that instead of the sensor trajectory, the posture change of the robot 10 may be used.

[0068] Furthermore, when the object 2 does not move during visual servoing, in the proper trajectory, the control trajectory, and the non-control trajectory, the object trajectory is unnecessary. This is because the positional relationship between the end effector 11 and the object 2 can be calculated from the sensor trajectory.

[0069] Note that when performing visual servoing to change the grasped object 3 to a target position and posture on the two-dimensional space of the captured image, a visual trajectory is necessary in the proper trajectory, the control trajectory, and the non-control trajectory as described above. On the other hand, when performing visual servoing to change the grasped object 3 to a target position and posture in the three-dimensional space, a visual trajectory is not necessarily required in the proper trajectory, the control trajectory, and the non-control trajectory. That is, if the proper trajectory (servo trajectory), the control trajectory, and the non-control trajectory include the trajectory of at least one component (object; member) related to the servo system 1, there is a possibility of obtaining the same effect as in the present embodiment.

[0070] Note that the interpretation of the claims is not limited only by the matters described in the embodiments. The interpretation of the claims also includes the scope described so that those skilled in the art can recognize that the problems of the invention can be solved by considering the common general knowledge in the art at the time of filing.

[0071] (Appendix 1) In a servo system (1) that performs visual servoing and has a sensor (20) for imaging and detecting an object (2) and a robot (10) including an end effector (11), A processing device (40) that determines at least one servo trajectory related to the servo system (1) and the start posture of the robot (10) at the start point of the visual servo, An acquisition means (410) for acquiring information on the initial posture of the robot (10) and information on the target posture of the robot (10) at the end point of the visual servo, 1) Determine a plurality of candidates for the start posture, and 2) Generate a plurality of trajectories of the end effector (11) when the posture of the robot (10) changes from the initial posture to the target posture via any one of the plurality of candidates. Planning means (420), From among a plurality of candidates for the servo trajectory corresponding to each of the plurality of trajectories of the end effector (11), determine the candidate with the highest evaluation value regarding the accuracy or speed of the visual servo as the servo trajectory, and determine the candidate for the start posture corresponding to the servo trajectory as the start posture. Determining means (420), A processing device (40) characterized by having the above.

[0072] (Appendix 2) Regarding a servo system (1) having a sensor (20) that images and detects an object (2) and a robot (10) equipped with an end effector (11) and performing visual servo, A processing method for determining at least one servo trajectory related to the servo system (1) and the start posture of the robot (10) at the start point of the visual servo, An acquisition step (S1001) of acquiring information on the initial posture of the robot (10) and information on the target posture of the robot (10) at the end point of the visual servo, 1) Determine a plurality of candidates for the start posture, and 2) Generate a plurality of trajectories of the end effector (11) when the posture of the robot (10) changes from the initial posture to the target posture via any one of the plurality of candidates. Planning steps (S1002, S1003), A determination step (S1007) of determining, from among a plurality of candidates for the servo trajectory corresponding to each of the plurality of trajectories of the end effector (11), the candidate having the highest evaluation value regarding the accuracy or speed of the visual servo as the servo trajectory, and determining a candidate for the start posture corresponding to the servo trajectory as the start posture. A processing method characterized by having the above.

Explanation of symbols

[0073] 1: Servo system, 2: Object, 3: Grasped object 10: Robot, 11: End effector, 12: Base 20: Sensor, 30: Control device, 40: Processing device 310: Simulator, 320: Controller 410: Input / output unit, 420: Planning unit, 430: Simulation unit 431: Virtual controller, 440: Evaluation unit

Claims

1. A servo system that performs visual servoing, comprising a sensor that images and detects an object, and a robot having an end effector, a processing device that determines a servo trajectory, which is a trajectory of the robot from an initial posture to a target posture, and a start posture, which is the posture of the robot at a point in the middle of the servo trajectory at which visual servoing of the robot starts, and a controller of the robot, wherein the processing device has an acquisition means for acquiring information on the initial posture and the target posture of the robot, a planning means for: 1) determining a plurality of candidates for the start posture; and 2) generating a plurality of trajectories of the end effector when the posture of the robot changes from the initial posture to the target posture via any one of the plurality of candidates, and a determination means for determining, as the servo trajectory, a candidate having the highest evaluation value regarding the accuracy or speed of the visual servoing from among a plurality of candidates for the servo trajectory corresponding to each of the plurality of trajectories of the end effector, and determining a candidate for the start posture corresponding to the servo trajectory as the start posture, wherein the controller performs control to change the posture of the robot according to the servo trajectory determined by the processing device, and controls the posture of the robot without executing the visual servoing during a period from the initial posture of the robot to the change to the start posture, and starts the visual servoing when the posture of the robot reaches the start posture. A servo system characterized by this.

2. The planning means determines a plurality of candidates for the start posture from among postures between the initial posture and the target posture. The servo system according to claim 1, characterized by this.

3. The servo trajectory includes at least any one of a trajectory of the sensor, a trajectory of a relative positional change between the end effector of the object, and a trajectory of the object in an imaging image imaged by the sensor. The servo system according to claim 1 or 2, characterized by this.

4. The servo system further has a simulation means for generating a plurality of candidates for the servo trajectory corresponding to each of the plurality of trajectories of the end effector, and simulating visual servoing according to each of the plurality of candidates for the servo trajectory. ​ The simulation means corrects each of a plurality of candidates for the trajectory of the servo system based on the simulation result, The determination means determines the servo trajectory from among the plurality of candidates for the corrected servo trajectory. The servo system according to any one of claims 1 to 3, characterized in that.

5. The controller has a first controller and a second controller, The first controller determines the position and orientation of the end effector corresponding to each of the plurality of trajectories of the end effector, The second controller controls the posture of the robot that realizes the position and orientation of the end effector determined by the first controller, The simulation means includes a third controller that emulates the first controller and a fourth controller that emulates the second controller, and uses the third controller and the fourth controller to simulate the visual servo. The servo system according to claim 4, characterized in that.

6. The servo system further includes evaluation means for evaluating each of the plurality of candidates for the servo trajectory. The servo system according to any one of claims 1 to 5, characterized in that.

7. The evaluation means calculates an evaluation value for each of the plurality of candidates for the servo trajectory based on a task score related to the movement of the robot and a sensing score related to the detection of the object when the visual servo is executed according to each of the plurality of candidates for the servo trajectory. The servo system according to claim 6, characterized in that.

8. The task score is a score based on at least one of the time required for the movement of the robot and the difference between the posture of the robot at the end point of the visual servo and the target posture. The servo system according to claim 7, characterized in that.

9. The servo system can detect the object, The sensing score is a score based on at least one of the total number of expected invalid detections during the execution of the visual servo, the distance between the detectable range in which the servo system can detect an object and the object, and the total number of expected detection errors. The servo system according to claim 7 or 8, characterized in that.

10. The total number of the expected invalid detections is the number in which at least one of the following occurs among the number of times the sensor has performed imaging: less than or equal to a first ratio of the object is included in the field of view range or the detectable range of the sensor, and other objects overlap with an area of at least a second ratio of the entire object in the captured image captured by the sensor. The servo system according to claim 9, characterized in that.

11. The planning means determines a plurality of candidates for the start posture from among a plurality of postures of the robot corresponding to each of a plurality of position postures of the end effector located within a predetermined range. The servo system according to any one of claims 1 to 10, characterized in that.

12. The planning means selects a plurality of candidates for the start posture from among a plurality of postures of the robot corresponding to each of a plurality of position postures of the sensor in which no other object is disposed between the sensor and the object when the sensor performs imaging and the object is included in the field of view range of the sensor. The servo system according to claim 11, characterized in that.

13. Regarding a servo system that performs visual servoing and has a sensor that images and detects an object and a robot equipped with an end effector, A determination process for determining a servo trajectory that is a trajectory of the robot from an initial posture to a target posture, and a start posture that is a posture of the robot at a point in the middle of the servo trajectory and at which visual servoing of the robot starts, A control process for performing control to change the posture of the robot according to the determined servo trajectory, and a processing method including: The determination process includes: An acquisition step of acquiring information on the initial posture and information on the target posture of the robot, 1) determining a plurality of candidates for the start posture, and 2) a planning step of generating a plurality of trajectories of the end effector when the posture of the robot changes from the initial posture to the target posture via any one of the plurality of candidates. A determination step of determining, as the servo trajectory, a candidate having the highest evaluation value regarding the accuracy or speed of the visual servo from among a plurality of candidates of the servo trajectory corresponding to each of the plurality of trajectories of the end effector, and determining, as the start posture, a candidate for the start posture corresponding to the servo trajectory; In the control process, the posture of the robot is controlled without executing the visual servo during a period from the initial posture to the change to the start posture of the robot, and the visual servo is started when the posture of the robot reaches the start posture.

Images