Robot control method

Abstract

The robot control method is a robot control method in a control device connected to a robot performing a task on a work object and sensors mounted on the robot. First, based on multiple environmental sensor information acquired from multiple positions and angles using sensors regarding the robot's surrounding environment, environmental information representing the robot's surroundings is calculated. Next, using sensors to acquire first sensor information, and executing the first step, based on target task information representing the robot's task on the work object, the environmental information, and the first sensor information, first control information is calculated to make the robot approach the task target, and the first control information is sent to the robot. After the first step, using sensors to acquire second sensor information, and executing the second step, based on the target task information and the second sensor information, second control information is calculated to make the robot approach the task target, and the second control information is sent to the robot.

Description

Technical Field

[0001] This disclosure relates to a robot control method that uses sensors mounted on the robot to perform operations. Background Technology

[0002] Patent Document 1 discloses a visual recognition device for controlling position and posture based on image data captured by a camera. In this disclosure, unrecognizable locations of the work object are pre-detected and designated as prohibited areas by visual servo-based position and posture control. Therefore, by controlling the movement to avoid these prohibited areas, the work object can be reliably identified while being moved and controlled to the target position.

[0003] Prior art literature

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2003-305675 Summary of the Invention

[0006] The purpose of this disclosure is to provide a robot control method that can perform tasks with accuracy even when it is impossible to detect the work object through sensors.

[0007] The robot control method in the first aspect of this disclosure is a robot control method in a control device connected to a robot performing work on a work object and sensors mounted on the robot. First, environmental information representing the robot's surrounding environment is calculated based on multiple environmental sensor information acquired from multiple positions and angles using sensors regarding the robot's surrounding environment. Next, first sensor information is acquired using sensors, and a first step is performed: based on target work information representing the robot's work target on the work object, the environmental information, and the first sensor information, first control information is calculated to make the robot approach the work target, and the first control information is sent to the robot. After the first step, second sensor information is acquired using sensors, and a second step is performed: based on the target work information and the second sensor information, second control information is calculated to make the robot approach the work target, and the second control information is sent to the robot.

[0008] Furthermore, the robot control method in the second aspect of this disclosure is a robot control method in a control device connected to a robot performing work on a work object and sensors mounted on the robot. First, environmental information representing the robot's surrounding environment is calculated based on multiple environmental sensor information acquired from multiple positions and angles using sensors regarding the robot's surrounding environment. Next, first sensor information is acquired using sensors, and a first step is performed: based on target work information representing the robot's work target on the work object, the environmental information, and the first sensor information, first control information is calculated to make the robot approach the work target. Furthermore, second sensor information is acquired using sensors, and a second step is performed: based on the target work information and the second sensor information, second control information is calculated to make the robot approach the work target. Further, a third step is performed: based on the first and second control information, third control information is calculated and sent to the robot.

[0009] According to the robot control method disclosed herein, even when it is impossible to detect the work object through sensors, the work can be performed while ensuring accuracy. Attached Figure Description

[0010] Figure 1 This is a system structure diagram showing the structure of the robot's control system in Embodiment 1.

[0011] Figure 2 This is a timing diagram showing the actions of the robot that calculates environmental information in Implementation 1.

[0012] Figure 3 This is a timing diagram showing the actions of the computational robot to obtain target task information in Implementation 1.

[0013] Figure 4 This is a timing diagram showing the control actions for implementing the robot's tasks in Implementation Method 1.

[0014] Figure 5 This is a block diagram showing the hardware structure of the robot's control system in Embodiment 1.

[0015] Figure 6 This is a flowchart illustrating the method for calculating environmental information in Implementation 1.

[0016] Figure 7 This is a schematic diagram illustrating environmental image information in Implementation 1.

[0017] Figure 8 This is a schematic diagram illustrating the environmental information of the robot in Embodiment 1.

[0018] Figure 9This is a flowchart illustrating the calculation of target task information in Implementation Method 1.

[0019] Figure 10 This is a schematic diagram illustrating the target state of the robot in Embodiment 1.

[0020] Figure 11 This is a schematic diagram showing the target image captured in Embodiment 1.

[0021] Figure 12 This is a schematic diagram illustrating the first target image information in Embodiment 1.

[0022] Figure 13 This is a schematic diagram showing the second target image information in Embodiment 1.

[0023] Figure 14 This is a flowchart illustrating the control actions of the robot in Embodiment 1.

[0024] Figure 15 This is a schematic diagram showing a camera image captured by the robot in its initial state in Embodiment 1.

[0025] Figure 16 This is a timing diagram showing the control actions for implementing the robot's tasks in Embodiment 2.

[0026] Figure 17 This is a block diagram showing the hardware structure of the robot control system 1 in Embodiment 2.

[0027] Figure 18 This is a flowchart illustrating the control actions of the robot in Embodiment 2. Detailed Implementation

[0028] [The process of realizing this disclosure]

[0029] Robots that replace humans in performing tasks are controlled by a control unit, handling tasks such as picking up and placing objects. In order to achieve precise control that adapts to the environment, feedback control is implemented using information acquired from sensors. The control unit controls the robot based on the sensor information obtained from the sensors. Thus, by using sensors, the robot can perform tasks that adapt to its environment.

[0030] However, in actual robot operations, the sensor data may not necessarily contain information about the object being worked on. For example, when using a camera as a sensor, the object needs to be captured in the images obtained from the camera in order for the robot to begin its work. Therefore, a dedicated operating environment design and physical equipment adjustments are required.

[0031] Without information about the object to be handled in the sensor data, it is impossible to control the robot based on captured images. For example, when grasping an object, the grasping position is determined by identifying the object based on sensor information. Therefore, if the object cannot be identified, the grasping position cannot be determined, and the task cannot be performed.

[0032] Therefore, in the following implementation, control is performed using not only information about the object being worked on, but also information about the worktable (objects existing in the robot's surrounding environment). Thus, even when the object being worked on is not present in the sensor information, the task can be performed while ensuring accuracy.

[0033] The following is a brief reference to the appendix. Figure 1 The implementation methods will be described in detail. However, sometimes detailed descriptions that are not necessary will be omitted. For example, detailed descriptions of matters that are already widely known or repetitive descriptions of substantially the same structures will be omitted. This is to avoid making the following description unnecessarily lengthy and to facilitate understanding by those skilled in the art.

[0034] Furthermore, the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand this disclosure, and are not intended to limit the subject matter described herein.

[0035] Furthermore, in the following embodiments, robot control using a camera as a sensor will be described.

[0036] [Implementation Method 1]

[0037] The following uses Figures 1-15 To illustrate implementation method 1.

[0038] [Explanation of Control System Diagram]

[0039] Figure 1 This is a system structure diagram showing the structure of the robot control system in Embodiment 1. The control system 1 in Embodiment 1 includes at least a robot 10, a control device 2, a threaded hole 11 (as the work object), a base plate 12 (as a worktable), and a wall 13. The control device 2 can connect to and communicate with the robot 10. Alternatively, the control device 2 can be built into the robot 10. Furthermore, the robot 10 includes a robot arm 101, a screwdriver 102 (as an end effector), and a camera 103 (as a sensor). The control system 1 is not limited to... Figure 1 The structure shown.

[0040] The robotic arm 101 has a mechanism with one or more joints. The robotic arm 101 can control the position and posture of the screwdriver 102. Alternatively, the robotic arm may not necessarily have joints. For example, it may have a telescopic mechanism. Furthermore, the robotic arm may use a structure that is generally acceptable to the user.

[0041] Screwdriver 102 is an end effector capable of inserting a screw into the threaded hole 11. Screwdriver 102 is mounted on the tip of robotic arm 101. For example, as... Figure 1 As shown, the orientation of the link that mounts the screwdriver 102 to the front end of the robot arm 101 is parallel to the orientation of the screwdriver 102. Furthermore, the end effector is not limited to the screwdriver 102. For example, a structure with one or more fingers can be used as the end effector. As another example, the end effector can be a soft, spherical object capable of grasping the work object in a way that encloses it. However, it is not limited to this; a typical end effector for the user can also be used.

[0042] Camera 103 is used as a sensor to acquire information about the environment surrounding robot 10. For example... Figure 1 As shown, camera 103 captures images of the threaded hole 11 (the work object) or substrate 12 and wall 13 (the worktable) by capturing images of the shooting area 1031. Camera 103 is mounted on the front end of robot arm 101. For example, as Figure 1 As shown, a camera 103 is mounted on a connecting rod on which a screwdriver 102 is mounted, allowing the camera to capture images of the part being worked on by the screwdriver 102. Alternatively, the camera 103 can be mounted outside the tip of the robotic arm 101. For example, it can also be mounted on a connecting rod at the base of the robotic arm 101 to capture images of the surrounding environment in a rotating manner. Furthermore, the camera 103 can also acquire information beyond the captured images. For example, the camera 103 can acquire its own parameters. For example, the camera 103 can be a depth sensor to acquire depth images. For example, the camera 103 can be a 3D distance sensor to acquire depth information. Not limited to these, the camera 103 can also be a camera that is generally suitable for the user.

[0043] The threaded hole 11 (work object) is the object that the robot arm 101 or screwdriver 102 performs work on when the robot 10 is working. For example, Figure 1 As shown, in the case of thread tightening operation, the robot 10 operates by inserting a screw into the threaded hole 11 using a screwdriver 102.

[0044] The base plate 12 and the wall 13 (workbench) are the work areas for objects existing in the environment surrounding the robot 10. For example, such as Figure 1 As shown, in the case of performing thread tightening operations, there is a base plate 12 with threaded holes 11 and a wall 13 surrounding the work area around the robot 10.

[0045] In addition, the environment refers to the place where the work is carried out, including the threaded hole 11 (the object to be worked on), the substrate 12, and the wall 13 (the worktable).

[0046] Here is another example of the work object and the worktable. In control system 1, when performing a transport operation on substrate 12, the robot 10 uses its two-fingered hand as an end effector to grasp substrate 12 and transport it to a designated location. In this case, the work object is substrate 12. Furthermore, the worktable is wall 13. Thus, even when performing operations for the same control system 1, the work object and the worktable may sometimes differ depending on the task at hand.

[0047] Additionally, the workbench may include objects unrelated to the task. For example, the workbench may include a ceiling. It may also include a separate table, different from the workbench where robot 10 is located. In other words, when an image is captured by camera 103, the workbench includes all objects within the capture area 1031.

[0048] [Instructions for the assignment]

[0049] In this embodiment, a thread-tightening operation of robot 10 is considered. Robot arm 101 is a 6-axis industrial robot with series-connected links. A screwdriver 102 is used as the end effector. The screw is attached to the front end of the screwdriver 102. Camera 103 acquires RGB images. The work object is a threaded hole 11, and the worktable consists of a substrate 12 and a wall 13. It is assumed that the substrate 12 and wall 13 have patterns from which feature points can be extracted through image processing. Figure 1 The pattern is illustrated simply in the diagram. Furthermore, the environment refers to the threaded hole 11, the substrate 12, and the wall 13.

[0050] When performing thread tightening operations, such as Figure 1 As shown, the robot 10 initially faces the wall 13. The control device 2 uses images captured by a camera to perform control, enabling the robot 10 to perform thread tightening operations. The robot 10's posture during thread tightening, i.e., the target state, is when the screwdriver 102 is perpendicular to the threaded hole 11 and positioned directly above it. A schematic diagram of the robot 10 in the target state is shown below. Figure 10 Described later.

[0051] [Explanation of the timing diagram]

[0052] The timing diagram illustrates the data exchange between operator S1, control device 2, robot 10, and camera 103. Operator S1 refers to the person operating robot 10. Furthermore, control device 2, robot 10, and camera 103 use a data-driven... Figure 1 The structure has been explained.

[0053] [Time series of computational environment information]

[0054] Figure 2 This is a timing diagram showing the actions of the robot that calculates environmental information in Implementation 1.

[0055] The control device 2 calculates environmental information based on environmental image information obtained by image processing of images captured from camera 103. Environmental image information refers to image information of the environment based on images captured by camera 103. Environmental information refers to 3D positional information representing features of the environment. Here, environmental information is calculated based on environmental image information; therefore, to accurately acquire environmental information, it is desirable to acquire multiple environmental image images from various positions / angles of camera 103. In this embodiment, environmental image information refers to a group of images capturing the threaded hole 11, substrate 12, and wall 13, and environmental information refers to a group of 3D points representing the 3D positions of the features of the threaded hole 11, substrate 12, and wall 13. Captured images are an example of sensor data, and environmental image information is an example of environmental sensor information.

[0056] The following is a timing diagram illustrating the calculation of environmental information based on environmental image information. First, operator S1 sends a command to control device 2 to acquire environmental information (S001). Additionally, operator S1 can define robot actions for acquiring environmental image information and send them to control device 2. Furthermore, to acquire environmental image information, operator S1 can dynamically send control information to control device 2 using a programmer or similar device. Moreover, operator S1 can also directly control the robot's movements manually. Thus, operator S1 can interactively input control information into control device 2 and operate robot 10.

[0057] After step S001, the control device 2 sends control information to the robot 10, including actions for acquiring environmental information, so that the robot 10 can move (S002).

[0058] Furthermore, the control device 2 sends a command to the camera 103 to acquire environmental image information (S003). Additionally, it is not necessary to mount the camera 103 on the robot 10 for taking pictures in order to acquire environmental images. That is, the operator S1 can also hold the camera 103 by hand to acquire images. Furthermore, the order of steps S002 and S003 can be reversed.

[0059] After steps S002 and S003, camera 103 acquires and sends the captured images to control device 2, which then stores them (S004). In step S004, since the camera 103 captures images during the robot 10's movement in step S002, control device 2 receives and stores one image from each of the multiple positions / poses captured by camera 103.

[0060] When robot 10 finishes the action specified in step S002, robot 10 sends a signal to control device 2 that the action of robot 10 has ended (S005).

[0061] After step S005, the control device 2 sends a command to the camera 103 to end the shooting (S006). After step S006, the camera 103 ends the shooting and sends a signal to the control device 2 to end the shooting (S007). At this time, the control device 2 acquires environmental image information obtained by the camera 103 comprehensively capturing the environment.

[0062] After step S007, control device 2 calculates environmental information based on the environmental image information obtained in step S004 (S008). Control device 2 stores the calculated environmental information. The method for calculating environmental information is through... Figure 6 The calculation process for environmental information will be described later.

[0063] After step S008, the control device 2 sends an environmental information acquisition signal to the operator S1 and notifies that the calculation of the environmental information has ended (S009). The operator S1 can receive the notification from the control device that the calculation of the environmental information has been terminated and execute the calculation of the target job information for the next job.

[0064] [Time sequence for calculating target job information]

[0065] Figure 3 This is a timing diagram illustrating the actions of calculating the target task information of the robot in Embodiment 1. The calculation of the target task information of robot 10 requires acquiring information based on... Figure 2 The execution will proceed after providing the environmental information.

[0066] The control device 2 acquires target operation information based on environmental information and target image information. Target image information refers to the image captured by camera 103 during the thread tightening operation. Target operation information is the information used by robot 10 during the thread tightening operation, calculated using the target image information and environmental information. This includes the position / pose of camera 103 during the thread tightening operation and the feature points of the threaded hole 11 in the target image information. Feature points are an example of feature information.

[0067] The following is a timing diagram illustrating the calculation of target task information based on target image information. First, operator S1 sends a command (Si01) to control device 2 to acquire target task information. Alternatively, operator S1 can define robot actions for acquiring target task information and send them to control device 2. Furthermore, to acquire target task information, operator S1 can dynamically send control information to control device 2 using a programmer or similar device. Additionally, operator S1 can directly manipulate the robot manually. Thus, operator S1 can interactively input control information into control device 2 and operate robot 10.

[0068] After step S101, the control device 2 sends a command (control information) to the robot 10 to acquire target image information, causing the robot 10 to move (S102).

[0069] After step S102, robot 10 sends a notification to control device 2 to end the action (S103). After step S103, the posture of robot 10 becomes the posture at which the task was completed.

[0070] After step S103, the control device 2 sends a command to the camera 103 to acquire target image information (S104). After step S104, the camera 103 acquires the target image information and sends it to the control device 2, which then stores the target image information (S105). The threaded hole 11 is captured in the target image information acquired from the camera 103. Furthermore, it is not necessary to mount the camera 103 on the robot 10 to capture the target image information. That is, the operator S1 can also hold the camera 103 by hand to acquire the image.

[0071] Following step S105, target operation information is calculated based on environmental information and target image information (S106). The method for calculating target operation information is as follows: Figure 9 The calculation process for the target task information will be described later.

[0072] After step S106, the control device 2 sends a target task information acquisition signal to the operator S1 and notifies that the calculation of the target task information has ended (S107). The operator S1 can receive the notification from the control device that the calculation of the target task information has been terminated and execute the control of the robot 10 as the next task.

[0073] [Timing of robot control]

[0074] Figure 4 This is a timing diagram illustrating the control actions for implementing the robot's tasks in Implementation Method 1. Robot control requires acquiring data based on... Figure 2Explanation of environmental information and based on Figure 3 The target task information will be explained and then executed.

[0075] The robot control in this embodiment is characterized by a two-stage control process, where a second step is performed after the first step. The first step consists of three processes: The first process acquires first image information from camera 103. The second process calculates first control information based on environmental information, the first image information, and target task information. The third process executes control based on the first control information. The second step also consists of three processes: The first process acquires second image information from camera 103. The second process calculates second control information based on the second image information and target task information. The third process executes control based on the second control information. Image information is an example of sensor information.

[0076] The control achieved through these two steps is based on information from a single target image. In other words, the results of control based on step 1 and step 2 are ideally the same. Step 1 differs from step 2 in that it uses not only information about the target object but also environmental information, thus enabling control even when the target object is not present in the captured image. This overcomes the limitation of step 2, which cannot control when the target object is not present in the captured image.

[0077] However, in this embodiment, when calculating environmental information, errors may occur during the transformation from a 2D image to 3D positional information. Therefore, transformation-related errors may arise in the control of the first step. On the other hand, unlike the first step, in the second step, the 2D image is not transformed and its features are used directly, thus avoiding the transformation-related errors described above. That is, compared to the first step, the second step achieves control with higher accuracy.

[0078] Therefore, in this embodiment, control implemented in step 2 is executed after control implemented in step 1. First, by executing control implemented in step 1, the task object is captured in the image even when it is not present, thus enabling step 2 to proceed. Next, considering errors in converting to environmental information, control implemented in step 2 is executed so that the robot 10 is precisely controlled to achieve the target task information even when features of the task object are present in the image. Thus, precise control can be performed regardless of the position of the task object.

[0079] In this embodiment, the first image information refers to the feature points of the image obtained by the camera 103 from capturing images of the environment. Furthermore, the second image information refers to the feature points of the threaded hole 11 in the image captured by the camera 103. Additionally, the first and second control information are control quantities for the joint angle positions of the robot 10, and the robot 10 uses the control information to perform joint angle position control.

[0080] First, operator S1 sends a command to control device 2 to start robot control (S201).

[0081] After step S201, the control device 2 sends a command to the camera 103 to acquire the first image information (S202). After step S202, the camera 103 acquires the first image information and sends the first image information to the control device 2, and the control device 2 stores the first image information (S203).

[0082] After step S203, control device 2 calculates first control information based on environmental information, target operation information, and first image information (S204). The method for calculating the first control information is as follows: Figure 14 The calculation process for the robot's control actions will be described later.

[0083] After step S204, the control device 2 sends the first control information to the robot 10 (S205). After step S205, after the robot 10 is controlled based on the received control information, it sends a signal indicating the end of the first control to the control device 2 (S206).

[0084] After step S206, the control device 2 sends a command to the camera 103 to acquire the second image information (S207). After step S207, the camera 103 acquires the second image information and sends the second image information to the control device 2, and stores the second image information in the control device 2 (S208).

[0085] After step S208, the control device 2 calculates the second control information based on the target operation information and the second image information (S209). The method for calculating the second control information is as follows: Figure 14 The calculation process for the robot's control actions will be described later.

[0086] After step S209, the control device 2 sends the second control information to the robot 10 (S210). After step S210, the robot 10, after being controlled based on the received second control information, sends a signal indicating the end of the second control to the control device 2 (S211).

[0087] After step S211, the control device 2 sends a signal to the operator S1 indicating the end of control and notifies that the control of the robot 10 has ended (S212).

[0088] Alternatively, each timing sequence can be executed sequentially without being commanded by operator S1.

[0089] [Explanation of the diagram]

[0090] Figure 5 This is a block diagram illustrating the hardware structure of the control system 1 of the robot 10 in Embodiment 1. The robot 10 and camera 103 are based on... Figure 1 The robot and camera are described in the same way, so the description is omitted.

[0091] The memory 21 stores information used for processing by the processor 22. For example, it may contain RAM (Random Access Memory) and ROM (Read Only Memory). RAM is a working memory used to temporarily store data generated or acquired by the processor 22. ROM is a memory used to store programs that define the processing performed by the processor 22. For example, the memory 21 may also store captured images, environmental image information, target image information, first image information, and second image information acquired from the image information acquisition unit 221, environmental information acquired from the environmental information calculation unit 224, and target job information acquired from the target job information calculation unit 225. Furthermore, the memory 21 may also store data processed by each acquisition unit and calculation unit. Additionally, the memory 21 may also store parameters of the camera 103 and the robot 10.

[0092] The processor 22 executes processing by referring to a program stored in the memory 21. For example, the processor 22 may be constructed using a CPU (Central Processing Unit) or a FPGA (Field Programmable Gate Array). For example, the processor 22 functionally implements an image information acquisition unit 221, a control unit 222, a status acquisition unit 223, an environmental information calculation unit 224, a target task information calculation unit 225, a first control information calculation unit 226, and a second control information calculation unit 227.

[0093] The image information acquisition unit 221 acquires images captured by the camera 103. The image information acquisition unit 221 is capable of performing a first image processing step to extract features of the environment and a second image processing step to extract features of the threaded hole 11. Here, the first image processing is image processing for control implemented in the first step, and the second image processing is image processing for control implemented in the second step.

[0094] When calculating environmental information, the image information acquisition unit 221 calculates environmental image information by performing first image processing on the captured image group and stores the environmental image information in the memory 21. When calculating target operation information, the image information acquisition unit 221 calculates first target image information and second target image information by performing first image processing and second image processing on the captured images. When calculating first control information, the image information acquisition unit 221 calculates first image information by performing first image processing on the captured images. When calculating second control information, the image information acquisition unit 221 calculates second image information by performing second image processing on the captured images. Additionally, the image information acquisition unit 221 may also store the data calculated by each processing in the memory 21. Furthermore, the image information acquisition unit 221 may also acquire parameters of the camera 103 from the camera 103 and store the parameters of the camera 103 in the memory 21.

[0095] The control unit 222 acquires first control information or second control information from the first control information calculation unit 226 or the second control information calculation unit 227. The control unit 222 transforms the first control information or second control information into a control signal that the robot 10 can receive. The control unit 222 sends the control signal to the robot 10. Upon receiving the control signal, the robot 10 acts according to the received control signal. For example, when the control unit 222 acquires the numerical arrangement of joint angle positions as the first control information or second control information, the control unit 222 transforms the control information into a control signal represented by a string and sends it to the robot 10. However, it is not limited to this; the control information and control signal can also use signals that are generally acceptable to the user.

[0096] The state acquisition unit 223 acquires the state of the robot 10 from the robot 10. The state acquisition unit 223 stores the acquired state of the robot 10 in the memory 21. In addition, the state acquisition unit 223 can also acquire the parameters of the robot 10 from the robot 10 and store the parameters of the robot 10 in the memory 21.

[0097] The environmental information calculation unit 224 acquires environmental information based on the environmental image information acquired from the image information acquisition unit 221. The environmental information calculation unit 224 stores the acquired environmental information in the memory 21.

[0098] The target job information calculation unit 225 acquires first target job information based on the first target image information and environmental information acquired from the image information acquisition unit 221. The target job information calculation unit 225 acquires second target job information based on the second target image information acquired from the image information acquisition unit 221. The first and second target job information are combined as target job information. The target job information calculation unit 225 stores the acquired target job information in the memory 21. Here, the first target job information is job information for control implemented in the first step, and the second target job information is job information for control implemented in the second step.

[0099] The first control information calculation unit 226 calculates first control information for controlling the robot 10 based on the first image information, environmental information, and first target task information acquired from the image information acquisition unit 221. The first control information calculation unit 226 stores the calculated first control information in the memory 21. Alternatively, the first control information calculation unit 226 may also calculate the first control information based on the state of the robot 10 acquired by the state acquisition unit 223.

[0100] The second control information calculation unit 227 calculates second control information for controlling the robot 10 based on the second image information and the second target operation information acquired from the image information acquisition unit 221. The second control information calculation unit 227 stores the calculated second control information in the memory 21. Alternatively, the second control information calculation unit 227 may also calculate the second control information based on the state of the robot 10 acquired by the state acquisition unit 223.

[0101] The control device 2 may further include components other than the memory 21 and the processor 22. For example, it may further include an input device for instructing processing and a function block for commanding the processor to execute processing based on the input. For example, it may further include an output device for outputting the processed data and a function block for referencing the output information. For example, the control device 2 may further include a function block for acquiring control signals from the programmer, calculating control information, and sending the control information to the control unit 222. For example, the control device 2 may further include a communication device for connecting to the Internet and a function block for processing based on the communication signals.

[0102] [Flowchart Explanation]

[0103] In this embodiment, a flowchart is used to illustrate the actions that enable the robot to perform its tasks.

[0104] [Flowchart of computing environment information]

[0105] Figure 6This is a flowchart illustrating the method for calculating environmental information in Implementation 1.

[0106] First, the control unit 222 causes the robot 10 to move (S301).

[0107] After step S301, the environmental information calculation unit 224 acquires environmental image information from the image information acquisition unit 221 (S302). The environmental image information is obtained from the captured image taken by the camera 103 through the first image processing.

[0108] Furthermore, known methods for implementing first-order image processing include edge detection, corner detection, Hough transform, SHOT (Signature of Histograms of Orientations), FAST (Features from Accelerated Segment Test), and SURF (Speeded-Up Robust Features). To acquire a wide range of information about the environment, first-order image processing is desired to be a general-purpose image processing method.

[0109] After step S302, it is further determined whether to end the acquisition of environmental image information (S303). For example, the process for acquiring environmental image information is described in advance and the process is performed accordingly. If all the processes are completed (S303: Yes), the acquisition of environmental image information is ended. Otherwise (S303: No), environmental image information is acquired.

[0110] If further environmental image information is acquired before the process for acquiring environmental image information has ended (S303: No), return to the process in step S301. By repeating the processes in steps S301 and S302, multiple pieces of environmental image information are acquired for calculating environmental information in step S304, which will be described later.

[0111] Figure 7 This is a schematic diagram illustrating the environmental image information in Embodiment 1. The circle with a cross inside represents features in two-dimensional space of the environmental image information acquired through the first image processing. In this embodiment, the features are four points on the circumference of the threaded hole 11, the vertex of the substrate 12, the vertex of the wall 13, and the endpoints of the pattern, represented by a set of coordinate points in the two-dimensional image. The environmental image information is represented by drawing feature points on the captured images. Feature points P1 to P9 are points of particular interest among the acquired feature points for explanation. Although the environmental image information E1 to E3 based on three captured images is used as an example, more captured images are actually acquired as needed.

[0112] exist Figure 6 After step S302, if the acquisition of environmental image information ends (S303: Yes), the environmental information calculation unit 224 calculates environmental information based on the environmental image information (S304). Alternatively, a video of the environment can be acquired beforehand, converted into multiple images, and used as environmental images. Alternatively, environmental images can be added midway through the process, with the environmental information being updated online while the camera 103 adds environmental images.

[0113] In addition, a well-known method for computing environmental information is SLAM (Simultaneous Localization and Mapping). SLAM is a technique that simultaneously performs sensor-based self-position estimation and map generation based on information acquired from the sensors. Among the feature-based methods of SLAM that utilize image feature points, there are PTAM and ORB-SLAM. Among the dense methods of SLAM that utilize the overall brightness of the image, there are DTAM, LSD-SLAM, DSO, and SVO.

[0114] Figure 8 This is a schematic diagram illustrating the environmental information of the robot in Embodiment 1. The circle with a cross inside represents a 3D point set of environmental information. Environmental information is obtained through... Figure 1 Feature points are drawn on the environment shown in the image to represent it. Figure 8 The feature point P1 of the 3D point group shown in the environmental information is based on Figure 7 The feature points P1 of environmental image information E1 and environmental image information E2 are calculated. Similarly, Figure 8 The environmental information shows the feature points P3 of the 3D point group based on... Figure 7 Feature points P3 of environmental image information E2 and environmental image information E3 are calculated. Thus, based on the feature points of multiple environmental image information, environmental information is transformed into environmental information represented by feature points of a 3D point group.

[0115] [Flowchart for calculating target job information]

[0116] Figure 9 This is a flowchart illustrating the method for calculating target task information in Embodiment 1. The calculation of target task information is performed after the environmental information is calculated, using the calculated environmental information and the target captured image. The target task information includes first target task information corresponding to the first target image information and second target task information corresponding to the second target image information.

[0117] First, the control unit 222 sends control information to the robot 10 to acquire target image information, causing the robot 10 to move (S401). Through step S401, the robot 10 assumes a posture capable of performing the task, which is the target state of the robot 10.

[0118] Figure 10 This is a schematic diagram illustrating the target state of the robot in Embodiment 1.

[0119] exist Figure 9 After step S401, the image information acquisition unit 221 acquires the captured image by the camera 103 (S402). This is referred to as the target captured image. The target captured image may also be an image other than the image used to calculate the target task information.

[0120] Figure 11 This is a schematic diagram showing a target image captured in Embodiment 1. The threaded hole 11 is captured in the target image. Furthermore, the substrate 12 and the wall 13 can also be captured in the target image.

[0121] exist Figure 9 After step S402, the image information acquisition unit 221 calculates the first target image information based on the environmental information and the target captured image, and the target operation information calculation unit 225 calculates the first target operation information based on the environmental information and the first target image information (S403). Figure 12 This is a schematic diagram illustrating the first target image information in Embodiment 1. Figure 12 The circle with a cross inside represents the features in 2D space obtained by processing the target image through the first image processing step. Features are plotted using... Figure 11 The feature points P11 to P19 are obtained from the target image through the first image processing step. The first image processing step refers to the image processing used to acquire environmental information.

[0122] Based on such Figure 8 The environmental information shown in the image and such Figure 12 The first target image information shown in the figure is for Figure 8 Feature points P1 to P9 and Figure 12 Feature points P11 to P19 are mapped to each other to calculate the first target operation information. By establishing this mapping, the captured data can be calculated. Figure 12 time Figure 8 The camera's position / pose is obtained through this processing, and the resulting camera position / pose is the first target task information. SLAM is a well-known method for calculating the first target task information.

[0123] exist Figure 9After step S403, the image information acquisition unit 221 calculates the second target image information based on the target captured image, and the target operation information calculation unit 225 calculates the second target operation information based on the second target image information (S404).

[0124] Figure 13 This is a schematic diagram showing the second target image information in Embodiment 1. Figure 13 The circle with a cross inside represents the features in 2D space after the target image has been processed through the second image processing step. Features are plotted in... Figure 11 The target image is described. The second image processing refers to image processing focusing on the object being processed (threaded hole 11).

[0125] The second image processing in this embodiment refers to focusing on the threaded hole 11 and extracting its feature points. Specifically, as features of the threaded hole 11 extracted by the second image processing, the center coordinates of the threaded hole 11, as well as the intersection points of the major axis and minor axis with the contour of the threaded hole 11 when the threaded hole 11 is identified as an ellipse, are extracted.

[0126] Furthermore, known methods for implementing the second image processing include Hough transform and marker detection. In order to obtain local information about the work object (threaded hole 11), it is desirable that the second image processing be a limited image processing that only applies to the work object (threaded hole 11).

[0127] The second target task information can be directly derived from the second target image information. Alternatively, the second target task information can be calculated separately based on the second target image information. For example, when the second target image information is set as a set of feature points in two-dimensional space, feature points can be selected from these feature points, and other feature points can be ignored.

[0128] [Robot Control Process]

[0129] Figure 14 This is a flowchart illustrating the control actions of the robot in Implementation Method 1. Robot control is performed after calculating environmental information and target task information.

[0130] First, the image information acquisition unit 221 acquires the captured image, performs first image processing on the acquired captured image, and acquires first image information (S501).

[0131] Figure 15 This is a schematic diagram showing an image captured by camera 103 in the initial state of robot 10 in Embodiment 1. Figure 1As shown, in the initial state of robot 10, robot arm 101, screwdriver 102, and camera 103 are facing wall 13, so the threaded hole 11 is not captured in the image. When the first image processing is performed, feature points can be obtained from the pattern of wall 13. However, when the second image processing is performed, since the threaded hole 11 is not present in the captured image, the feature points of the threaded hole 11 cannot be extracted. Therefore, control cannot be performed using conventional methods that utilize the feature points of the threaded hole 11. Therefore, in this embodiment, control is performed by also utilizing feature points of the environment.

[0132] exist Figure 14 After step S501, control implemented in step 1 is performed (S502-S505). First, the first control information calculation unit 226 calculates the first control information based on environmental information, the first target operation information, and the first image information (S502). Next, the control process of the first operation information will be described.

[0133] First, based on environmental information and first image information, first job information at the shooting time point is obtained from the environmental information. Here, in this embodiment, job information is the position / pose of camera 103. Furthermore, SLAM is a known method for calculating the first job information.

[0134] Next, first control information is calculated such that the difference between the first target operation information (in this embodiment, the position / pose of the camera that becomes the target) and the first operation information (in this embodiment, the current position / pose of the camera) is 0. Furthermore, visual servoing is a known method for calculating the first control information.

[0135] After step S502, the control unit 222 uses the calculated first control information to control the robot 10 (S503). After step S503, the image information acquisition unit 221 acquires the third image information in the same way as in step S501 (S504).

[0136] After step S504, the processor 22 determines whether the threaded hole 11 exists in the third image information obtained in step S503 (S505). For example, it performs second image processing on the third image information to extract feature points of the threaded hole 11, and determines "yes" if the feature points can be extracted, and "no" otherwise. Specifically, it performs second image processing on the third image information to extract feature points of the threaded hole 11, and determines "yes" if the feature points are within 100 pixels of the center of the captured image, and "no" otherwise. For example, it uses YOLO (YouOnly Look Once) on the captured image, and determines "yes" if the threaded hole 11 exists in the captured image, and "no" otherwise.

[0137] Furthermore, the processing in step S505 is not limited to the above-described processing. For example, it is also possible to determine whether the task objective has been achieved. For example, based on environmental information and third image information, third task information is obtained, and if the third task information converges to the first target task information, it is determined to be "yes", and otherwise it is determined to be "no". Specifically, when comparing the third task information (in this embodiment, the camera position / pose) and the first target task information (in this embodiment, the position / pose of the camera that becomes the target), if the error of the camera position is within 5 [mm] and the error of the camera pose is within 0.1 [rad], it is determined to be "yes", and otherwise it is determined to be "no".

[0138] In step S505, if the threaded hole 11 is not present in the third image information (S505: No), the third image information is replaced with the first image information, and the processing of step 1 (S502 to S505) is performed. Since the target job information includes information about the threaded hole 11, by repeatedly performing step 1 (S502 to S505), the threaded hole 11 is included in the third image information.

[0139] In step S505, if the threaded hole 11 is present in the third image information (S505: Yes), the image information acquisition unit 221 acquires image information and performs second image processing on the acquired image to calculate second image information (S506). Alternatively, the image used in step S504 to calculate the third image information can also be used to calculate the second image information. Furthermore, if second image processing is performed in step S505, the information at that time can also be used to calculate the second image information.

[0140] After step S506, control implemented in the second step is performed (S507-S511). First, the second control information calculation unit 227 calculates the second control information based on the second target job information and the second image information (S507). Next, the control process of the second job information will be described.

[0141] The calculation produces second control information such that the difference between the second target operation information (in this embodiment, the elliptical feature of the threaded hole 11 that becomes the target) and the current operation information (in this embodiment, the elliptical feature of the threaded hole 11) is 0. Furthermore, visual servoing is a known method for calculating the second control information.

[0142] After step S507, the control unit 222 uses the calculated second control information to control the robot 10 (S508). After step S508, the image information acquisition unit 221 acquires the fourth image information in the same way as in step S506 (S509).

[0143] After step S509, processor 22 determines whether the fourth image information obtained in step S509 has achieved the target job information (S510). Furthermore, the fourth image information achieving the target job information means, for example, that the error between the features in the second target job information and the features in the fourth image information is below a certain value.

[0144] If the fourth image information achieves the second target task information in step S509 (S510: Yes), control ends. For example, when comparing each element of the fourth image information and the second target task information, it can be determined as "Yes" if the error of each element is less than 1 pixel, and as "No" otherwise.

[0145] In step S509, if the second target operation information is not achieved in the fourth image information (S510: No), it is determined whether a threaded hole 11 exists within a specific range of the fourth image information (S511). The determination in step S511 can also be performed in the same way as in step S505.

[0146] In step S511, if it is determined that a threaded hole 11 exists within a specific range of the fourth image information (S511: Yes), the fourth image information is replaced with the second image information, and the processing of step 2 (S507 to S511) is performed. By repeating step 2 (S507 to S511), the features of the fourth image information converge to the target operation information.

[0147] If it is determined in step S511 that the threaded hole 11 does not exist within a specific range of the fourth image information (S511: No), step 2 cannot be performed again. Therefore, in order to capture the threaded hole 11 again, we return to step 1 (S502 to S505).

[0148] [Effects, etc.]

[0149] As described above, in this embodiment, environmental information and target image information are calculated in advance, first control information is calculated based on the environmental information, and control based on the first control information is executed. Then, second control information is calculated based on the environmental information, and control based on the second control information is executed.

[0150] Therefore, even if the threaded hole 11 is not visible in the image captured by the camera, the operation can be performed while ensuring accuracy. Thus, the robot can be controlled and operated without requiring strict design of the environment around it.

[0151] [Implementation Method 2]

[0152] The following uses Figures 16-18 To illustrate Implementation Method 2, the same operations as in Implementation Method 1 are used in Implementation Method 2, and therefore are omitted.

[0153] [Explanation of the timing diagram]

[0154] In this embodiment, there is no difference between the timing diagram for calculating environmental information and the timing diagram for calculating target task information; therefore, only the control timing of the robot is described.

[0155] [Timing of robot control]

[0156] Figure 16 This is a timing diagram illustrating the control actions for implementing the operation of robot 10 in Embodiment 2. Steps S201-S204, S207-S209, and S212 are referenced. Figure 4 .

[0157] After step S209, the control device 2 calculates the third control information based on the first control information and the second control information (S601).

[0158] After step S601, the control device 2 sends the third control information to the robot 10 (S602). After step S602, the robot 10 performs an action based on the received control information (S603).

[0159] [Explanation of the diagram]

[0160] Figure 17This is a block diagram showing the hardware structure of the control system 1A of the robot 10 in Embodiment 2. The control system 1A, control device 2A, and processor 22A differ from those in Embodiment 1 in that they include a third control information calculation unit 228. Structures identical to those in Embodiment 1 are labeled with the same reference numerals, and descriptions are omitted.

[0161] The third control information calculation unit 228 calculates the third control information based on the first control information calculated from the first control information calculation unit 226 and the second control information calculated from the second control information calculation unit 227, and sends the third control information to the control unit 222.

[0162] [Flowchart Explanation]

[0163] The control system 1A of Implementation 2 is no different from the control system 1 of Implementation 1 for the flowcharts of calculating environmental information and calculating target task information, so only the control actions of the robot will be described.

[0164] [Robot Control Process]

[0165] Figure 18 This is a flowchart illustrating the robot's control actions in Embodiment 2. Robot control is performed after calculating environmental information and target task information.

[0166] First, the processor 22A acquires the captured image from the image information acquisition unit 221, calculates the first image information through the first image processing, and calculates the second image information through the second image processing (S701). Here, unlike Embodiment 1, in Embodiment 2, it is not necessary to calculate both the first image information and the second image information based on the captured image.

[0167] After step S701, processor 22A determines whether the captured image has achieved the task objective (S702). Step S702 can be performed in the same way as step S510 in Embodiment 1.

[0168] If the task objective is achieved by capturing images in step S702 (S702: Yes), control ends.

[0169] If the image capture in step S702 fails to achieve the task objective (S702: No), the processor 22A simultaneously performs the processing of step 1 (S703) and step 2 (S704). Step 3 (S705 and S706) is not executed until the processing of step 1 (S703) and step 2 (S704) is completed.

[0170] After step S702, processor 22A calculates first control information based on environmental information, first target operation information, and first image information (S703). Step S703 can be calculated in the same way as step S502 of embodiment 1.

[0171] After step S702, processor 22A calculates second control information based on the second target job information and the second image information (S704). Step S704 can be calculated in the same way as step S507 of embodiment 1.

[0172] After steps S703 and S704, processor 22A calculates third control information based on the first control information and the second control information (S705). For example, the third control information can also be obtained by weighted sum of the first control information and the second control information. For example, if the second control information cannot be obtained, the third control information can be obtained by directly using the first control information. For example, regarding the third control information, the smaller the error between the second target job information and the second image information, the larger the weight of the second control information should be.

[0173] Thus, by using the first control information and the second control information, it is possible to perform control that combines the advantages of the first control information (i.e., the ability to control even when the object being worked on is not present in the image captured by the camera) and the advantages of the second control information (i.e., the ability to control with high precision). In Embodiment 1, the above is achieved through two-stage control. In contrast, in Embodiment 2, control without a layered structure is possible.

[0174] After step S705, the control unit 222 uses the calculated second control information to control the robot 10 (S706). After step S706, the process returns to step S701.

[0175] [Effects, etc.]

[0176] As described above, in this embodiment, environmental information and target image information are calculated in advance, and third control information is calculated based on the first control information calculated based on the environmental information and the second control information calculated based on the environmental information, and control based on the third control information is executed.

[0177] Therefore, even when the object to be worked on is not present in the image captured by the camera, the task can be performed with precision. Thus, the robot can be controlled and operated without requiring strict design of its surrounding environment.

[0178] [Other Implementation Methods]

[0179] As described above, Embodiment 1 and Embodiment 2 have been presented as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited thereto and can also be applied to embodiments with modifications, substitutions, additions, omissions, etc. Furthermore, new embodiments can be created by combining the constituent elements described in Embodiment 1 and Embodiment 2.

[0180] In Embodiments 1 and 2, the features of the captured image are a set of coordinate points in the 2D image. However, other examples can also be considered. For example, in the case of a line, the feature is a set of paired coordinates of the start and end points of the 2D image. For example, in the case of an ellipse, the feature is a set of paired coordinates of the center of the ellipse and the vectors of its major and minor axes. For example, in the case of a rectangle, the feature is a set of paired coordinates of a point in the 2D image and two vectors. For example, in the case of a cuboid, the feature is a set of paired coordinates of a point in the 2D image and three vectors. Furthermore, these examples can be combined. Thus, planar graphics and solid graphics can ultimately be represented by a set of parameters of points or lines. In addition, they can be represented by parameters such as the color and brightness of the image. Therefore, the features of the captured image can be represented by a set of parameters representing features in a 2D image.

[0181] In Embodiments 1 and 2, the environmental information is characterized by a set of coordinate points in 3D space. However, other examples can also be considered. For example, when using lines for the environmental information, it can be represented by a set of paired start and end points in 3D space. For example, when using ellipses for the environmental information, it can be represented by a set of paired coordinates of the center of the ellipse in 3D space and vectors of the major and minor axes. For example, when using rectangles for the environmental information, it can be represented by a set of paired coordinates of a point in 3D space and two vectors. For example, when using cuboids for the environmental information, it can be represented by a set of paired coordinates of a point in 3D space and three vectors. Furthermore, these examples can be combined. Thus, planar graphics and solid graphics can ultimately be represented by a set of parameters of points or lines. Furthermore, the color and brightness of an image can be represented by parameters projected into 3D space. Therefore, environmental information can be represented by a set of parameters representing features in 3D space.

[0182] In Embodiments 1 and 2, the first target task information and the first task information are the camera's position / pose. However, the first target task information and the first task information can be calculated based on environmental information and the first target image information or the first image information. For example, it can be the relative position of the task object when viewed from the camera's position. For example, it can include features outside the shooting range when viewed from the camera's position.

[0183] In Embodiments 1 and 2, the second target task information and the second task information are sets of coordinate points in the captured image. However, the second target task information and the second task information can be calculated based on the second target image information or the second image information. For example, it can be the position / pose of the task object in the image. For example, it can be a combination of the features and position / pose of the task object in the image.

[0184] In Embodiments 1 and 2, environmental information is calculated before control is initiated. However, sometimes the environment changes during control, resulting in discrepancies between the actual environment and the calculated environmental information. Therefore, the environmental information can be updated using captured images to adapt it to the actual environment.

[0185] In Embodiments 1 and 2, the target job information is calculated before control is initiated. However, it is worth considering whether other jobs may be performed during control or whether the target job information may be adapted to changes in the environment. Therefore, the target job information may also be dynamically changed to match the desired job or the actual environment.

[0186] In Embodiments 1 and 2, a camera was used as a sensor. However, any sensor other than a camera can be used as long as environmental information can be acquired. For example, consider using a depth sensor as a sensor. When using a depth sensor, the portion with large depth changes is extracted as the first image processing. This allows the calculation of environmental features corresponding to a pattern of the environment as in Embodiment 1. The portion of the depth with an elliptical shape is extracted as the second image processing. This allows the extraction of features of the work object corresponding to the hole 11 of the threaded hole as in Embodiment 1. The environmental information is represented as a 3D point set using the features of the depth information extracted in the first image processing. The target work information uses the depth information acquired in the robot's target state, utilizing the information extracted in the first and second image processing. By processing the depth information and using it as features, control can be performed in the same way as described in Embodiments 1 and 2.

[0187] In Implementation 1, a robot control method is described in a control device connected to a robot performing work on a target object and sensors mounted on the robot. Control is executed based on a first step and a second step. In the first step, environmental information representing the robot's surrounding environment is calculated based on multiple environmental image information acquired from multiple positions and angles regarding the robot's surroundings. First image information is acquired using a camera. Based on target work information representing the robot's work target, environmental information, and the first image information, first control information is calculated to move the robot closer to the target object, and this first control information is sent to the robot. Following the first step, in the second step, second image information is acquired using a camera. Based on the target work information and the second image information, second control information is calculated to move the robot closer to the target object, and this second control information is sent to the robot. Through the first step, control can be performed using environmental information even when the target object is not captured in the captured image. Furthermore, through the second step, high-precision control based on the captured image is possible.

[0188] After performing step 1, image information is acquired using a camera. If, based on the image information, it is determined that the object to be worked on does not exist in the image information, step 1 is executed again. In step 1, control is executed to bring the first job information closer to the first target job information. Here, the first target job information includes information about the object to be worked on. Therefore, control can be executed from the point where information about the object to be worked on is not found in the image information until information about the object to be worked on is found, thus enabling step 2 to be executed.

[0189] After performing step 1, image information is acquired via a camera. If, based on environmental and image information, it is determined that the robot does not meet the task objective, step 1 is executed again. Thus, if the task objective cannot be achieved with one control attempt, control is performed again to bring the robot closer to the objective, thereby enabling control to achieve the task objective.

[0190] After performing step 2, image information is acquired via a camera. If, based on the image information, it is determined that the robot does not meet the task objective, step 2 is executed again. Thus, if the task objective cannot be achieved with one control attempt, control is performed again to bring the robot closer to the objective, thereby enabling control that achieves the task objective.

[0191] After performing step 2, image information is acquired via the camera. If the target object is not found within a certain range of the current image information, step 1 is executed. Therefore, even if the target object is missed in step 2, control can be adjusted to return to step 1 to recapture the target object.

[0192] The target operation information includes the feature information of the operation object at the end of the operation, the environmental information includes the 3D position information corresponding to the feature information of the operation object and the workbench in the environment, and the first image information includes the feature information of at least one of the operation object and the workbench at the acquisition time point.

[0193] 3D positional information has the unique characteristics of an object composed of parameters representing points, lines, planar graphics, or solid graphics.

[0194] The target task information includes the feature information of the task object at the time of task execution, and the second image information includes the feature information of the task object at the time of acquisition.

[0195] The target task information includes the characteristic information of the task object when the task is performed.

[0196] Feature information is the unique representation of an object composed of parameters representing points, lines, planar figures, or three-dimensional figures.

[0197] The first and second image information are used to update the environmental information. Therefore, even when the environment changes, the system can adapt to environmental changes by updating the environmental information.

[0198] Environmental image information is captured images taken by a camera.

[0199] In Implementation 2, a robot control method is described in a control device connected to a robot performing work on a target object and sensors mounted on the robot. Control is performed based on steps 1, 2, and 3. In step 1, environmental information representing the robot's surrounding environment is calculated based on multiple environmental image information obtained by taking pictures from multiple positions and angles regarding the robot's surrounding environment. First image information and second image information are acquired using a camera. First control information, which indicates the target object being worked on by the robot, is calculated based on target work information, environmental information, and first image information, such as moving the robot closer to the target object. In step 2, second control information, which indicates moving closer to the target object, is calculated based on the target work information and second image information. In step 3, third control information is calculated based on the first and second control information, and the third control information is sent to the robot. Therefore, even if the target object is not captured in the first step, control can be performed as follows: calculate control information that enables control, calculate high-precision control information for the target object in the second step, and combine the first and second steps in the third step to compensate for each other's shortcomings.

[0200] After performing step 3, if it is determined that the robot does not meet the task objective, based on the target task information, environmental information, and the third image information acquired using the camera, fourth control information is calculated to make the robot approach the task objective; based on the target task information and the fourth image information acquired using the camera, fifth control information is calculated to approach the task objective; based on the fourth and fifth control information, sixth control information is calculated and sent to the robot. Therefore, if the task objective is not achieved with one control attempt, it can be brought closer to the task objective by performing control again, thus achieving the desired task objective.

[0201] Furthermore, the above-described embodiments are intended to illustrate the technology in this disclosure, and various changes, substitutions, additions, omissions, etc., can be made within the scope of the claims or their equivalents.

[0202] Industrial availability

[0203] This disclosure can be applied to control devices for robots to perform operations in various environments. Specifically, this disclosure can be applied to assembly operations in factories, sorting operations in warehouses, and back-end delivery / product shelving operations.

[0204] Symbol Explanation

[0205] 1. 1A Control System

[0206] 10 robots

[0207] 101 robotic arm

[0208] 102 screwdriver

[0209] 103 camera

[0210] 1031 Shooting Range

[0211] 11 Threaded hole

[0212] 12 substrate

[0213] 13 walls

[0214] 2.2A control device

[0215] 21 memory

[0216] 22, 22A processor

[0217] 221 Image Information Acquisition Department

[0218] 222 Control Department

[0219] 223 Status Acquisition Department

[0220] 224 Environmental Information Computing Department

[0221] 225 Target Operation Information Calculation Department

[0222] 226 First Control Information Calculation Department

[0223] 227 Second Control Information Calculation Department

[0224] 228 Third Control Information Calculation Department

[0225] E1, E2, E3 environmental image information

[0226] Feature points P1~P9 and P11~P19.

Claims

1. A robot control method, comprising a robot performing work on a work object and a control device connected to sensors, i.e., cameras, mounted on the robot. In the robot control method, Based on multiple images captured by the camera from multiple positions and angles surrounding the robot, environmental information representing the surrounding environment of the robot is calculated. The environmental information includes 3D position information corresponding to the feature information of the work object and the feature information of surrounding objects in the surrounding environment. The camera acquires first sensor information, which includes feature information of the work object and feature information of the surrounding objects in the first image captured at the first acquisition time point. In step 1, based on the first target task information representing the robot's task target for the task object, the environmental information, and the first sensor information, the first control information for making the robot approach the task target is calculated, and the first control information is sent to the robot for control. The first target task information represents the target position and posture of the camera. After step 1, the camera acquires second sensor information, which includes feature information of the work object in the second image captured at the second acquisition time point; In the second step, based on the second target task information and the second sensor information, second control information is calculated to make the robot approach the task target, and the second control information is sent to the robot for control. The second target task information represents the position and posture of the task object in the second image.

2. The robot control method according to claim 1, wherein, After performing the first step, the camera acquires information from the third sensor. If the third sensor information indicates that the object to be worked does not exist, the first step is then performed.

3. The robot control method according to claim 1 or 2, wherein, After performing the first step, the camera acquires information from the third sensor. If, based on the environmental information and the third sensor information, it is determined that the robot does not meet the operational objective, the first step is performed.

4. The robot control method according to claim 1 or 2, wherein, After performing the second step, the camera acquires information from the fourth sensor. If, based on the fourth sensor information, it is determined that the robot does not meet the task objective, the second step is performed.

5. The robot control method according to claim 1 or 2, wherein, After performing the second step, the camera acquires information from the fourth sensor. If the fourth sensor information indicates that the object to be worked does not exist, the first step is then performed.

6. The robot control method according to claim 1 or 2, wherein, The 3D positional information has the unique characteristics of an object composed of parameters representing points, lines, planar graphics, or three-dimensional graphics.

7. The robot control method according to claim 1 or 2, wherein, The feature information has the unique characteristics of an object composed of parameters representing points, lines, planar figures, or three-dimensional figures.

8. The robot control method according to claim 1 or 2, wherein, The environmental information is updated using the information from the first sensor and the information from the second sensor.

9. A robot control method, comprising a robot performing work on an object and a control device connected to sensors, i.e., cameras, mounted on the robot. In the robot control method, Based on multiple images captured by the camera from multiple positions and angles surrounding the robot, environmental information representing the surrounding environment of the robot is calculated. The environmental information includes 3D position information corresponding to the feature information of the work object and the feature information of surrounding objects in the surrounding environment. The camera is used to acquire first sensor information and second sensor information. The first sensor information includes feature information of the work object and feature information of the surrounding objects in the first image captured at the first acquisition time point. The second sensor information includes feature information of the work object in the second image captured at the second acquisition time point. In step 1, based on first target task information representing the robot's task target towards the task object, the environmental information, and the first sensor information, first control information is calculated to make the robot approach the task target, wherein the first target task information represents the target position and posture of the camera; In the second step, based on the second target task information and the second sensor information, second control information is calculated to make the robot approach the task target. The second target task information represents the position and posture of the task object in the second image. Step 3 is executed, based on the first control information and the second control information, the third control information is calculated, and the third control information is sent to the robot for control.

10. The robot control method according to claim 9, wherein, After performing step 3, if it is determined that the robot does not meet the task objective, Based on the first target task information, the environmental information, and the third sensor information acquired using the camera, fourth control information is calculated to make the robot approach the task target; Based on the second target task information and the fourth sensor information acquired using the camera, fifth control information is calculated to make the robot approach the task target; Based on the fourth and fifth control information, the sixth control information is calculated and sent to the robot for control.

Images