ROBOT CONTROL SYSTEM, ROBOT SYSTEM, ROBOT CONTROL METHOD, AND ROBOT CONTROL PROGRAM
The integrated approach of environmental recognition and action possibility in the robot control system allows robots to efficiently perform tasks in dynamic environments by adaptively extracting and approaching task areas, enhancing accuracy and efficiency.
Patent Information
- Application Number
- JP2023534460
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-07-12
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-07-12
AI Technical Summary
Existing robot systems struggle to efficiently perform tasks in unknown or dynamic environments due to separate considerations of environmental recognition and action possibility, leading to inefficiencies and inaccuracies.
A robot control system that integrates environmental recognition and action possibility by using active sensing to extract a task area, allowing the robot to adaptively approach and perform tasks by repeatedly extracting the task area as conditions change.
Enables robots to efficiently perform tasks with high accuracy by continuously adapting to environmental changes and optimizing movements, even with low sensor sensitivity or calibration errors.
Smart Images

Figure 0007590577000001 
Figure 0007590577000002 
Figure 0007590577000003
Abstract
Description
[Technical field]
[0001] One aspect of the present disclosure relates to a robot control system, a robot system, a robot control method, and a robot control program. [Background technology]
[0002] Patent Document 1 describes an active environment recognition system that uses various sensors to recognize the external environment around the sensor. This system includes one or more sensor mechanisms that collect information about the external environment, an actuator mechanism that changes the external environment, and a hierarchical information processing mechanism that generates appropriate motion commands for the actuator mechanism based on the sensor information from the sensor mechanism. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Japanese Patent Application Publication No. 8-30327 Summary of the Invention [Problem to be solved by the invention]
[0004] It is desirable to have robots perform tasks efficiently. [Means for solving the problem]
[0005] A robot control system according to one aspect of the present disclosure includes an extraction unit that extracts a task area in which the robot can execute a task for processing an object from an observation area surrounding the robot, and a robot control unit that controls the robot so that the robot approaches the object in response to the extraction of the task area, and the extraction unit again extracts the task area after the robot approaches the object.
[0006] A robot control method according to one aspect of the present disclosure is a robot control method executed by a robot control system having at least one processor, and includes the steps of extracting a task area in which the robot can perform a task for processing an object from an observation area around the robot, and controlling the robot to approach the object in response to the extraction of the task area, wherein in the extraction step, the task area is extracted again after the robot has approached the object.
[0007] A robot control program according to one aspect of the present disclosure causes a computer to execute the steps of extracting a task area in which the robot can execute a task for processing an object from an observation area around the robot, and controlling the robot in response to the extraction of the task area so that the robot approaches the object, and in the extraction step, the task area is extracted again after the robot approaches the object. Effect of the Invention
[0008] According to one aspect of the present disclosure, it is possible to cause a robot to efficiently execute a task. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of an application of a robot control system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer used for the robot control system. [Diagram 3] 11 is a flowchart illustrating an example of a process in a robot control system. [Figure 4] FIG. 13 is a diagram illustrating an example of extraction of a task area. [Diagram 5] FIG. 2 is a diagram illustrating an example of robot control by a robot control system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are designated by the same reference numerals, and duplicated explanations will be omitted.
[0011] [System configuration] In this embodiment, an example of a robot control system according to the present disclosure is illustrated as one component of a robot system 1. The robot system 1 is a mechanism for automating a given task by causing a robot to perform an action for achieving a given purpose.
[0012] FIG. 1 is a diagram showing an example of the configuration of a robot system 1 and also showing an example of application of a robot control system. In this example, the robot system 1 includes a robot control system 10, one or more robots 2, and one or more robot controllers 3 corresponding to the one or more robots 2. FIG. 1 shows one robot 2 and one robot controller 3, and shows a configuration in which one robot 2 is connected to one robot controller 3. However, the number of devices and the connection method are not limited to the example of FIG. 1. For example, one robot controller 3 may be connected to multiple robots 2. The communication network connecting the devices may be a wired network or a wireless network. The communication network may be configured to include at least one of the Internet and an intranet. Alternatively, the communication network may be simply realized by one communication cable.
[0013] The robot control system 10 is a computer system for operating the robot 2 autonomously in at least some circumstances. The robot control system 10 executes a given calculation to generate a command signal for controlling the robot 2. In one example, the command signal includes data for controlling the robot 2, for example, a path indicating the trajectory of the robot 2. The trajectory of the robot 2 refers to the path of movement of the robot 2 or its components. For example, the trajectory of the robot 2 may be the trajectory of the tip. The robot control system 10 transmits the generated command signal to the robot controller 3.
[0014] The robot controller 3 is a device that operates the robot 2 according to a command signal from the robot control system 10. In one example, the robot controller 3 calculates joint angle target values (angle target values of each joint of the robot 2) for matching the position and posture of the tip with the target values indicated by the command signal, and controls the robot 2 according to the angle target values.
[0015] The robot 2 is a device or machine that performs work in place of a human. In one example, the robot 2 is a multi-axis serial-link vertical articulated robot. The robot 2 includes a manipulator 2a and an end effector 2b, which is a tool attached to the tip of the manipulator 2a. The robot 2 can execute various processes using the end effector 2b. The robot 2 can freely change the position and orientation of the end effector 2b within a given range. The robot 2 may be a six-axis vertical articulated robot, or a seven-axis vertical articulated robot with one additional redundant axis added to the six axes.
[0016] The robot 2 operates based on the control of the robot control system 10 to execute a given task. In this disclosure, a task refers to a series of processes that the robot 2 is made to execute in order to achieve a certain purpose. The robot 2 executes a task to obtain a result that the user of the robot system 1 desires. For example, a task is set to process some kind of object. Examples of tasks include "grasping an object and placing it on a conveyor," "grasping a part and attaching it to a workpiece," and "spray painting an object." In this disclosure, an object refers to a tangible object that is processed by the robot 2.
[0017] In a situation where the surrounding environment of the robot 2 is unknown, the robot control system 10 actively extracts an area where the robot 2 is expected to be able to execute a task, i.e., an area where an object can be processed, as a task area, and causes the robot 2 to approach the object (task area). The robot control system 10 repeatedly extracts the task area while causing the robot 2 to approach the object. When the robot 2 reaches the task area, the robot control system 10 causes the robot 2 to execute the task. In the present disclosure, the robot 2 reaching the task area means that the robot 2 has approached the object sufficiently to be able to execute the task.
[0018] The robot control system 10 extracts a task area based not only on the recognition of the surrounding environment of the robot 2 but also on the potential for action, which indicates whether the robot can act in that environment. That is, the robot control system 10 extracts a task area by considering both environmental recognition and potential for action, rather than considering these two elements separately. By extracting the task area in this way, it becomes possible to control the robot 2 more efficiently than when considering these two elements separately.
[0019] To recognize unknown surroundings, the robot control system 10 uses active sensing, a technique for actively changing sensor conditions to search for and gather necessary information. This technique allows the robot 2 to recognize a target to aim at even when object or environmental conditions change frequently or are difficult or impossible to model in advance. Active sensing is a technique for discovering unknown targets and is therefore different from visual feedback, which positions a mechanical system toward a known target.
[0020] Considering the possibility of action can be said to be considering the meaning that the recognized environment gives to the behavior of the robot 2. Therefore, the robot control by the robot control system 10 can be said to be a mechanism that utilizes the concept of affordance. The robot control system 10 can realize advanced decision-making regarding robot control by linking perception (sensors) and the robot's behavior and symbolizing them. Considering the possibility of action makes it possible to flexibly process similar tasks without using a strict model. In addition, since it is sufficient to process information directly related to the robot's behavior, calculations for robot control can be executed efficiently.
[0021] An example of a sensor for recognizing the three-dimensional space around the robot 2 is a visual sensor such as a camera. In one example, the robot 2 includes a camera 20 that captures the periphery of the end effector 2b. The coverage of the camera 20 may be set so as to capture at least a part of the end effector 2b. The camera 20 may be disposed on the manipulator 2a, for example, attached near the tip of the manipulator 2a. The camera 20 moves in response to the operation of the robot 2. This movement may include a change in at least one of the position and the attitude of the camera 20. As long as the camera 20 moves in response to the operation of the robot 2, it may be provided in a location different from the robot 2. For example, the camera 20 may be attached to another robot, or may be movably provided on a ceiling, a wall, or a camera stand.
[0022] The robot control system 10 extracts a task area from an observation area around the robot 2 based on information obtained from at least one given sensor. When the camera 20 is used, the "information obtained from the sensor" is an image (still image or video) obtained from the camera 20. If the task area can be extracted, the robot control system 10 controls the robot 2 so that the robot 2 approaches the object. The robot control system 10 extracts the task area again after the robot 2 approaches the object, and if the task area can be extracted, causes the robot 2 to approach the object further. In this manner, the robot control system 10 repeats the extraction of the task area and the control of the robot 2 to approach the object. When the robot 2 reaches the task area, the robot control system 10 causes the robot 2 to execute the task. By repeating the extraction of the task area and the approach to the object, the object can be processed while adapting to changes in the surrounding situation. Furthermore, the repeated processing allows the task to be executed with high accuracy even when the sensitivity of the sensor is not high or there is a calibration error in the sensor.
[0023] When the robot control system 10 causes the robot 2 to execute a plurality of tasks that are different from one another (that is, a plurality of types of tasks), the robot control system 10 extracts a task area corresponding to each type of task.
[0024] In one example, the task includes a step in which the robot 2 contacts an object. Examples of such a task include a task including grasping an object, and a task including pushing or pulling an object. When causing the robot 2 to execute such a task, the robot control system 10 extracts, as a task area, an area in which the robot 2 can contact the object to execute the task.
[0025] In the present disclosure, the observation area refers to an area that is set as a range for extracting a task area. The observation area can also be said to be an area captured by a sensor. For example, the observation area is an area photographed by the camera 20.
[0026] When a task area cannot be extracted in a certain observation area, the robot control system 10 changes the position of the observation area and attempts to extract a task area in the changed observation area. The situation where "a task area cannot be extracted in the observation area" includes a case where there are no unprocessed task areas in the observation area. In one example, the robot control system 10 repeats extraction of a task area and control of the robot 2 to approach an object in each observation area while changing the position of the observation area in the three-dimensional space around the robot 2. In this way, the robot control system 10 controls the robot 2 while selectively using exploration and exploitation. The robot 2 autonomously processes one or more objects located in the vicinity of the robot 2 according to the control.
[0027] FIG. 1 also shows an example of a functional configuration of the robot control system 10. In one example, the robot control system 10 includes a search unit 11, an observation history database 12, an image acquisition unit 13, an extraction unit 14, and a robot control unit 15 as functional components. The search unit 11 is a functional module that sets the position of an observation area. The search unit 11 records observation data indicating an observation area in which a task area has been searched. The observation history database 12 is a functional module that stores the observation data. The image acquisition unit 13 is a functional module that acquires an image of the observation area from a camera 20. The extraction unit 14 is a functional module that extracts a task area from the observation area based on the image. The robot control unit 15 is a functional module that controls the robot 2 based on the task area. In one example, the robot control unit 15 causes the robot 2 to approach an object in response to the extraction of the task area, and causes the robot 2 to execute a task in response to the robot 2 reaching the task area.
[0028] The robot control system 10 can be realized by any type of computer. The computer may be a general-purpose computer such as a personal computer or a business server, or may be incorporated in a dedicated device that executes a specific process.
[0029] 2 is a diagram showing an example of a hardware configuration of a computer 100 used for the robot control system 10. In this example, the computer 100 includes a main body 110, a monitor 120, and an input device .
[0030] The main body 110 is a device having a circuit 160. The circuit 160 has at least one processor 161, a memory 162, a storage 163, an input / output port 164, and a communication port 165. The storage 163 records a program for configuring each functional module of the main body 110. The storage 163 is a computer-readable recording medium such as a hard disk, a non-volatile semiconductor memory, a magnetic disk, or an optical disk. The memory 162 temporarily stores a program loaded from the storage 163, a calculation result of the processor 161, and the like. The processor 161 configures each functional module by executing a program in cooperation with the memory 162. The input / output port 164 inputs / outputs an electric signal between the monitor 120 or the input device 130 in response to a command from the processor 161. The input / output port 164 may input / output an electric signal between another device such as the robot controller 3. The communication port 165 performs data communication with another device via the communication network N in response to a command from the processor 161.
[0031] Monitor 120 is a device for displaying information output from main body 110. Monitor 120 may be any device capable of displaying graphics, and a specific example of such a device is a liquid crystal panel.
[0032] The input device 130 is a device for inputting information to the main body 110. The input device 130 may be any device capable of inputting desired information, and specific examples thereof include an operation interface such as a keypad, a mouse, or an operation controller.
[0033] The monitor 120 and the input device 130 may be integrated as a touch panel. For example, the main body 110, the monitor 120, and the input device 130 may be integrated as a tablet computer.
[0034] [Robot control method] As an example of a robot control method according to the present disclosure, an example of a series of processing steps executed by the robot control system 10 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of processing in the robot control system 10 as processing flow S1. That is, the robot control system 10 executes the processing flow S1.
[0035] In step S11, the extraction unit 14 extracts a task area from the observation area. The extraction unit 14 extracts the task area from the observation area set to an initial position based on a given preset setting, or from the observation area whose position has been changed by the search unit 11. In one example, the image acquisition unit 13 acquires an image of the observation area from the camera 20, and the extraction unit 14 extracts the task area based on the image. The extraction unit 14 can extract multiple task areas in one observation area.
[0036] The extraction unit 14 may extract a task area based on the configuration of the robot 2 and the current situation in the observation area. The current situation in the observation area refers to the current situation around the robot 2. The extraction unit 14 may determine, as the current situation in the observation area, whether or not an object exists in the observation area, may determine the structure of the object, or may determine whether or not an obstacle different from the object exists in the observation area.
[0037] The extraction unit 14 extracts a task area corresponding to a task to be executed. When it is possible for the robot 2 to execute a plurality of types of tasks, the extraction unit 14 extracts a task area corresponding to a task to be executed.
[0038] In one example, the extraction unit 14 analyzes each pixel of the image using techniques such as machine learning using deep learning (e.g., convolutional neural network (CNN)), color filtering, template matching, etc., and calculates the probability that an object exists for each pixel. The extraction unit 14 then generates a grayscale image indicating the probability at each pixel. This grayscale image can be said to be information that probabilistically indicates the position of the object within the observation area. The extraction unit 14 may extract an area in which the object appears as a task area based on the grayscale image.
[0039] Alternatively, the extraction unit 14 may obtain a depth image in addition to the grayscale image, and extract the task region based on these two types of images. The extraction unit 14 may use the depth image generated by the camera 20, or may generate the depth image based on the image from the camera 20 and the depth information. The extraction unit 14 identifies the depth at a position where the object is expected to exist based on the grayscale image and the depth image. The extraction unit 14 calculates whether the robot 2 can process the object at that position based on the depth and the configuration of the end effector 2b. In this manner, the extraction unit 14 may extract the task region by comparing the configuration of the end effector 2b with the structure of the object. In one example, the extraction unit 14 uses a Gaussian function in the calculation to probabilistically calculate whether the robot 2 can process the object. When the task includes grasping the object, the extraction unit 14 may perform the calculation using Fast Graspability Evaluation (FGE), which is a method for identifying a position where the object can be grasped. The extraction unit 14 calculates the distribution of the probability that a task region exists in the observation region by such a probabilistic calculation. In this disclosure, the distribution is also called a region map. In one example, the extraction unit 14 extracts a region where the probability reaches a given threshold as a task region.
[0040] As in the above example, the extraction unit 14 may extract the task area based at least on the presence of an object in the observation area. The extraction unit 14 may extract the task area based further on the absence of interference with an obstacle when the robot 2 executes a task. The use of FGE is an example of extracting the task area taking into account the interference.
[0041] 4 is a diagram showing an example of extraction of a task region. In this example, the robot 2 executes a task of grasping an orange and moving the orange to a given position. To realize this autonomous processing, the extraction unit 14 analyzes the image 200 obtained from the camera 20 and extracts a task region in which the robot 2 is expected to be able to grasp the orange. In other words, the extraction unit 14 extracts, as the task region, an area in which the robot 2 can come into contact with an object (an orange) to execute the task.
[0042] Image 200 shows an orange 221 resting on a floor 211 and an orange 222 held above a fence 212. Several other objects 231 that may be obstacles in the performance of the task are placed around the orange 221. Image 200 also shows the tip of a gripper, which is an example of an end effector 2b.
[0043] The extraction unit 14 analyzes the image 200 to extract a task region 311 corresponding to the orange 221 and a task region 312 corresponding to the orange 222. The region map 300 shows the distribution of the probability of the existence of a task region. In the region map 300, a region with a high probability is shown relatively white, and a region with a low probability is shown relatively black. The task region 311 is shown in dark gray, whereas the task region 312 is shown in white. This means that the possibility of grasping the orange 222 is higher than the possibility of grasping the orange 221. Such a difference is expected to be due to at least one of the presence of another object 231 near the orange 221 and the orange 221 being farther away than the orange 222.
[0044] Returning to FIG. 3, if at least one task area is extracted (YES in step S12), the process proceeds to step S13. In step S13, the robot control unit 15 determines an approach target position, which is a position to which the robot 2 should further approach. This process can also be said to be determining where in the observation area to search in more detail. In one example, the robot control unit 15 may calculate a score representing the possibility that the robot 2 can execute the task for the observation area, and determine the position with the maximum score as the approach target position. The robot control unit 15 may use the above-mentioned area map as the distribution of the scores. Alternatively, the robot control unit 15 may calculate the position of the center of the detected object as the approach target position.
[0045] When multiple task areas are extracted, the robot control unit 15 selects one of the multiple task areas and determines the approach target position for the selected task area. For example, the robot control unit 15 may select the task area corresponding to the maximum score.
[0046] In step S14, the robot control unit 15 determines whether or not the robot 2 has reached the task area.
[0047] In one example, the robot control unit 15 may calculate the distance between the end effector 2b and the approach target position and perform the determination based on this distance. If the calculated distance is equal to or less than a given threshold, the robot control unit 15 determines that the robot 2 has reached the task area. On the other hand, if the calculated distance is greater than the threshold, the robot control unit 15 determines that the robot 2 has not reached the task area.
[0048] In another example, the robot control unit 15 may calculate a target configuration indicating a state of the robot 2 required to execute a task in the task area and its uncertainty. The target configuration includes, for example, the position and posture of the robot 2 for starting the task. The target configuration will be described in detail later. If the calculated uncertainty is equal to or less than a given threshold, the robot control unit 15 determines that the robot 2 has reached the task area. On the other hand, if the calculated uncertainty is greater than the threshold, the robot control unit 15 determines that the robot 2 has not reached the task area.
[0049] If it is determined that the robot 2 has not reached the task area (NO in step S14), the process proceeds to step S15. In step S15, the robot control unit 15 controls the robot 2 toward the task area. That is, the robot control unit 15 controls the robot 2 so that the robot 2 approaches an object. If one of a plurality of task areas is selected, the robot control unit 15 controls the robot 2 so that the robot 2 approaches the selected task area (object).
[0050] In one example, the robot control unit 15 generates, by planning, a path for the robot 2 to move the end effector 2b from the current position to the approach target position. Alternatively, the robot control unit 15 may generate, by planning, a path (trajectory) for the robot 2 so that the distance is reduced and the approach target position is captured at the center of the image of the camera 20. The robot control unit 15 may select an appropriate position and posture from candidates of positions and postures of the robot 2 set in advance, and generate, by planning, a path for realizing the selected position and posture. As at least a part of the path planning, the robot control unit 15 may execute at least one of generating a path in which the robot 2 does not interfere with an obstacle and generating a path in which a singular posture (singular point) of the robot 2 does not occur. A singular posture (singular point) refers to a posture in which the robot 2 cannot be controlled from the viewpoint of the structure of the robot 2.
[0051] The robot control unit 15 outputs a command signal indicating the generated path to the robot controller 3, and the robot controller 3 controls the robot 2 according to the command signal. As a result, the robot 2 approaches the target object along the path.
[0052] After step S15, the process returns to step S11, and the robot control system 10 executes the processes from step S11 onward again. That is, the robot control system 10 extracts the task area again after the robot 2 approaches the target object, and executes the subsequent processes based on the result of this extraction.
[0053] If it is determined that the robot 2 has reached the task area (YES in step S14), the process proceeds to step S16. In step S16, the robot control unit 15 causes the robot 2 to execute the task.
[0054] In one example, the robot control unit 15 calculates a target configuration and generates a path by planning based on the target configuration. As described above, the target configuration may include the position and posture of the robot 2 for starting a task. For example, in the case of making the robot 2 grip an object, the robot control unit 15 calculates the position and posture of the robot 2, including the opening width of the gripper for gripping. In the case of a task including a plurality of main operations (for example, a task including an operation of gripping an object at a first position and an operation of placing the object at a second position), the robot control unit 15 may calculate a target configuration for each main operation. Next, the robot control unit 15 generates a path for executing the target configuration and the task by planning. In this planning, the robot control unit 15 may also execute at least one of generating a path that does not cause interference between the robot 2 and an obstacle and generating a path that does not cause a singular posture (singular point) of the robot 2.
[0055] The robot control unit 15 outputs a command signal indicating the generated path to the robot controller 3, and the robot controller 3 controls the robot 2 according to the command signal. The robot control unit 15 may output a command signal indicating a path for executing a target configuration to the robot controller 3, and then output a command signal indicating a path for executing a task to the robot controller 3. In either case, the robot 2 moves toward a position and posture based on the target configuration, and then executes the task.
[0056] In step S17, the robot control unit 15 determines whether or not to end the robot control. The robot control unit 15 may perform this determination based on any termination condition. For example, the robot control unit 15 may determine to end the robot control when a task has been executed a specified number of times, and may determine to continue the robot control when the task has been executed less than the specified number of times. Alternatively, the robot control unit 15 may determine to end the robot control when an error occurs in the robot control, and may determine to continue the robot control when the error does not occur.
[0057] If it is determined that the robot control is to be ended (YES in step S17), the process proceeds to step S18. In step S18, the robot control system 10 executes a termination process. In this termination process, the robot control unit 15 may return the robot 2 to its initial posture and position. Alternatively, the robot control unit 15 may notify the user by visual information or audio that all tasks have been completed.
[0058] If the robot control is to be continued (NO in step S17), the process proceeds to step S19. In step S19, the robot control unit 15 performs preparation for the next task. For example, the robot control unit 15 may return the robot 2 to its initial posture and position. Alternatively, the robot control unit 15 may notify the user by visual information or audio that the next task will be executed.
[0059] After step S19, the process returns to step S11, and the robot control system 10 executes the processes from step S11 onwards for the next task.
[0060] The above-mentioned series of processing steps mainly relate to the use of the search function. Next, the processing steps mainly related to the search function will be described.
[0061] If the task area has not been extracted (NO in step S12), the process proceeds to step S20. In step S20, the search unit 11 records observation data indicating at least one observation area observed so far. For example, the search unit 11 records observation data indicating a plurality of observation areas observed while the robot 2 is approaching an object. In one example, the search unit 11 generates a record of observation data for each observation area. The record may include, as data items, representative coordinates of the observation area, the observation time, and a score indicating the feasibility of the task at the representative coordinates. Each record may further include at least one of an area map, an extracted task area, the number of objects processed, and a task execution history. The search unit 11 stores at least one record in the observation history database 12.
[0062] In step S21, the search unit 11 determines whether or not to continue searching for the object. In one example, the search unit 11 determines whether or not there is a possibility of discovering a new object based on the observation data in the observation history database 12. If the search unit 11 determines that there is a possibility, it determines to continue the search, and if it determines that there is no possibility, it determines to end the search. For example, the search unit 11 processes the observation data by Gaussian process regression to calculate a predicted value and uncertainty about the possibility of an action in the three-dimensional space around the robot. If there is a position where the uncertainty is greater than a given threshold, the search unit 11 determines to continue the search, and if there is no such position, it determines to end the search. If the uncertainty is greater than the threshold, it means that there is no confidence in the judgment of whether or not a task area will be extracted, so it is worth trying to search such a place.
[0063] Alternatively, the search unit 11 may determine to continue the search if a position whose distance to the nearest representative coordinate indicated by the observation data is longer than a given threshold value exists in a subject plane corresponding to the three-dimensional space around the robot, and may determine to end the search if such a position does not exist.
[0064] Alternatively, the search unit 11 may determine to continue the search if an unobserved area exists within a pre-set observation range, and may determine to end the search if the unobserved area does not exist. The observation range is, for example, the whole or a part of the three-dimensional space around the robot. This method can be said to be a method for searching the whole observation range without omission.
[0065] If it is determined to continue the search (YES in step S21), the process proceeds to step S22. In step S22, the search unit 11 changes the position of the observation area in the three-dimensional space around the robot 2. Step S22 can be executed when a task area is not extracted from the observation area (NO in step S12). That is, the search unit 11 changes the position of the observation area in response to the extraction unit 14 not extracting a task area.
[0066] In one example, the search unit 11 dynamically changes the position of the observation area. This dynamic change means that the next position of the observation area is not determined in advance, and the search unit 11 determines the next position of the observation area according to the current situation around the robot 2. The search unit 11 may dynamically change the next position of the observation area based on the observation data in the observation history database 12. For example, the search unit 11 may calculate a score representing the possibility that the task area will be extracted in the three-dimensional space around the robot 2 based on the result of extraction of the task area in the observation area at at least one past position, and change the position of the observation area based on the score. The search unit 11 may move the observation area to the position with the highest score. Alternatively, the search unit 11 may calculate a distribution of the uncertainty of the existence of the task area in the three-dimensional space based on the observation data, and change the position of the observation area based on the distribution. The search unit 11 may move the observation area to the position with the highest uncertainty. As a method of considering both the score and the uncertainty, the search unit 11 may execute Bayesian optimization using the observation data as input, and change the position of the observation area based on the acquisition function obtained by this Bayesian optimization. For example, the search unit 11 moves the observation region to the position where the value of the acquisition function is the highest.
[0067] Alternatively, the search unit 11 may select the position with the longest shortest distance to the representative coordinates indicated by the observation data from the subject plane corresponding to the three-dimensional space around the robot, and move the observation area to that position. This process is also an example of dynamically changing the position of the observation area.
[0068] Alternatively, the search unit 11 may change the position of the observation area according to a predetermined order, or may randomly determine the next position of the observation area.
[0069] In response to the search unit 11 determining the next position in the observation area, the robot control unit 15 generates a path for the robot 2 by planning so that the camera 20 captures the next observation area. For example, the robot control unit 15 may plan a path so that the next position is captured at the center of the camera. The robot control unit 15 may select an appropriate position and posture from pre-set candidate positions and postures of the robot 2 so that the camera 20 captures the next observation area, and generate a path for realizing the selected position and posture. As at least a part of the path planning, the robot control unit 15 may execute at least one of generating a path in which the robot 2 does not interfere with an obstacle and generating a path in which a singular posture (singular point) of the robot 2 does not occur.
[0070] The robot control unit 15 outputs a command signal indicating the generated path to the robot controller 3, and the robot controller 3 controls the robot 2 according to the command signal. As a result, the camera 20 captures an image of the next observation area.
[0071] After step S21, the process returns to step S11, and the robot control system 10 executes the processes from step S11 onward. In step S11, the extraction unit 14 extracts a task area from the changed observation area. That is, the extraction unit 14 extracts a task area every time the position of the observation area is changed.
[0072] When it is determined that the search is to be ended (NO in step S21), the process proceeds to step S18, where the robot control system 10 executes an end process.
[0073] An example of the robot's operation based on the process flow S1 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the robot control by the robot control system 10. In this example, the robot 2 executes a task of storing balls present in the vicinity in a box 410. That is, the object in this example is a ball. Fig. 5 shows a series of operations of the robot 2 in sequence by scenes S31 to S39. The following explanation also shows the correspondence with the process flow S1.
[0074] In scene S31, the camera 20 captures an image of the ball 421, and the image is acquired by the image acquisition unit 13. The robot control system 10 extracts a task area corresponding to the ball 421 from the image (observation area) (step S11).
[0075] In scene S32, the robot control system 10 controls the robot 2 so that the robot 2 approaches the ball 421 (steps S12 to S15). This control shortens the distance between the end effector 2b and the ball 421. After this, the robot control system 10 repeats the extraction of the task area and the control of moving the robot 2 closer to the ball 421 (steps S11 to S15).
[0076] In scene S33, in response to the robot 2 reaching the task area (YES in step S14), the robot control system 10 causes the robot 2 to execute the task (step S16).
[0077] From scene S33 to scene S34, the robot 2 executes the task under the control of the robot control system 10 (step S16). The robot 2 grasps the ball 421 with the end effector 2b and puts the ball 421 into the box 410.
[0078] Thereafter, the robot control system 10 attempts to extract a task area while changing the position of the observation area (steps S11, S12, S20 to S22). As a result of this search, in scene S35, the camera 20 acquires an image showing two balls 422 and 423, and the image acquisition unit 13 acquires the image. In this image, the ball 422 is shown as if it is blocked by an obstacle 430, and the entire ball 423 is shown. In one example, the robot control system 10 extracts a task area corresponding to the ball 423 from the image (observation area) (step S11), but does not extract the part corresponding to the ball 422 as a task area.
[0079] In scene S36, the robot control system 10 controls the robot 2 so that the robot 2 approaches the ball 423 (steps S12 to S15). This control shortens the distance between the end effector 2b and the ball 423. After this, the robot control system 10 repeats the extraction of the task area and the control of moving the robot 2 closer to the ball 423 (steps S11 to S15).
[0080] In scene S37, in response to the robot 2 reaching the task area (YES in step S14), the robot control system 10 causes the robot 2 to execute the task (step S16).
[0081] From scene S37 to scene S39, the robot 2 executes the task (step S16) under the control of the robot control system 10. The robot 2 grasps the ball 423 with the end effector 2b (scene S37), carries the ball 423 toward the box 410 (scene S38), and puts the ball 423 into the box 410 (scene S39).
[0082] [program] Each functional module of the robot control system 10 is realized by loading a robot control program onto the processor 161 or memory 162 and having the processor 161 execute the program. The robot control program includes code for realizing each functional module of the robot control system 10. The processor 161 operates the input / output port 164 or the communication port 165 in accordance with the robot control program, and executes reading and writing of data in the memory 162 or the storage 163. Each functional module of the robot control system 10 is realized by such processing.
[0083] The robot control program may be provided in a state in which it is permanently recorded on a non-transitory recording medium such as a CD-ROM, a DVD-ROM, a semiconductor memory, etc. Alternatively, the robot control program may be provided via a communication network as a data signal superimposed on a carrier wave.
[0084] [effect] As described above, a robot control system according to one aspect of the present disclosure includes an extraction unit that extracts a task area in which the robot can execute a task for processing an object from an observation area surrounding the robot, and a robot control unit that controls the robot so that the robot approaches the object in response to the extraction of the task area, and the extraction unit again extracts the task area after the robot approaches the object.
[0085] A robot control method according to one aspect of the present disclosure is a robot control method executed by a robot control system having at least one processor, and includes the steps of extracting a task area in which the robot can perform a task for processing an object from an observation area around the robot, and controlling the robot to approach the object in response to the extraction of the task area, wherein in the extraction step, the task area is extracted again after the robot has approached the object.
[0086] A robot control program according to one aspect of the present disclosure causes a computer to execute the steps of extracting a task area in which the robot can execute a task for processing an object from an observation area around the robot, and controlling the robot in response to the extraction of the task area so that the robot approaches the object, and in the extraction step, the task area is extracted again after the robot approaches the object.
[0087] In this aspect, the task area is extracted every time the robot approaches an object, so that the robot can be controlled while adapting to changes in the situation around the robot, thereby enabling the robot to efficiently execute tasks.
[0088] In the robot control system according to another aspect, the robot control unit may cause the robot to execute the task in response to the robot reaching the task area. With this configuration, the robot can execute the task when it reaches a position where it can process the object. Therefore, it is possible to cause the robot to execute the task efficiently.
[0089] In another aspect, the robot control system may further include an image acquisition unit that acquires an image of the observation area from a camera that moves in response to the robot's movement, and the extraction unit may extract the task area based on the image. By using the image from such a camera, the robot's surroundings can be clearly and reliably recognized. As a result, the task area can be extracted with high accuracy.
[0090] In a robot control system according to another aspect, the extraction unit may extract a plurality of task areas, and the robot control unit may select one task area from the plurality of task areas and control the robot so that the robot approaches the selected task area. With this configuration, even when a plurality of task areas are extracted, the robot can be appropriately caused to approach a specific task area.
[0091] In the robot control system according to another aspect, the extraction unit may extract a task region corresponding to each of a plurality of tasks different from one another. With this configuration, the robot can be efficiently controlled in accordance with each type of task.
[0092] In the robot control system according to another aspect, the extraction unit may extract the task area based on the configuration of the robot and the current situation in the observation area. By taking these two factors into consideration, the task area can be extracted with high accuracy.
[0093] In a robot control system according to another aspect, the current situation may indicate whether or not an object is present within the observation area, and the extraction unit may extract the task area based at least on the presence of the object within the observation area. By causing the robot to approach only after recognizing the presence of the object, it is possible to cause the robot to execute a task while suppressing unnecessary movements of the robot.
[0094] In a robot control system according to another aspect, the current situation may further indicate whether or not an obstacle different from the target object is present within the observation area, and the extraction unit may extract the task area based further on the fact that the robot does not interfere with the obstacle when executing the task. By approaching the robot while taking into consideration the presence of the obstacle, it is possible to cause the robot to execute the task while suppressing unnecessary movements of the robot.
[0095] In a robot control system according to another aspect, the configuration of the robot may include a configuration of an end effector attached to the robot, the current situation may further indicate a structure of an object, and the extraction unit may extract the task region by comparing the configuration of the end effector with the structure of the object. By approaching the robot to the object while taking into consideration the structure of the object and the configuration of the end effector that handles the object, it is possible to cause the robot to execute a task while suppressing unnecessary movements of the robot.
[0096] In the robot control system according to another aspect, the extraction unit may calculate a probability that the task area exists, and extract the task area when the probability reaches a given threshold. By extracting the task area probabilistically, it is possible to control the robot while flexibly adapting to the situation around the robot.
[0097] In the robot control system according to another aspect, the extraction unit may calculate a probability distribution in the observation area and extract an area where the probability reaches a threshold as the task area. By taking the probability distribution into consideration, the task area can be extracted from the observation area with high accuracy and efficiency.
[0098] In another aspect, the robot control system may further include a search unit that changes the position of an observation area in a three-dimensional space around the robot in response to the extraction unit not extracting the task area. By changing the position of the observation area when the task area is not extracted, it is possible to cause the robot to execute a task while suppressing unnecessary movements of the robot.
[0099] In the robot control system according to another aspect, the search unit may dynamically change the position of the observation area in three-dimensional space. With this configuration, the observation area can be appropriately set according to the situation around the robot.
[0100] In a robot control system according to another aspect, the search unit may calculate a score representing a possibility that the task area will be extracted in the three-dimensional space based on a result of extracting the task area in the observation area at least one past position, and change the position of the observation area based on the score. By setting the observation area based on the degree of possibility that the task area will be extracted based on the past result, it becomes possible to efficiently extract the task area.
[0101] In the robot control system according to another aspect, the search unit may calculate a distribution of uncertainty of the existence of the task area in three-dimensional space and change the position of the observation area based on the distribution of the uncertainty. By taking the uncertainty into consideration, the observation area is set efficiently, and as a result, the task area can be extracted efficiently.
[0102] In the robot control system according to another aspect, the extraction unit may extract the task area each time the position of the observation area is changed. By extracting the task area while changing the observation area, it is possible to cause the robot to execute the task efficiently.
[0103] In the robot control system according to another aspect, the task may include a step of causing the robot to come into contact with an object, and the extraction unit may extract, as the task area, an area in which the robot can come into contact with the object in order to execute the task. With this configuration, it is possible to cause the robot to efficiently execute the task involving contact with the object.
[0104] A robot system according to one aspect of the present disclosure includes the robot control system and a robot that operates under the control of a robot controller. In this aspect, it is possible to cause the robot to efficiently execute tasks.
[0105] [Variations] The present disclosure has been described in detail above based on the embodiments. However, the present disclosure is not limited to the above embodiments. Various modifications are possible without departing from the gist of the present disclosure.
[0106] The functional configuration of the robot control system is not limited to the above example, and the robot control method according to the present disclosure may be executed using a functional configuration different from the above example.
[0107] The hardware configuration of the system is not limited to the embodiment in which each functional module is realized by executing a program. For example, at least a part of the functional modules in the above example may be configured by a logic circuit specialized for the function, or may be configured by an ASIC (Application Specific Integrated Circuit) that integrates the logic circuit.
[0108] The processing procedure of the method executed by at least one processor is not limited to the above example. For example, some of the above steps (processing) may be omitted, or each step may be executed in a different order. Any two or more of the above steps may be combined, or some of the steps may be modified or deleted. Alternatively, other steps may be executed in addition to each of the above steps. In the above processing flow S1, each step may be executed directly, or some steps may be executed in parallel.
[0109] When comparing the magnitude relationship of two numerical values within a computer system or computer, either of the two criteria of "greater than or equal to" and "greater than" may be used, or either of the two criteria of "less than or equal to" and "less than." The selection of such criteria does not change the technical meaning of the process of comparing the magnitude relationship of two numerical values. [Explanation of symbols]
[0110] 1...robot system, 2...robot, 11...search section, 12...observation history database, 13...image acquisition section, 14...extraction section, 15...robot control section, 2b...end effector, 3...robot controller, 10...robot control system, 20...camera, 100...computer, 110...main body, 120...monitor, 130...input device, 200...image, 221, 222...orange (object), 300...area map, 311, 312...task area, 421-423...ball (object).
Claims
1. An image acquisition unit that acquires images of an observation area around a robot from a camera that moves in response to the movement of the robot; an extraction unit that extracts, from the observation area, an area in which the robot can come into contact with the object in order to execute a task for processing the object, based on the image; and a robot control unit that controls the robot so that the robot approaches the object in response to the extraction of the task area; Equipped with The extraction unit extracts the task area again after the robot approaches the object. Robot control system.
2. the robot control unit causes the robot to execute the task in response to the robot reaching the task area; The robot control system of claim 1 .
3. The extraction unit extracts a plurality of the task areas, The robot control unit includes: selecting a task area from the plurality of task areas; controlling the robot such that the robot approaches the selected task area; The robot control system according to claim 1 or 2.
4. the extraction unit extracts, for each of a plurality of tasks different from one another, the task area corresponding to the task; The robot control system according to any one of claims 1 to 3.
5. The extraction unit extracts the task area based on a configuration of the robot and a current situation in the observation area. The robot control system according to any one of claims 1 to 4.
6. The current situation indicates whether the object is present within the observation area; The extraction unit extracts the task area based at least on the presence of the object in the observation area. The robot control system of claim 5.
7. The current situation further indicates whether an obstacle different from the object is present within the observation area; The extraction unit extracts the task area based further on the fact that the robot does not interfere with the obstacle when performing the task. The robot control system of claim 6.
8. the configuration of the robot includes a configuration of an end effector attached to the robot; The current situation further indicates a structure of the object; The extraction unit compares a configuration of the end effector with a structure of the object to extract the task region.
8. A robot control system according to claim 6 or 7.
9. The extraction unit is Calculating the probability that the task area exists; extracting the task region if the probability reaches a given threshold; The robot control system according to any one of claims 1 to 8.
10. The extraction unit is Calculating a distribution of the probabilities in the observation region; extracting a region where the probability reaches the threshold as the task region; The robotic control system of claim 9.
11. The robot control system according to any one of claims 1 to 10, further comprising a search unit that changes the position of the observation area in a three-dimensional space around the robot in response to the extraction unit not extracting the task area.
12. The search unit dynamically changes a position of the observation region in the three-dimensional space. The robotic control system of claim 11.
13. The search unit is calculating a score representing a possibility that the task area will be extracted in the three-dimensional space based on a result of the extraction of the task area in the observation area at at least one past position; changing a position of the observation region based on the score; 13. A robot control system according to claim 11 or 12.
14. The search unit is Calculating a distribution of uncertainty of the existence of the task region in the three-dimensional space; changing a position of the observation region based on the distribution of the uncertainties; The robotic control system of claim 13.
15. The extraction unit extracts the task area every time the position of the observation area is changed. A robot control system according to any one of claims 11 to 14.
16. A robot control system according to any one of claims 1 to 15; a robot that operates under the control of the robot control unit; A robot system comprising:
17. 1. A robot control method executed by a robot control system having at least one processor, comprising: acquiring an image of an observation area around the robot from a camera that moves in response to the movement of the robot; extracting, from the observation area, an area in which the robot can contact an object in order to execute a task for processing the object, based on the image, as a task area; In response to the task area being extracted, controlling the robot so that the robot approaches the object; Including, In the extracting step, the task area is extracted again after the robot approaches the object. A method for controlling a robot.
18. A step of acquiring an image of an observation area around a robot from a camera that moves in response to the movement of the robot; extracting, from the observation area, an area in which the robot can contact an object in order to execute a task for processing the object, based on the image, as a task area; In response to the task area being extracted, controlling the robot so that the robot approaches the object; Run the following on your computer: In the extracting step, the task area is extracted again after the robot approaches the object. Robot control program.
Citation Information
Patent Citations
Active environment recognition system
JP1996030327A
Mobile body tracking apparatus and mobile body tracking method
JP2012225752A
Mobile body control system, mobile body, display method, and program
JP2020127102A
Imaging apparatus, method and program
JP2021044749A
Autonomous mobile robot and control program for autonomous mobile robot
JP2021060849A
