Control device and control method

Abstract

To provide a control device that can make highly accurate decisions based on the surrounding environment of a robot. [Solution] The control device according to the present invention is a control device for controlling the movement of a robot, comprising: an acquisition unit that acquires an image of the work environment; an extraction unit that extracts a plurality of objects included in the image using machine learning; and a determination unit that makes a determination regarding whether or not a movement operation to be performed by the robot is possible, using a scene graph that shows the relationships between the objects in the work environment, which is identified based on feature quantities that indicate at least one of the positions and states of the extracted objects.

Description

Technical Field

[0001] The present invention relates to a control device and a control method for a robot.

Background Art

[0002] There is known a control technique for a robot that generates a trajectory of an arm considering obstacles by a machine learning model. In Patent Document 1, based on the position of an object detected in the operating environment of a device and the operating position of the device, an operating path of the device is specified, and based on the operating information generated based on the position information of a plurality of points included in the operating environment, etc., a technique for controlling the device is described.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique described in Patent Document 1, a new annotation work is required every time the robot working environment is changed. Therefore, from the viewpoint of selecting an optimal operation according to the position of an obstacle or the like in the working environment, a technique that can more accurately determine the situation around the robot without performing annotation work is desired.

[0005] An object of the present invention is to provide a control device that can control a robot with high accuracy based on the situation around the robot.

Means for Solving the Problems

[0006] The control device according to the present invention is a control device for controlling the movement of a robot, comprising: an acquisition unit that acquires an image of the work environment; an extraction unit that extracts a plurality of objects included in the image; and a determination unit that determines whether or not a movement operation performed by the robot is permissible, using a scene graph that shows the relationships between the objects in the work environment, which is identified based on a feature quantity that indicates at least one of the positions and states of the extracted objects. [Effects of the Invention]

[0007] The present invention provides a control device that can control a robot with high precision based on the surrounding conditions. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the functional configuration of a control device according to the first embodiment of the present invention. [Figure 2] This is a flowchart showing the control method executed by the control device according to the first embodiment of the present invention. [Figure 3] This figure shows an example of a scene graph according to the first embodiment of the present invention. [Figure 4] This figure shows an example of a scene graph according to the first embodiment of the present invention. [Figure 5] This figure shows an example of a scene graph according to the first embodiment of the present invention. [Figure 6] This figure shows an example of a scene graph according to a second embodiment of the present invention. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings. Note that the drawings are simplified, and the technical scope of the embodiments should not be narrowly interpreted based on their depiction. Furthermore, the same elements are denoted by the same reference numerals, and redundant explanations are omitted. Also, in the following embodiments, when referring to the number of elements (including quantity, numerical value, amount, range, etc.), unless specifically stated or clearly limited to a particular number in principle, the number is not limited to that particular number and may be greater than or less than that number.

[0010] Furthermore, in the following embodiments, the components are not necessarily essential unless otherwise explicitly stated or considered to be clearly essential in principle. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc., of the components, etc., it shall include those that substantially approximate or resemble their shape, etc., unless otherwise explicitly stated or considered to be clearly not essential in principle. The same applies to the numbers, etc. (including the number of items, numerical values, quantities, ranges, etc.).

[0011] [First Embodiment] <Configuration of the control device 10> Figure 1 is a block diagram showing the functional configuration of a control device 10 according to a first embodiment of the present invention. The control device 10 includes an acquisition unit 11, an extraction unit 12, and a determination unit 13. The control device 10 controls the operation of the robot 3 based on the conditions of multiple objects 2 in the work environment 1 and the conditions around the robot 3.

[0012] The control unit 10 comprises a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and an input / output interface (I / F) as its hardware configuration. These are electrically connected to each other via a bus. The CPU controls the operation of the control unit 10. The ROM stores programs and other data executed by the CPU. The RAM is used as the CPU's workspace. The HDD stores various data such as programs. The input / output interface is an interface for inputting and outputting various signals and data to and from external devices.

[0013] The acquisition unit 11 of the control device 10 acquires images of the work environment 1. The work environment 1 is various facilities such as a factory. Preferably, the images of the work environment 1 acquired by the acquisition unit 11 include at least objects 2 related to the operation of the robot 3. An imaging device (not shown) that takes images is, for example, a digital camera with an image sensor consisting of a CCD (charge coupled device). The imaging device may also be a device built into a terminal device such as a mobile phone, smartphone, or tablet.

[0014] Furthermore, the acquisition unit 11 acquires temperature information from temperature sensors (not shown) that measure the temperature of the object 2, the robot 3, and their surroundings in the work environment 1. The temperature sensors are, for example, a thermographic camera, an infrared camera, or a thermal imaging camera.

[0015] Furthermore, the acquisition unit 11 may acquire distance information of the work environment 1 from a distance measuring sensor (not shown). Examples of distance measuring sensors include laser sensors (LiDAR: Light Detection and Ranging), radar sensors (RADAR: Radio Detection and Ranging), millimeter-wave radar, infrared sensors, and ultrasonic sensors.

[0016] Note that the acquisition unit 11 can acquire various types of information, not limited to the image, temperature information, and distance measurement information of the working environment 1. By acquiring temperature information, distance measurement information, etc. in addition to the image of the working environment 1, the control device 10 can control the operation of the robot 3 with higher precision based on the situation of the working environment 1.

[0017] The extraction unit 12 extracts a plurality of objects 2 included in the image of the working environment 1 that has been imaged. The extraction unit 12 uses an existing object detection algorithm to extract various facilities in the working environment 1, such as a person like an operator who is an object 2, an object that is an object to be manufactured and processed, etc. from the image of the working environment 1. An existing object detection algorithm is, for example, YOLO (You Only Look Once), but it is not limited to this.

[0018] The determination unit 13 makes a determination regarding the feasibility of the movement operation executed by the robot 3 using a scene graph indicating the relationship between the objects 2 in the working environment 1 identified based on at least one of the feature amounts indicating the position and state of the extracted objects 2. The scene graph may indicate the relationship between the object 2 and the robot 3.

[0019] In the control device 10 according to the present embodiment, the determination regarding the movement operation of the robot 3 performed by the determination unit 13 is assumed to be a determination regarding the feasibility of the movement operation in which the robot 3 picks up an object that is the object 2 and places it at a predetermined position. Note that the determination regarding the movement operation of the robot 3 is not limited to this, and may be a determination regarding various movement operations for moving the object.

[0020] The feature amount is calculated using data indicating the status of the object 2 and the robot 3 extracted from the image of the working environment 1, such as information regarding the distance between objects 2 and between the object 2 and the robot 3. Further, the feature amount includes information regarding the temperature of at least either the working environment 1 or the object 2. Furthermore, the feature amount may include information regarding the posture of the robot 3. The posture of the robot 3 may be calculated based on the distance measurement information acquired by the acquisition unit 11 from the distance measurement sensor.

[0021] The relationships between objects 2 and between the object 2 and the robot 3 include various relationships, such as, for example, the worker who is the object 2 touching the target object, the distance between the worker and the target object, and the distance between the worker and the robot 3. These relationships may be specified using a machine learning model. The machine learning model is generated, for example, by machine learning using the image of the working environment 1 as input data and the object 2, the robot 3, and information regarding these relationships as correct labels.

[0022] Also, the structure of the scene graph is generated in advance. The scene graph may be generated using a machine learning model. The machine learning model is generated by machine learning using the image of the working environment 1, the locations of the object 2 and the robot 3 included in the image, the type of the object 2, and the relationships between objects 2 etc. as correct labels. Further, the scene graph may be based on rules for making a determination regarding the feasibility of the movement operation executed by the robot 3.

[0023] The determination unit 13 makes a determination regarding the feasibility of the movement operation executed by the robot 3 by inputting the feature amount into the pre-created structure of the scene graph. More specifically, by using the scene graph, the feasibility of the movement operation of the robot 3 may be calculated as a probability. For example, if the probability that the robot 3 may execute the movement operation is 65% or more, the determination unit 13 can make a determination to cause the robot 3 to execute the movement operation. Note that the determination method by the determination unit 13 is not limited to this.

[0024] The control device 10 controls the movement of the robot 3 based on the determination made by the determination unit 13. The control device 10 extracts multiple objects 2 from the acquired image of the work environment 1, and determines whether or not the robot 3 can perform a movement operation on the objects 2 by taking an overview of the situation of each object 2 in the work environment 1.

[0025] Therefore, the control device 10 controls the movement of the robot 3 based on factors such as the position of object 2 that obstructs the robot 3's movement, and whether or not an operator is touching the object 2 that the robot 3 is to move. As a result, the control device 10 allows for highly accurate control of the robot 3 based on the surrounding conditions.

[0026] Furthermore, since the control device 10 uses a scene graph to determine whether to execute a movement for the robot 3, even if there are changes in the work environment 1, it can easily identify the situation around the robot 3 and control the robot 3's movements without performing annotation work.

[0027] <Control Method> Figure 2 is a flowchart showing the control method executed by the control device 10 according to the first embodiment of the present invention.

[0028] The acquisition unit 11 of the control device 10 acquires an image of the work environment 1 (S101). The extraction unit 12 extracts multiple objects 2 contained in the image of the work environment 1 acquired by the acquisition unit 11 (S102). The determination unit 13 inputs the relationships between the objects 2 in the work environment 1, etc., identified based on feature quantities indicating at least one of the positions and states of the extracted objects 2, into the scene graph (S103). The determination unit 13 uses the scene graph to determine whether or not the robot 3 can perform a movement (S104).

[0029] These steps carry out the control method according to one embodiment of the present invention. However, the control method according to one embodiment of the present invention may include other steps as appropriate, depending on the measurement conditions, measurement environment, etc.

[0030] <Examples> Figures 3 to 5 show examples of scene graphs according to the first embodiment of the present invention. The determination unit 13 uses each scene graph to determine whether or not the robot 3 can perform a movement. Figure 3 is a scene graph according to Embodiment 1, Figure 4 is an example according to Embodiment 2, and Figure 5 is an example according to Embodiment 3. Note that the scene graphs are not limited to the examples shown in Figures 3 to 5, and various other scene graphs can be used.

[0031] <Example 1> In Figure 3, (a) is a diagram showing an overview of the work environment 1 according to this embodiment, and (b) is an example of a scene graph showing the work environment 1 shown in (a).

[0032] The work environment 1 shown in Figure 3(a) contains multiple objects 2, namely object 2a, a worker 2b, desks 2c and 2d, and a robot 3. Worker 2b is manually processing object 2a at a position within 50 cm of object 2a. Robot 3 picks up object 2a from desk 2c and places it on desk 2d, which is a designated position. The work environment 1 is also assumed to have other equipment, such as doors, which are not shown.

[0033] The scene graph in Figure 3(b) is a scene graph generated based on feature quantities calculated based on the image of the work environment 1 shown in Figure 3(a). The acquisition unit 11 acquires the image and temperature information of the work environment 1. The thick lines in the illustrated scene graph indicate that worker 2b is performing manual work at a desk 2c at a distance of 50 cm or less from object 2a.

[0034] The scene graph indicates that it is dangerous for the robot 3 to move object 2a while worker 2b is working, and that there is a high probability that object 2a cannot be moved. In this case, the determination unit 13 of the control device 10 determines that the robot 3 should not perform the movement action.

[0035] The illustrated scene graph also shows that, in addition to the above, in working environment 1, if the temperature of object 2a is 60°C or higher, and if the distance between the door and object 2a is 50 cm or less and they interfere with each other, it is dangerous and therefore object 2a cannot be moved.

[0036] Furthermore, the scene graph indicates that the robot 3 is unable to move object 2a because it performs an abnormality correction when worker 2b and robot 3 are working together and the distance between them is 20 cm or less. In addition to the above, the scene graph indicates whether robot 3 can move object 2a based on various conditions in the work environment 1.

[0037] <Example 2> Figure 4 shows a scene graph where a bag is present as an obstacle in the work environment 1 shown in Figure 3(a), and the robot 3 moves the object 2a on a cart. In addition, the acquisition unit 11 of the control device 10 acquires only images of the work environment 1 and does not acquire temperature information.

[0038] The illustrated scene graph shows that when object 2a is placed on the trolley, robot 3 may move object 2a using the trolley by performing startup preparations. "Startup preparations" in the diagram indicates preparations to start the machine tool when processing object 2a to manufacture a product. "Error handling" indicates the action taken to eliminate an error if there is an error in the machine tool and an alarm has been triggered.

[0039] Furthermore, the illustrated scene graph shows that worker 2b and robot 3 are working together, and if the distance between them is 50 cm or less, abnormal action is being taken, indicating that the object 2a cannot be moved. In addition to the above, the scene graph shows whether or not robot 3 can move object 2a based on various conditions in the work environment 1.

[0040] By using the illustrated scene graph, the control device 10 can determine whether the robot 3 can move the object 2a based on the possibility of contact between, for example, the worker 2b and the robot 3 if there are obstacles between them. Therefore, it becomes unnecessary to limit the size of the fence that should be placed between the robot 3 and the worker 2b.

[0041] <Example 3> The scene graph shown in Figure 5 illustrates a scenario where the work environment 1 is, for example, a kitchen or other cooking area. The acquisition unit 11 acquires images and temperature information of the work environment 1. The robot 3 aims to move the object 2a, which is a kettle.

[0042] The work environment 1 is assumed to contain a kettle, cutting board, knife, vegetables, and pot. The temperature information acquired by the acquisition unit 11 indicates the temperature of the kettle. In the illustrated scene graph, the types of objects 2 can be classified as vegetables, pots, etc., and the status of the kettle, etc., can be classified as in use, processing, etc.

[0043] The illustrated scene graph shows, for example, that if a kettle with a handle is below 40°C, robot 3 can move the kettle. It also shows that if the kettle's temperature is above 80°C, robot 3 cannot move it because it is heating up. In addition to these, the illustrated scene graph shows whether robot 3 can move the object 2a (the kettle) based on various conditions in the work environment 1.

[0044] By using the illustrated scene graph, the control device 10 can roughly determine the status of, for example, the object 2a, which is a kettle, and can select a safe and efficient method for moving the kettle to control the operation of the robot 3.

[0045] <Effects of the control device 10 according to this embodiment> The control device 10 surveys the status of each object 2 in the work environment 1 and uses a scene graph to determine whether or not the robot 3 can perform movement actions on the objects 2. Therefore, even if there are changes in the work environment 1, the control device 10 can easily identify the situation around the robot 3 without performing annotation work. Furthermore, the control device 10 can control the robot 3 simply and with high accuracy based on the situation around the robot 3.

[0046] [Second Embodiment] The control device 10 according to this embodiment, like the control device 10 according to the first embodiment, includes an acquisition unit 11, an extraction unit 12, and a determination unit 13, and controls the operation of the robot 3. Note that components identical to those already described are denoted by the same reference numerals, and redundant explanations are omitted.

[0047] In this embodiment, unlike the first embodiment, the control device 10 does not determine whether or not the robot 3, which is the object to be controlled, can move object 2, but rather makes a determination regarding the movement path of the robot 3 itself. More specifically, the determination unit 13 of the control device 10 makes a determination regarding the movement of the robot 3, which involves controlling changes to the robot 3's movement path within the work environment 1, deceleration, straight movement, and stopping.

[0048] Figure 6 shows an example of a scene graph according to a second embodiment of the present invention. The work environment 1 is assumed to include an object 2, which is a worker 2b, a door, a trolley as an obstacle, and various equipment. The acquisition unit 11 acquires distance information from an image of the work environment 1 and a distance sensor.

[0049] The mobile robot 3 is configured, for example, as an autonomous mobile robot that moves autonomously. Robot 3 is, for example, an autonomous mobile robot (AMR) and an automated guided vehicle (AGV), but is not limited to these and may be any device that can move within the work environment 1.

[0050] The illustrated scene graph shows that, for example, if worker 2b and a cart (an obstacle) are in the path of robot 3, and there is empty space on either side of them, the control device 10 determines that robot 3 should change its path. Also, for example, if worker 2b is moving the cart, if the distance between worker 2b and the door is 2m or less and there is a possibility that the door will open, and if worker 2b is in the path, the control device 10 determines that robot 3 should be slowed down.

[0051] Furthermore, the control device 10 determines whether to move the robot 3 in a straight line, for example, when worker 2b is not moving, or when worker 2b is debugging equipment. In addition, if the distance between the robot 3 and worker 2b is 50 cm or less, there is no room for movement, and the control device 10 stops the robot 3. The illustrated scene graph shows whether or not the robot 3 can move based on various conditions in the work environment 1, in addition to these.

[0052] <Effects of the control device 10 according to this embodiment> The control device 10 can appropriately determine the path and speed to avoid obstacles according to the conditions of the work environment 1, and control the robot 3.

[0053] Although the present invention has been described above in accordance with the embodiments described above, the present invention is not limited to the configuration of the embodiments described above, and of course includes various modifications, alterations, and combinations that can be made by a person skilled in the art within the scope of the claims of the present patent application. [Explanation of Symbols]

[0054] 1. Working Environment 2 objects 3 Robots 10 Control device 11 Acquisition Department 12 Extraction part 13 Judgment section

Claims

1. A control device for controlling the movement of a robot, An acquisition unit that acquires images of the work environment, An extraction unit for extracting multiple objects contained in the aforementioned image, A determination unit that determines whether or not the robot can perform a movement, using a scene graph that shows the relationships between the objects in the work environment, which is identified based on feature quantities that indicate at least one of the positions and states of the extracted objects, A control device equipped with the following features.

2. The determination made by the determination unit regarding the movement operation is a determination of whether or not the robot can pick up the object and place it in a predetermined position. The control device according to claim 1.

3. The determination made by the determination unit regarding the movement is a determination regarding the movement path of the robot itself. The control device according to claim 1.

4. The acquisition unit further acquires temperature information, The aforementioned feature includes information regarding the temperature of at least one of the working environment and the object. The control device according to claim 1.

5. A control method performed by a control device that controls the movement of a robot, The steps include acquiring images of the work environment, The steps include extracting multiple objects contained in the aforementioned image, A step of determining whether the robot can perform a movement, using a scene graph that shows the relationships between the objects in the work environment, which is identified based on feature quantities that indicate at least one of the positions and states of the extracted objects; A control method including

Images