Control device and control method
By acquiring images of the working environment and generating scene diagrams to determine the feasibility of robot movement actions, the problem of decreased control accuracy when the robot's working environment changes is solved, and high-precision robot control is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TOYOTA JIDOSHA KK
- Filing Date
- 2025-11-06
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, whenever the robot's operating environment changes, new annotation work is required to select the best action, which leads to a decrease in control accuracy.
By acquiring images of the working environment, extracting multiple objects, and using a machine learning model to generate a scene map, the feasibility of the robot's movement actions is determined, and the robot's actions are controlled.
It enables high-precision control of robot movements without the need for annotation when the robot's operating environment changes, thus improving control accuracy.
Smart Images
Figure CN122008178A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a control device and control method for a robot. Background Technology
[0002] A robot control technique is known to generate an arm trajectory that takes obstacles into account using a machine learning model. Patent Document 1 describes a technique that determines the motion path of a device based on the position of an object detected in the device's motion environment and the device's motion position, and controls the device based on motion information generated from position information of multiple points contained in the motion environment.
[0003] Patent Document 1: Japanese Patent Application Publication No. 2022-077229 Summary of the Invention
[0004] In the technology described in Patent Document 1, new annotation work is required whenever the robot's working environment changes. Therefore, from the viewpoint of selecting the optimal action based on the position of obstacles and the like in the working environment, a technology is desired that can determine the robot's surroundings with higher accuracy without performing annotation work.
[0005] The purpose of this invention is to provide a control device that can control a robot with high precision based on the robot's surrounding conditions.
[0006] The control device of the present invention controls the actions of a robot, comprising: an acquisition unit that acquires an image of the work environment; an extraction unit that extracts a plurality of objects contained in the image; and a determination unit that uses a scene graph representing the relationships between the objects in the work environment, determined based on feature quantities representing at least one of the positions and states of the extracted objects, to determine whether a movement action performed by the robot is permissible.
[0007] Invention Effects
[0008] This invention provides a control device that can control a robot with high precision based on the robot's surrounding conditions. Attached Figure Description
[0009] Figure 1 This is a block diagram illustrating the functional structure of the control device according to the first embodiment of the present invention.
[0010] Figure 2 This is a flowchart illustrating the control method executed by the control device according to the first embodiment of the present invention.
[0011] Figure 3 This is an example of a scene diagram according to the first embodiment of the present invention.
[0012] Figure 4 This is an example of a scene diagram according to the first embodiment of the present invention.
[0013] Figure 5 This is an example of a scene diagram according to the first embodiment of the present invention.
[0014] Figure 6 This is an example of a scene diagram according to the second embodiment of the present invention. Detailed Implementation
[0015] Hereinafter, embodiments will be described with reference to the accompanying drawings. However, since the drawings are simplified, the technical scope of the embodiments should not be narrowly interpreted based on the description in the drawings. Furthermore, the same symbols are used to denote the same elements, and repeated descriptions are omitted. Also, in the following embodiments, when referring to the quantity of elements (including number, value, quantity, range, etc.), the quantity is not limited to a specific number, except in particularly explicit cases or cases where the quantity is clearly limited in principle; it may be more than a specific number or less.
[0016] Furthermore, in the following embodiments, the constituent elements are not necessarily essential, except in cases where they are particularly clear or where they are obviously considered necessary in principle. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc., of constituent elements, etc., this includes those that are substantially similar or analogous to their shape, etc., except in cases where they are particularly clear or where they are obviously not considered necessary in principle. This also applies to the quantities (including number, value, quantity, range, etc.) mentioned above.
[0017] [First Implementation]
[0018] <Structure of Control Device 10>
[0019] Figure 1 This is a block diagram illustrating the functional structure of the control device 10 according to the first embodiment of the present invention. The control device 10 includes an acquisition unit 11, an extraction unit 12, and a determination unit 13. The control device 10 controls the actions of the robot 3 based on the conditions of multiple objects 2 in the working environment 1 and the surrounding conditions of the robot 3.
[0020] The control device 10 comprises a Central Processing Unit (CPU), Read-Only Memory (ROM), Random Access Memory (RAM), Hard Disk Drive (HDD), and an Input / Output Interface (I / F) as its hardware structure. These components are electrically connected to each other via a bus. The CPU controls the operation of the control device 10. The ROM stores programs executed by the CPU. The RAM serves as the CPU's working area. The HDD stores programs and other data. The I / F interface is used for inputting and outputting various signals or data with external devices.
[0021] The acquisition unit 11 of the control device 10 acquires images of the working environment 1. The working environment 1 is various facilities such as a factory. The images of the working environment 1 acquired by the acquisition unit 11 preferably include at least the object 2 related to the movement of the robot 3. The imaging device (not shown) that captures the images is, for example, a digital camera with a built-in imaging element composed of a charge-coupled device (CCD). The imaging device can be a device built into a terminal device such as a mobile phone, smartphone, or tablet computer.
[0022] Furthermore, the acquisition unit 11 acquires temperature information of the object 2, robot 3, and their surroundings in the measurement working environment 1 from a temperature sensor not shown in the figure. The temperature sensor may be, for example, a thermal imaging camera, an infrared camera, or a thermal imager.
[0023] Furthermore, the acquisition unit 11 can acquire ranging information of the operating environment 1 from a ranging sensor not shown in the figure. Regarding the ranging sensor, for example, light detection and ranging (LiDAR) as a laser sensor, radio detection and ranging (RADAR) as a radar sensor, millimeter-wave radar, infrared sensors, and ultrasonic sensors can be used.
[0024] Furthermore, the acquisition unit 11 is not limited to images, temperature information, and distance measurement information of the working environment 1, but can acquire various types of information. By acquiring temperature information and distance measurement information in addition to images of the working environment 1, the control device 10 can control the robot 3's movements with higher precision based on the conditions of the working environment 1.
[0025] The extraction unit 12 extracts multiple objects 2 contained in the image captured of the work environment 1. The extraction unit 12 uses existing object detection algorithms to extract various equipment in the work environment 1, such as workers as objects 2 and objects as manufacturing and processing objects, from the image of the work environment 1. Existing object detection algorithms include, for example, "YOU Only Look Once" (YOLO), but are not limited to this.
[0026] The determination unit 13 uses a scene diagram representing the relationships between objects 2 in the work environment 1, determined based on feature quantities representing at least one of the extracted object 2's position and state, to determine whether a movement action performed by the robot 3 is permissible. The scene diagram can represent the relationship between object 2 and robot 3.
[0027] In the control device 10 according to this embodiment, the determination unit 13 determines whether the robot 3's movement action is permissible by picking up the object 2 and placing it in a predetermined position. However, the determination of the robot 3's movement action is not limited to this; it can also be a determination of various movement actions that move the object.
[0028] The feature quantities include, for example, information about the distances between the objects 2 and between the objects 2 and the robot 3, calculated using data representing the conditions of the objects 2 and the robot 3 extracted from the image of the working environment 1. Furthermore, the feature quantities include information about the temperature of at least one of the working environment 1 and the objects 2. Additionally, the feature quantities may include information about the posture of the robot 3. The posture of the robot 3 can be calculated based on ranging information acquired by the acquisition unit 11 from the ranging sensor.
[0029] The relationships between objects 2 and between objects 2 and robot 3 include, for example, various relationships such as the worker's contact with objects 2, the distance between the worker and the objects, and the distance between the worker and robot 3. These relationships can be determined using a machine learning model. This machine learning model is generated, for example, by using an image of the work environment 1 as input data and machine learning to correctly label objects 2 and robot 3, along with information about these relationships.
[0030] Furthermore, the structure of the scene graph is pre-generated. The scene graph can be generated using a machine learning model. This machine learning model generates the scene graph by using machine learning to correctly label the image of the work environment 1 and the locations of the objects 2 and robot 3 contained in the image, the types of objects 2, and the relationships between objects 2. Additionally, the scene graph can also be based on rules used to determine the feasibility of the movement actions performed by robot 3.
[0031] The determination unit 13 determines whether a movement action performed by the robot 3 is permissible by inputting feature quantities into the structure of a pre-created scene graph. More specifically, by using the scene graph, the permissibility of the robot 3's movement action can be calculated as a probability. For example, if the probability that the robot 3 can perform the movement action is 65% or higher, the determination unit 13 can determine whether the robot 3 should perform the movement action. However, the determination method based on the determination unit 13 is not limited to this.
[0032] The control device 10 controls the actions of the robot 3 based on the determination made by the determination unit 13. The control device 10 extracts multiple objects 2 from the acquired image of the work environment 1, and determines whether the robot 3 can perform the movement action of the object 2 by looking at the status of each object 2 in the work environment 1.
[0033] Therefore, the control device 10 controls the movement of the robot 3 based on factors such as the position of the object 2 that becomes an obstacle to the robot 3's movement and whether the operator comes into contact with the object 2 that enables the robot 3 to move. Thus, according to the control device 10, the robot 3 can be controlled with high precision based on the surrounding conditions.
[0034] Furthermore, the control device 10 uses a scene map to determine whether the robot 3 should perform a movement action. Therefore, even if the working environment 1 changes, the surrounding conditions of the robot 3 can be easily determined without annotation work, and the actions of the robot 3 can be controlled.
[0035] <Control Methods>
[0036] Figure 2 This is a flowchart illustrating the control method executed by the control device 10 according to the first embodiment of the present invention.
[0037] The acquisition unit 11 of the control device 10 acquires an image of the work environment 1 (S101). The extraction unit 12 extracts multiple objects 2 contained in the image of the work environment 1 acquired by the acquisition unit 11 (S102). The determination unit 13 inputs the relational relationship between the objects 2 in the work environment 1, determined based on feature quantities representing at least one of the positions and states of the extracted objects 2, into the scene graph (S103). The determination unit 13 uses the scene graph to determine whether the movement action performed by the robot 3 is permissible (S104).
[0038] These steps implement the control method according to one embodiment of the present invention. However, the control method according to one embodiment of the present invention may appropriately include other steps depending on the measurement conditions or measurement environment.
[0039] <Example>
[0040] Figures 3-5This is an example of a scene diagram according to the first embodiment of the present invention. The determination unit 13 uses the scene diagram to determine whether a movement action performed by the robot 3 is permissible. Figure 3 This is a scene diagram related to Example 1. Figure 4 This is a scene diagram related to Example 2. Figure 5 This is the scene diagram involved in Example 3. Furthermore, the scene diagram is not limited to... Figures 3-5 The examples shown can use various scene diagrams.
[0041] <Example 1>
[0042] exist Figure 3 (a) is a diagram showing an outline of the operating environment 1 involved in this embodiment. Figure 3 (b) means Figure 3 (a) is an example of a scene diagram of the work environment 1 shown.
[0043] exist Figure 3 (a) The work environment 1 shown contains an object 2a (which is one of several objects 2), a worker 2b, tables 2c and 2d, and a robot 3. The worker 2b processes the object 2a manually from a position within 50cm of it. The robot 3 picks up the object 2a from table 2c and places it on table 2d at a designated location. The work environment 1 also contains other equipment, such as a door (not shown).
[0044] Figure 3 (b) The scene diagram is based on Figure 3 (a) The scene diagram is generated from the feature quantities calculated from the image of the work environment 1 shown. The acquisition unit 11 acquires the image and temperature information of the work environment 1. The thick line portion of the scene diagram shown represents the worker 2b performing manual work on the table 2c at a distance of less than 50cm from the object 2a.
[0045] The scenario diagram indicates that it is dangerous for robot 3 to move object 2a when operator 2b is performing the work, and there is a high probability that object 2a cannot be moved. In this case, the determination unit 13 of control device 10 makes a negative determination on the movement action performed by robot 3.
[0046] In addition to the above, the scene diagram also shows that in the working environment 1, when the temperature of object 2a is above 60°C and the distance between the door and object 2a is less than 50cm and they interfere with each other, object 2a cannot be moved due to the accompanying danger.
[0047] Furthermore, the scenario diagram illustrates the collaborative work of operator 2b and robot 3. If the distance between them is less than 20cm, an abnormal situation is handled, resulting in robot 3 being unable to move object 2a. In addition to the above, the scenario diagram also indicates the feasibility of robot 3 moving object 2a based on various conditions in the work environment 1.
[0048] <Example 2>
[0049] Figure 4 Indicates in Figure 3 (a) shows a scene where the package exists as an obstacle in the work environment 1, and the robot 3 places the object 2a onto the trolley and moves it. Furthermore, the acquisition unit 11 of the control device 10 only acquires an image of the work environment 1 and does not acquire temperature information.
[0050] The illustrated scenario shows that robot 3 can move object 2a using the trolley during startup preparation, with object 2a placed on it. "Startup preparation" in the diagram refers to the preparation for starting the machine when processing object 2a to produce a product. Furthermore, "Abnormal handling" refers to the handling of an error that is resolved if an alarm is triggered due to an abnormality in the machine.
[0051] Furthermore, the scenario diagram illustrates that worker 2b and robot 3 are working together. If the distance between them is less than 50cm, an abnormal situation is triggered, preventing the movement of object 2a. In addition to the above, the scenario diagram also shows the feasibility of moving object 2a by robot 3 based on various conditions in the work environment 1.
[0052] By using the illustrated scenario diagram, the control device 10 can determine the feasibility of the robot 3 moving the object 2a based on the possibility of contact, for example, when there are obstacles between the operator 2b and the robot 3. Therefore, there is no need to limit the barrier that should be set between the robot 3 and the operator 2b.
[0053] <Example 3>
[0054] Figure 5 The scene diagram shown represents a work environment 1, such as a kitchen or other place where food is prepared. The acquisition unit 11 acquires images and temperature information of the work environment 1. The robot 3's objective is to move the kettle, which is the object 2a.
[0055] The work environment 1 contains a kettle, a cutting board, a knife, vegetables, and a pot. The temperature information acquired by the acquisition unit 11 represents the temperature of the kettle. In the illustrated scene diagram, the types of objects 2 can be classified as vegetables and pots, and the status of the kettle and other items can be classified as being in use or processing.
[0056] The illustrated scenario diagrams, for example, show that if the kettle has a handle and its temperature is below 40°C, robot 3 can move the kettle. Furthermore, they show that if the kettle's temperature is above 80°C, robot 3 cannot move it because the kettle is being heated. In addition to these examples, the illustrated scenario diagrams also show the feasibility of robot 3 moving object 2a, i.e., the kettle, based on various conditions in the operating environment 1.
[0057] By using the illustrated scene diagram, the control device 10 can, for example, roughly determine the condition of the kettle, which is the object 2a, and select a safe and effective method to move the kettle, thereby controlling the action of the robot 3.
[0058] <Effects of the control device 10 according to this embodiment>
[0059] The control device 10 oversees the status of each object 2 in the work environment 1 and uses a scene diagram to determine whether the robot 3 can perform movement actions on the object 2. Therefore, even if changes occur in the work environment 1, the control device 10 can easily determine the surrounding conditions of the robot 3 without performing annotation work. Furthermore, the control device 10 can easily and accurately control the robot 3 based on the surrounding conditions of the robot 3.
[0060] [Second Implementation]
[0061] The control device 10 according to this embodiment is similar to the control device 10 according to the first embodiment, including an acquisition unit 11, an extraction unit 12, and a determination unit 13, and controls the movement of the robot 3. In addition, structural parts that are the same as those already described are marked with the same symbols, and repeated descriptions are omitted.
[0062] In this embodiment, unlike the first embodiment, the control device 10 determines the movement path of the robot 3 itself, rather than determining whether the movement action of the object 2 performed by the robot 3, which is the controlled object, is permissible. More specifically, the determination of the movement action of the robot 3 performed by the determination unit 13 of the control device 10 is a determination of control actions such as changing the movement path of the robot 3 within the working environment 1, decelerating, moving straight, and stopping.
[0063] Figure 6 This diagram illustrates an example of a scene according to the second embodiment of the present invention. The work environment 1 includes a worker 2b (object 2), a door, a trolley (obstacle), and various equipment. The acquisition unit 11 acquires an image of the work environment 1 and obtains distance information from a distance sensor.
[0064] The robot 3, as a mobile body, is configured, for example, as an autonomous mobile robot that moves autonomously. The robot 3 is, for example, an autonomous mobile robot (AMR) and an automated guided vehicle (AGV), but is not limited to these, and can also be various devices capable of moving within the working environment 1.
[0065] The illustrated scenario shows a worker 2b and a trolley acting as an obstacle on the path of robot 3. As long as there is free space to the left and right of these obstacles, the control device 10 determines whether to bend the path of robot 3. Furthermore, the control device 10 determines whether to slow down robot 3, for example, when worker 2b moves the trolley, when worker 2b is less than 2 meters from the door and the door may open, or when worker 2b is on the path.
[0066] Furthermore, in situations such as when operator 2b is not moving or when operator 2b is adjusting the equipment, the control device 10 determines whether the robot 3 should move straight. Additionally, if the distance between the robot 3 and operator 2b is less than 50cm, the control device 10 stops the robot 3 because there is no room for movement. The illustrated scenario diagram, in addition to these, also shows the feasibility of the robot 3's movement based on various conditions in the work environment 1.
[0067] <Effects of the control device 10 according to this embodiment>
[0068] The control device 10 can appropriately determine the path and speed to avoid obstacles based on the conditions of the working environment 1, and control the robot 3.
[0069] The present invention has been described above according to the above embodiments, but the present invention is not limited to the above embodiments. Various modifications, alterations and combinations that can be made by those skilled in the art within the scope of the claims of this application are naturally included within the scope of the present invention.
[0070] Symbol Explanation
[0071] 1-Working environment, 2-Object, 3-Robot, 10-Control device, 11-Acquisition unit, 12-Extraction unit, 13-Judgment unit.
Claims
1. A control device for controlling the actions of a robot, characterized in that, have: The acquisition unit acquires images of the work environment. An extraction unit that extracts multiple objects contained in the image; and The determination unit uses a scene graph representing the relationships between the objects in the work environment, determined based on feature quantities representing at least one of the extracted objects' positions and states, to determine whether a movement action performed by the robot is permissible.
2. The control device according to claim 1, characterized in that, The determination made by the determination unit regarding the movement action is a determination of whether the robot's action of picking up the object and placing it in a predetermined position is permissible.
3. The control device according to claim 1, characterized in that, The determination made by the determination unit regarding the movement action is a determination regarding the movement path of the robot itself.
4. The control device according to claim 1, characterized in that, The acquisition unit also acquires temperature information. The feature quantity includes information about the temperature of at least one of the working environment and the object.
5. A control method, executed by a control device for controlling the actions of a robot, characterized in that, include: Steps for acquiring images of the work environment; The step of extracting multiple objects contained in the image; and The step of determining the feasibility of a movement action performed by the robot is to use a scene graph representing the relationships between objects in the work environment, determined based on feature quantities representing at least one of the extracted objects' positions and states.