Robot system, self-traveling robot, and control method for self-traveling robot

The robot system addresses the issue of unsuitable image angles and ranges by using independently positioned cameras connected to self-propelled robots based on conditions, ensuring accurate and efficient robot movement control and reducing communication bandwidth.

JP2024168911APending Publication Date: 2024-12-05SEIKO EPSON CORP

Patent Information

Application Number
JP2023085958
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing robot systems struggle to obtain images with angles and imaging ranges suitable for controlling robot movements during work, leading to inefficiencies in operation and accuracy.

Method used

A robot system comprising self-propelled robots equipped with control units that communicably connect to independently positioned cameras, allowing for image acquisition and operation control based on images from these cameras, with the connection being established and disconnected based on predetermined conditions.

Benefits of technology

Enables accurate and efficient control of robot movements by ensuring images with suitable angles and ranges are used, reducing communication bandwidth and enabling parallel work by multiple robots with shared cameras.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024168911000001_ABST
    Figure 2024168911000001_ABST
Patent Text Reader

Abstract

To obtain an image at an angle and in an imaging range which are suitable for controlling an operation of a robot during working.SOLUTION: A robot system includes: a plurality of self-traveling robots each including a control part; and a first camera which is arranged independently from the plurality of self-traveling robots and can image a work space where the plurality of self-traveling robots perform work. The control part is communicably connected to the first camera when a prescribed starting condition is satisfied, makes the self-traveling robots perform work in the work space by controlling an operation of the self-traveling robot provided with the control part out of the plurality of self-traveling robots on the basis of an image acquired by the first camera, and thereby can disconnect the first camera when a prescribed finishing condition is satisfied.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a robot system, a self-propelled robot, and a control method for a self-propelled robot. [Background technology]

[0002] Conventionally, there is a technology relating to an imaging unit that can be selectively attached to each of a plurality of robots and acquires an image of an object to be inspected. In the technology of Patent Document 1, the imaging unit can be attached to and detached from the plurality of robots. The plurality of robots are used to inspect the object to be inspected. The same imaging unit is attached to the plurality of robots in turn, and images are acquired using the imaging unit. Each image inspection unit connected to each robot uses for inspection an image that has been acquired by the same imaging unit and has been subjected to processing such as aberration correction based on the same optical characteristic data. Therefore, images of a certain quality can be used for inspection by all image inspection units, and the accuracy of the inspection can be stabilized.

[0003] On the other hand, there is technology that controls the robot's operation based on images acquired by a camera when the robot performs some kind of task on an object, rather than inspecting the object. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-104093 A Summary of the Invention [Problem to be solved by the invention]

[0005] The purpose of the operation of the multiple robots in the technology of Patent Document 1 is to obtain an image of the object to be inspected for inspection. Therefore, when the technology of Patent Document 1 is used to control the operation of the robot based on an image obtained by a camera, it is not necessarily the case that an image with an angle and imaging range suitable for controlling the operation of the robot during work is obtained. [Means for solving the problem]

[0006] According to one embodiment of the present disclosure, a robot system is provided. The robot system includes a plurality of self-propelled robots each having a control unit, and a first camera that is arranged independently of the plurality of self-propelled robots and can capture an image of a workspace in which the plurality of self-propelled robots perform work. The control unit is configured to be communicatively connected to the first camera when a predetermined start condition is satisfied, to control the operation of a self-propelled robot that is equipped with the control unit among the plurality of self-propelled robots based on an image acquired by the first camera, thereby causing the self-propelled robot to perform the work in the workspace, and to disconnect the connection with the first camera when a predetermined end condition is satisfied. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram of a robot system 1 according to a first embodiment of the present disclosure. [Diagram 2] 2 is a block diagram showing the functional parts of a control unit 100 of the self-propelled robot AMR. FIG. [Diagram 3] 1 is an explanatory diagram showing a configuration for realizing mutual communication between a self-propelled robot AMR, a camera CM, and a server SV. [Figure 4] 11 is a flowchart showing the process performed by the self-propelled robot AMR when performing work. [Diagram 5] 13 is a flowchart showing a process when the self-propelled robot AMR performs work in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] A. First embodiment: 1 is an explanatory diagram of a robot system 1 according to a first embodiment of the present disclosure. The robot system 1 is a system that causes a plurality of robots to perform work on a workpiece WP in a workspace OS. The robot system 1 includes self-propelled robots AMR1 and AMR2, a first camera CM1, a second camera CM2, and a server SV.

[0009] The self-propelled robots AMR1 and AMR2 autonomously move within the factory where the robot system 1 is installed based on work instructions received from the server SV, and perform work on a workpiece WP in the workspace OS (see the lower left and upper right parts of FIG. 1). In this specification, the term "self-propelled robot" refers to a robot that can move while avoiding obstacles without following instructions from outside. Hereinafter, when referring to the self-propelled robots AMR1 and AMR2 without distinguishing between them, they will be referred to as "self-propelled robot AMR."

[0010] The self-propelled robot AMR can acquire images from a first camera CM1 and a second camera CM2. The self-propelled robot AMR performs work on a workpiece WP based on the images acquired by the first camera CM1 and the second camera CM2. Each of the self-propelled robots AMR includes a control unit 100, a robot arm 200, a hand 300, and a vehicle 400.

[0011] The robot arm 200 has six joints. Three of the six joints are bending joints. The other three of the six joints are torsion joints. Each joint is provided with a servo motor and a position sensor. The servo motor generates a rotation output for driving each joint. The position sensor detects the angular position of the output shaft of the servo motor. The robot arm 200 is controlled by the control unit 100 to drive the servo motor of each joint, thereby allowing the hand 300 connected to the tip of the robot arm 200 to be placed at a target position in a three-dimensional space in a target posture.

[0012] The hand 300 is attached to the tip of the robot arm 200. The hand 300 is a so-called end effector. The hand 300 is controlled by the control unit 100 to grasp a workpiece WP, which is the object of the work, and is also capable of releasing the grasped workpiece. The hand 300 and the robot arm 200 are controlled by the control unit 100 to grasp a workpiece and move it to a target position in three-dimensional space.

[0013] The vehicle 400 supports the robot arm 200. The vehicle 400 is controlled by the control unit 100 and can move the self-propelled robot AMR to any position on the floor surface.

[0014] The control unit 100 controls the robot arm 200, the hand 300, and the vehicle 400. The configuration and functions of the control unit 100 will be described later.

[0015] The first camera CM1 is disposed independently of the self-propelled robot AMR (see the middle left part of FIG. 1). Note that the camera being "disposed independently of the robot" means that the camera is not fixed to the robot, but is fixed at a fixed position, and does not move with the robot. The first camera CM1 can capture an image of the workspace OS in which the self-propelled robot AMR performs work. The first camera CM1 is connected to the self-propelled robot AMR, and can transmit image data obtained by capturing the image to the self-propelled robot AMR. The first camera CM1 is a monocular digital video camera. The first camera CM1 is compatible with ROS2 (Robot Operating System 2).

[0016] The first camera CM1 can be connected to the self-propelled robot AMR without going through the server SV. Therefore, there is less time delay due to communication compared to a mode in which the first camera CM1 and the self-propelled robot AMR communicate via the server SV. As a result, for example, even when image feedback control is performed on the self-propelled robot AMR based on an image obtained by the first camera CM1, accurate control can be performed without reducing the operating speed of the self-propelled robot AMR.

[0017] The second camera CM2 is arranged independently of the self-propelled robot AMR (see the lower right part of Figure 1). The second camera CM2 can capture an image of the workspace OS in which the self-propelled robot AMR performs work. The hardware configuration of the second camera CM2 is the same as the hardware configuration of the first camera CM1. However, the angle and range at which the second camera CM2 captures the workspace OS are different from the angle and range at which the first camera CM1 captures the workspace OS. The range at which the first camera CM1 captures the workspace OS is shown in Figure 1 as the first imaging range IR1 (see the middle left part of Figure 1). The range at which the second camera CM2 captures the workspace OS is shown in Figure 1 as the second imaging range IR2 (see the lower right part of Figure 1).

[0018] The first imaging range IR1 includes the entire workspace OS. Meanwhile, the second imaging range IR2 includes the workpiece WP arranged in the workspace OS, but does not include a part of the workspace OS. The second camera CM2 captures a larger image of the vicinity of the workpiece WP. Therefore, the second camera CM2 can capture images of more minute configurations than the first camera CM1. The self-propelled robot AMR can perform more minute tasks based on the images received from the second camera CM2.

[0019] Other than that, the second camera CM2 is the same as the first camera CM1. In the following, when referring to the cameras without distinguishing between the first camera CM1 and the second camera CM2, they will be referred to as the camera CM.

[0020] The server SV is communicatively connected to the self-propelled robots AMR1, AMR2, the first camera CM1, and the second camera CM2. More specifically, the server SV is connected to the self-propelled robots AMR1, AMR2, the first camera CM1, and the second camera CM2 via a wireless LAN (Local Area Network). The server SV can control the self-propelled robots AMR1, AMR2, the first camera CM1, and the second camera CM2.

[0021] The server SV is equipped with a processor, a central processing unit (CPU), a random access memory (RAM), and a read-only memory (ROM). The RAM includes a main memory, which is a semiconductor memory, and a hard disk, which is an auxiliary storage device. The hard disk stores an operating system, a program for controlling the self-propelled robot AMR, a program for controlling the camera CM, and the like. The CPU loads the computer programs stored on the hard disk into the main memory and executes them to realize various functions, such as sending work instructions to the self-propelled robot AMR.

[0022] 2 is a block diagram showing the functional parts of the control unit 100 of the self-propelled robot AMR. The control unit 100 includes a CPU 110, which is a processor, a memory 120, an arm control interface 140, a hand control interface 150, and a camera control interface 170.

[0023] The memory 120 includes a RAM and a ROM. The RAM includes a main memory, which is a semiconductor memory, and a hard disk, which is an auxiliary storage device. The hard disk stores an operating system, middleware, a robot control program, and the like. More specifically, the operating system is Linux (registered trademark). More specifically, the middleware is ROS2 (Robot Operating System 2). The CPU 110 loads the computer programs stored in the hard disk into the main memory and executes them to realize real-time control of the self-propelled robot AMR.

[0024] The CPU 110 controls the robot arm 200 via the arm control interface 140. The functional unit of the CPU 110 that controls the robot arm 200 is shown as an arm control unit 112 in FIG. 2. The CPU 110 controls the hand 300 via the hand control interface 150. The functional unit of the CPU 110 that controls the hand 300 is shown as a hand control unit 113 in FIG. 2. The CPU 110 communicates with the first camera CM1 and the second camera CM2 via the camera control interface 170. The functional unit of the CPU 110 that communicates with the first camera CM1 and the second camera CM2 is shown as a camera control unit 115 in FIG. 2.

[0025] Based on the image data received from the first camera CM1 and the second camera CM2, the CPU 110 recognizes the position and orientation of an object contained in the image data. Based on the recognized object, the CPU 110 can recognize the current position of the self-propelled robot AMR. The functional unit of the CPU 110 that recognizes the position and orientation of an object is shown in FIG. 2 as an object recognition unit 116.

[0026] CPU 110 determines the movement path of self-propelled robot AMR on the floor surface in the factory based on information on the position of self-propelled robot AMR in the factory, the position of workspace OS in the factory, and the positions and volumes of obstacles that impede the movement of self-propelled robot AMR. Information on the position of workspace OS in the factory and the positions and volumes of obstacles that impede the movement of self-propelled robot AMR is received from server SV (see FIG. 1). The movement path of self-propelled robot AMR on the floor surface in the factory is determined so as not to collide with obstacles. The functional part of CPU 110 that determines the movement path of self-propelled robot AMR on the floor surface is shown in FIG. 2 as trajectory planning part 111.

[0027] The CPU 110 determines the movement path of the hand 300 based on information on the angles of each joint of the robot arm 200, the position of the workpiece WP in the workspace OS, and the positions and volumes of obstacles that impede the movement of the robot arm 200 and the hand 300. The movement path of the hand 300 is determined so that the hand 300 does not collide with the obstacles or the self-propelled robot AMR itself. The functional part of the CPU 110 that determines the movement path of the hand 300 is shown as a path planning part 114 in FIG. 2.

[0028] Fig. 3 is an explanatory diagram showing a configuration for realizing mutual communication between the self-propelled robot AMR, the camera CM, and the server SV. The camera CM has an imaging system and a communication layer (see the right part of Fig. 3). The imaging system is hardware such as an optical system and a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The communication layer has a wireless LAN layer, a UDP (User Datagram Protocol) layer, and a DDS (Data Distribution Service) layer. With this configuration, the camera CM realizes publish / subscribe type communication. Specifically, with the above configuration, the camera CM repeatedly distributes captured images to the self-propelled robot AMR and the server SV at regular intervals.

[0029] The server SV has a camera control unit (see the center of FIG. 3). The camera control unit of the server SV has a wireless LAN layer, a UDP layer, and a DDS layer, just like the camera CM. The camera control unit 115 of the self-propelled robot AMR also has a wireless LAN layer, a UDP layer, and a DDS layer. With this configuration, publish / subscribe type communication is realized between the server SV and the self-propelled robot AMR. Specifically, the server SV and the self-propelled robot AMR subscribe to images captured by the camera CM from the camera CM. As a result, images captured by the camera CM are repeatedly transmitted to the self-propelled robot AMR and the server SV.

[0030] Fig. 4 is a flowchart showing the processing when the self-propelled robot AMR performs work. The processing in Fig. 4 is executed by the control unit 100 of the self-propelled robot AMR (see the lower left and middle right parts of Fig. 1). The following description will be given taking as an example a case where the self-propelled robot AMR1, out of the self-propelled robots AMR1 and AMR2, performs work.

[0031] In step S100, the control unit 100 determines whether or not a work start instruction has been received from the server SV. The work start instruction includes (i) information that uniquely represents the work to be started, (ii) information that represents one or more work contents included in the work to be started, and (iii) information that associates at least one of the first camera CM1 and the second camera CM2 with the work to be started. The control unit 100 waits until the work start instruction is received from the server SV. If the work start instruction is received from the server SV, the process proceeds to step S200. That is, the following process is performed on the condition that the self-propelled robot AMR1 equipped with the control unit 100 starts a task to be executed in the workspace OS.

[0032] In step S200, the control unit 100 selects a camera CM to be used for the task to be performed, i.e., one or more cameras CM communicably connected to the control unit 100. The camera CM is selected based on information that associates one or both of the first camera CM1 and the second camera CM2 with the task, which information is included in the instruction to start the task received from the server SV.

[0033] By performing such processing, it is possible to obtain images from a camera that can provide images with an angle and imaging range more suitable for controlling the operation of the self-propelled robot AMR1 performing work, and control the operation of the self-propelled robot AMR1 during work. In the following, the explanation will be given assuming that the first camera CM1 is selected.

[0034] In step S300, the control unit 100 establishes a communicative connection with the selected camera CM via the camera control unit 115 (see the middle left part of FIG. 2). As a result, subscription by the self-propelled robot AMR to images captured by the camera CM is started. This process is also called a "subscription start process." The connection between the self-propelled robot AMR1 and the first camera CM1 is shown as CN1 in FIG. 1. The connection between the self-propelled robot AMR1 and the second camera CM2 is shown as CN2 in FIG. 1. Note that in the process of the robot system 1, connections that are connected when a predetermined condition is satisfied and that are disconnected when a predetermined condition is satisfied are shown by dashed lines in FIG. 1.

[0035] In this embodiment, the control unit 100 is communicatively connected to the camera CM on the condition that work is started. Therefore, compared to a mode in which the control unit 100 is communicatively connected to the camera before receiving an instruction to start work, it is possible to reduce the bandwidth of the communications CM1, CM2 between the control units 100 of the self-propelled robots AMR1, AMR2 and the cameras.

[0036] In steps S400 to S600, the control unit 100 controls the operation of the self-propelled robot AMR1 in which the control unit 100 itself is equipped based on images acquired by the connected camera CM, thereby causing the self-propelled robot AMR1 to perform tasks in the workspace OS.

[0037] In step S400, the control unit 100 receives an image captured by the connected camera CM from the camera CM.

[0038] In step S500, the object recognition unit 116 of the control unit 100 recognizes a target position based on an image received from the connected camera CM. For example, the target position where the workpiece WP is placed in the workspace OS is recognized.

[0039] The path planning unit 114 of the control unit 100 determines the movement path of the hand 300 based on information on the angles of each joint of the robot arm 200, the position of the workpiece WP in the workspace OS, and the positions and volumes of obstacles that impede the movement of the robot arm 200 and the hand 300 (see the middle left part of Figure 2).

[0040] In step S600, the control unit 100 executes one of the work commands included in the job. For example, the control unit 100 moves the hand 300 to the next relay point on the movement path of the hand 300, causes the hand 300 to grasp a workpiece, or causes the hand 300 to release the workpiece, in accordance with the one work command.

[0041] Specifically, the arm control unit 112 of the control unit 100 controls the robot arm 200 via the arm control interface 140 (see the upper left part of FIG. 2). The hand control unit 113 of the control unit 100 controls the hand 300 via the hand control interface 150 (see the middle left part of FIG. 2).

[0042] In step S700, the control unit 100 determines whether or not there are any remaining work instructions among the work instructions included in the work. If there are any remaining work instructions, the process returns to step S600, and the next work instruction is executed. If there are no remaining work instructions, the process proceeds to step S800.

[0043] In step S800, the control unit 100 disconnects the connection with the camera CM. That is, the connection between the control unit 100 of the self-propelled robot AMR1 and the camera CM is disconnected on the condition that the work performed by the self-propelled robot AMR1, in which the control unit 100 is provided, in the workspace OS is completed. As a result, the subscription by the self-propelled robot AMR to the images captured by the camera CM is terminated. After that, no images are transmitted from the camera CM to the self-propelled robot AMR1. After that, the processing in FIG. 4 ends.

[0044] By performing such processing, the bandwidth of communications CN1, CN2 between each control unit 100 of the self-propelled robots AMR1, AMR2 and the camera CM can be reduced compared to a configuration in which the control unit 100 is communicatively connected to the camera CM even after the work is completed.

[0045] The above describes the processing in the robot system 1, taking as an example a case where the self-propelled robot AMR1 performs a task using an image from the first camera CM1. However, the processing is similar when the self-propelled robot AMR1 performs a task using an image from the second camera CM2. The same processing is also performed in the self-propelled robot AMR2.

[0046] In this embodiment, the orientation and imaging range of the camera arranged independently of the self-propelled robots AMR1 and AMR2 are set in advance so that an image suitable for use in controlling the operation of the robots working in the workspace OS can be acquired, thereby obtaining the following effects (see IR1 and IR2 in FIG. 1). That is, while using the same camera CM for the self-propelled robots AMR1 and AMR2, the operation of each of the self-propelled robots AMR1 and AMR2 can be controlled based on the image acquired by the camera CM without the imaging angle and imaging range being limited by the position and angle of the hand 300 of the self-propelled robots AMR1 and AMR2 working. Therefore, the operation of the self-propelled robot AMR1 working can be appropriately controlled based on the image with the angle and imaging range suitable for controlling the operation of the self-propelled robot AMR1 working.

[0047] Furthermore, since the same camera CM can be used by the self-propelled robots AMR1 and AMR2, work can be performed by the self-propelled robots AMR1 and AMR2 with a smaller number of cameras than in a configuration in which two cameras are provided for each of the self-propelled robots AMR1 and AMR2 in positions where the shooting range is not obstructed by the hand 300 or the robot arm 200 during work.

[0048] Furthermore, since the same camera CM can be used in the self-propelled robots AMR1 and AMR2, common values ​​can be used as operation parameters such as image recognition parameters based on images from the camera CM, feedback control parameters based on images from the camera CM, and machine learning parameters for work in the control unit 100 of the self-propelled robots AMR1 and AMR2. Therefore, it is not necessary to set those parameters for each of the self-propelled robots AMR1 and AMR2. In addition, a common algorithm can be used between the self-propelled robots AMR1 and AMR2 as an algorithm that uses images from the camera CM. Even when the self-propelled robots AMR1 and AMR2 are replaced with other self-propelled robots due to maintenance or a breakdown, the parameters and algorithms that were used up until that point can be applied to the new robot. As a result, the start-up time after maintenance or repair can be shortened.

[0049] In this embodiment, the condition for which the success or failure is determined in step S100, i.e., the receipt of an instruction to start work from the server SV, is also referred to as the "start condition." In this embodiment, the condition for which the success or failure is determined in step S700, i.e., that there are no remaining work instructions among the work instructions included in the work, is also referred to as the "end condition."

[0050] B. Second embodiment: In the robot system of the second embodiment, the configurations and functions of the first camera CM1 and the second camera CM2, and part of the processing of the control unit 100 of the self-propelled robots AMR1 and AMR2 during work are different from those of the first embodiment. Other points of the robot system of the second embodiment are the same as the robot system 1 of the first embodiment.

[0051] In the second embodiment, the first camera CM1 is configured to be able to change the shooting angle and shooting range. That is, the first camera CM1 has the functions of panning, tilting, and zooming. The first camera CM1 is controlled by a control unit 100 having the control right to control the first camera CM1 among the control units 100 of the self-propelled robots AMR1 and AMR2, thereby changing the shooting angle and shooting range. The "control right" of a certain camera CM is the right to change at least one of the shooting angle and shooting range of the camera CM by instruction. In this embodiment, the control unit 100 of the self-propelled robot AMR having the control right of the first camera CM1 can change both the shooting angle and shooting range of the first camera CM1 by instruction. The control right of the first camera CM1 cannot be held by multiple control units 100 at the same time. The first camera CM1 is configured to be controlled by only one control unit 100 having the control right among the control units 100 of the self-propelled robots AMR1 and AMR2.

[0052] On the other hand, while the first camera CM1 is being controlled by one of the control units 100 of the self-propelled robots AMR1, AMR2, it can provide images to the other control unit 100 of the self-propelled robots AMR1, AMR2 (see FIG. 3). That is, images are also distributed to the control units 100 of self-propelled robots AMR other than the control unit 100 of the self-propelled robot AMR that has the control right, and to the server SV (see FIG. 3).

[0053] With this configuration, while a self-propelled robot AMR1 is performing work based on an image from a first camera CM1, another self-propelled robot AMR2 can perform work based on an image from the same first camera CM1.

[0054] In the second embodiment, the second camera CM2 has the same configuration and functions as the first camera CM1.

[0055] FIG. 5 is a flowchart showing the processing when the self-propelled robot AMR performs work in the second embodiment. The processing in FIG. 5 is executed by the control unit 100 of the self-propelled robot AMR. In the processing in FIG. 5, steps S300B and S800B are executed instead of steps S300 and S800 in the processing in FIG. 4, respectively. In the processing in FIG. 5, steps S250, 300C, S350, S400C, S450, and S470 are further executed. Other points in the processing in FIG. 5 are the same as those in FIG. 4. The following describes an example in which the self-propelled robot AMR1 performs work, out of the self-propelled robots AMR1 and AMR2.

[0056] In the process of Fig. 5, the process of step S250 is performed after step S200. In step S250, the control unit 100 judges whether or not another self-propelled robot AMR has the control right of the camera CM to be used in the work to be performed. If another self-propelled robot AMR does not have the control right of the camera CM to be used in the work to be performed, the process proceeds to step S300B. If another self-propelled robot AMR has the control right of the camera CM to be used in the work to be performed, the process proceeds to step S300C.

[0057] In step S300B, the control unit 100 connects to the selected camera CM via the camera control unit 115 so as to be able to communicate with it. This process is the same as the process in step S300 in FIG. 4. Prior to the execution of the task, the control unit 100 further acquires control of the camera CM to be used in the task to be executed. Here, it is assumed that the control of the first camera CM1 has been acquired by the control unit 100 of the self-propelled robot AMR1. As a result, the shooting angle and shooting range of the first camera CM1 cannot be controlled by the control unit 100 of the self-propelled robot AMR2, but are controlled only by the control unit 100 of the self-propelled robot AMR1.

[0058] With this configuration, the control unit 100 can control the movement of the self-propelled robot AMR1 performing work based on images from an angle and imaging range that is more suitable for controlling the movement of the self-propelled robot AMR1 during work.

[0059] In the second embodiment, the camera CM is exclusively controlled by only one control unit 100. Therefore, when the self-propelled robots AMR1 and AMR2 are working in parallel, the self-propelled robot AMR2 can obtain images required for the other self-propelled robot AMR2 without impeding the self-propelled robot AMR1 from obtaining images required for control during the work.

[0060] The processes of steps S400, S500, S600, and S700 in Fig. 5 are the same as the processes of steps S400, S500, S600, and S700 in Fig. 4. If there are no remaining work orders in step S700, the process proceeds to step S800B.

[0061] In step S800B, the control unit 100 disconnects the connection with the camera CM. This process is the same as the process in step S800 in Fig. 4. The control unit 100 further relinquishes the control right of the camera CM that it had held. After that, the process in Fig. 5 ends.

[0062] In this embodiment, the control right is relinquished after the task is performed (see S700, S800B in FIG. 5). Therefore, the camera CM can be selectively controlled by the control unit 100 of the self-propelled robots AMR1, AMR2.

[0063] On the other hand, in step S300C, the control unit 100 establishes a communicable connection with the selected camera CM via the camera control unit 115. This process is the same as the process of step S300 in Fig. 4. However, unlike the process of step S300B, in step S300C, the control unit 100 does not obtain control of the camera CM to be used in the work to be performed (see S300B in Fig. 5).

[0064] In step S400C, the control unit 100 receives an image captured by the connected camera CM from the camera CM. This process is the same as the process of step S400 in Fig. 4. However, in the second embodiment, the reception of an image from the camera CM in step S400 by the self-propelled robot AMR having the control right of the camera CM and the reception of an image from the camera CM in step S400C by the self-propelled robot AMR not having the control right of the camera CM can be executed simultaneously.

[0065] In step S450, the object recognition unit 116 of the control unit 100 determines whether or not the target position can be recognized based on the image received from the connected camera CM. For example, it is determined whether or not the target position where the workpiece WP is to be placed in the workspace OS can be recognized. If the target position cannot be recognized based on the image received from the connected camera CM, the process proceeds to step S470. If the target position can be recognized based on the image received from the connected camera CM, the process proceeds to step S500. Thereafter, the robot performs the work based on the image obtained from the camera CM without having the control right of the camera CM used for the work.

[0066] By performing this type of processing, the self-propelled robot AMR that was unable to obtain control can be effectively utilized to proceed with the work.

[0067] In step S470, the control unit 100 waits for a predetermined time. After that, the process returns to step S250. By adopting such an embodiment, the self-propelled robots AMR1 and AMR2 can take turns acquiring control of the camera to be used for work when necessary, and perform work based on a preferred image.

[0068] C. Other embodiments: C1. Alternative embodiment 1: (1) In the first embodiment, the camera CM is a monocular digital video camera. However, the camera may be a stereo camera or an RGBD camera that can obtain depth information in addition to an RGB image.

[0069] (2) In the first embodiment, the CPU 110 functions as a trajectory planning unit 111, a path planning unit 114, and an object recognition unit 116 (see the left part of FIG. 2). However, the control unit 100 can also include a GPU (Graphics Processing Unit) and cause the GPU to realize some or all of the functions of the trajectory planning unit 111, the path planning unit 114, and the object recognition unit 116.

[0070] (3) In the first embodiment, the workspace OS is constant (see the center of FIG. 1). However, the workspace OS may be determined for each task. The workspace OS may also be determined for each combination of a task and a robot performing that task.

[0071] (4) In the first embodiment, the camera CM, the self-propelled robot AMR, and the server SV perform publish / subscribe type communication using the ROS2 communication framework, i.e., DDS (see FIG. 3). However, the camera CM, the self-propelled robot AMR, and the server SV may transmit and receive image data using other methods, such as the ROS (Robot Operating System) communication framework.

[0072] (5) In the first embodiment, the camera CM included in the robot system publishes to the self-propelled robot AMR and the server SV (see FIG. 3). However, it is also possible to have an aspect in which publish / subscribe type communication does not take place between the camera CM and the self-propelled robot AMR and the server SV. In such an aspect, an instruction to capture an image is repeatedly sent from the self-propelled robot AMR and / or the server SV, and the camera CM repeatedly captures and transmits the image in response to the instruction.

[0073] (6) In the first embodiment, the camera CM is connected to the self-propelled robot AMR and the server SV via wireless LAN (see FIG. 3). However, the camera CM and the self-propelled robot AMR may be connected via a communication method based on other communication standards, such as Bluetooth (trademark).

[0074] (7) In the first embodiment, the control unit 100 selects the camera CM based on information that associates one or both of the first camera CM1 and the second camera CM2 with the task, which information is included in the instruction to start the task received from the server SV (see S200 in FIG. 2). However, the selection of the camera by the control unit may be performed in other ways.

[0075] For example, the information associating the camera CM with the task may be transmitted from the server SV to the self-propelled robot AMR separately from an instruction to start the task. Also, the information associating the camera CM with the task may be stored in advance in the memory 120 of the self-propelled robot AMR. The information identifying the camera may be an ID assigned to the camera or an IP address assigned to the camera. When the information identifying the camera is an IP address, the control unit 100 can connect to that IP address according to the task and transmit an image.

[0076] Also, for example, the data issued by the camera CM may include information representing a task, and the control unit 100 of the self-propelled robot AMR may perform the task by utilizing an image of data transmitted by multiple cameras CM that includes information representing the task to be performed by the self-propelled robot AMR.

[0077] Furthermore, when a certain task is performed using images from multiple cameras, the following configuration is also possible. That is, the data issued by the camera CM includes information representing the camera CM that captured the image and information representing the task to be performed based on the image. The control unit 100 of the self-propelled robot AMR executes the task command using images from multiple pieces of data, among the data transmitted from the multiple cameras CM, that include information representing the task to be performed by the self-propelled robot AMR.

[0078] The following embodiment may also be used: The data issued by the camera CM includes information representing a work command to be executed based on the image. The control unit 100 of the self-propelled robot AMR executes the work command using an image of the data including information representing the work command to be executed by the self-propelled robot AMR. In this case, the camera CM that transmits the data used to execute the work command may change for each work command.

[0079] Also, for example, a robot camera may be provided on each self-propelled robot, and an external camera within the field of view of the robot camera may be selected as the camera to be used for work. If the condition for connection with an external camera is that the external camera is within the field of view of the robot camera, it is preferable, for example, to attach a marker to each external camera to distinguish them. In such an embodiment, the self-propelled robot can be connected to the external camera based on the marker present in the image captured by the robot camera.

[0080] (8) In the first embodiment, the movement path of the hand 300 is determined based on the image from the camera CM (see S500 in FIG. 4 and 114 in FIG. 2). Then, the robot arm 200 and the hand 300 are controlled along the determined path (see S600 in FIG. 4). However, in controlling the robot arm 200 and the hand 300, the control unit 100 may perform image feedback control of the robot arm 200 and the hand 300 in real time based on the image received from the connected camera CM. For example, the control unit 100 may perform image feedback control so that the distance between a specific portion of the workpiece WP and the target position becomes zero. Furthermore, the control unit 100 may determine the end of the work when it recognizes that the distance between a specific portion of the workpiece WP and the target position becomes zero.

[0081] When multiple robots are included within the field of view of the camera and image feedback control is performed for each of one or more specific robots, it is preferable to, for example, attach two-dimensional codes or markers to the multiple robots in order to distinguish each robot. In such an embodiment, image feedback control of each robot can be performed based on the two-dimensional codes or markers present in the image captured by the camera.

[0082] (9) In the second embodiment, when the control unit 100 of the self-propelled robot AMR recognizes that another robot has the control right of the camera CM used in the work after starting the work, the control unit 100 executes the work or waits while the other robot has the control right of the camera CM (see S450, S470, and S500 in FIG. 5). However, for example, when the robot having the control right of the camera CM completes one work content, it is possible to determine which of the multiple robots performing the work using the camera should acquire the control right, and the determined robot acquires the control right.

[0083] (10) In the second embodiment described above, the first camera CM1 changes its shooting angle and shooting range by being controlled by the control unit 100 having the control authority to control the first camera CM1, among the control units 100 of the self-propelled robots AMR1 and AMR2. However, the control authority of the camera may be held by a server, not by the robot. In such an embodiment, the camera changes its shooting angle and shooting range by being controlled by the server. The server can set the camera to a shooting angle and shooting range suitable for use by the robot performing the work, and cause it to acquire images and transmit them to the robot performing the work.

[0084] (11) In the above first embodiment, an example has been described in which the self-propelled robot AMR1 works alone on a workpiece WP in the workspace OS (see AMR1 in FIG. 1). However, the technology disclosed herein can also be applied to cases in which two or more robots work in cooperation. In such cases, appropriate cooperative control can be achieved by performing image feedback control on each robot based on images sent from the same camera (see AMR1, AMR2, and IR1 in FIG. 1).

[0085] (12) In the first embodiment, after a camera is selected, the self-propelled robot AMR starts work, and the images from the selected camera are used until the work is completed (see S200, S400 to S700 in FIG. 4). However, the camera that delivers the images used in the work can be switched during the work.

[0086] (13) In the first or second embodiment, the server SV may store images received from the camera CM (see the upper center part of FIG. 1). The data stored in this way can be used to record the progress of work, analyze work, and learn robot control.

[0087] (14) In the first or second embodiment, another robot that is to perform work on the workpiece WP after the work performed by the self-propelled robots AMR1 and AMR2 is completed may receive images from the camera CM. In such an embodiment, the robot performing the downstream process can check the progress of the work performed by the self-propelled robots AMR1 and AMR2, estimate the waiting time, and reschedule the work.

[0088] (15) In the first embodiment described above, the robot system 1 includes self-propelled robots AMR1 and AMR2 (see FIG. 1). However, the robot system may include three or more self-propelled robots. The technology disclosed herein may also be applied to a robot system including multiple non-moving robots. Multiple non-moving robots can take turns using a common camera to perform tasks at different times. Multiple non-moving robots can also simultaneously receive images from a common camera and perform their own tasks based on the images.

[0089] C2. Alternative embodiment 2: In the first embodiment described above, the control unit 100 connects to the camera CM so that it can communicate with the camera CM when a task is started. More specifically, the control unit 100 connects to the camera CM so that it can communicate with the camera CM when it receives an instruction to start the task from the server SV (see S100 and S400 in FIG. 4). However, the condition for the control unit of the self-propelled robot to connect to the camera CM used for the task so that it can communicate with the camera CM can also be other conditions, such as the self-propelled robot or a specific part of the self-propelled robot coming within the field of view of the camera.

[0090] If the condition for connection with the camera is that the self-propelled robot or a specific part of the self-propelled robot is within the field of view of the camera, it is preferable, for example, to attach a two-dimensional code or marker to each self-propelled robot in order to distinguish each self-propelled robot. In such an embodiment, the self-propelled robot can be connected to the camera based on the two-dimensional code or marker present in the image captured by the camera.

[0091] The robot may use one camera for a task, or may use multiple cameras for a task. When only one camera is used for a task, the start condition for the control unit to connect to the camera used for the task so that it can communicate is different from the start condition for the control unit to connect to the other cameras so that it can communicate. On the other hand, when multiple cameras are used for a task, the start condition for the control unit to connect to the multiple cameras used for a task so that it can communicate can be a common condition for the cameras used for the task.

[0092] C3. Alternative embodiment 3: In the first embodiment described above, the connection between the control unit 100 and the camera CM is cut off when the work is completed. More specifically, the control unit 100 cuts off the connection to the camera CM when there are no remaining work instructions among the work instructions included in the work (see S700, S800 in FIG. 4). However, the condition for the control unit of the self-propelled robot to cut off the connection to the camera used for the work can also be other conditions, such as the self-propelled robot or a specific part of the self-propelled robot being no longer visible from the camera's field of view.

[0093] When only one camera is used for a task, the end condition for the control unit to disconnect communication with the camera being used is different from the start condition for the control unit to disconnect communication with the other cameras, whereas when multiple cameras are used for a task, the end condition for the control unit to disconnect communication with the multiple cameras being used for a task can be common to the cameras being used for the task.

[0094] In the case where the condition for disconnecting the connection with the camera is that the self-propelled robot or a specific part of the self-propelled robot is no longer within the field of view of the camera, it is preferable to attach a two-dimensional code or marker to each self-propelled robot to distinguish each self-propelled robot. In such an embodiment, the connection and disconnection between the self-propelled robot and the camera can be performed based on the presence or absence of a two-dimensional code or marker in an image captured by the camera. Even in an embodiment in which such a two-dimensional code or marker is not provided on the robot, the robot within the field of view of the camera can be identified by comparing the work command transmitted to the robot arm 200 and / or hand 300 of each robot with the movement of the robot arm 200 and / or hand 300 within the field of view of the camera.

[0095] C4. Alternative embodiment 4: In the second embodiment, the first camera CM1 and the second camera CM2 are configured to be able to change the shooting angle and shooting range (see S350 in FIG. 5). However, as shown in the first embodiment, the first camera CM1 and the second camera CM2 may be configured to have a fixed shooting angle and shooting range. Also, some of the multiple cameras provided in the robot system may be configured to be able to change the shooting angle and shooting range, and the other may be configured to have a fixed shooting angle and shooting range.

[0096] C5. Alternative Embodiment 5: In the second embodiment, the camera CM is configured to be controlled by only one of the control units 100 of the self-propelled robots AMR1, AMR2 that has the control authority (see S250 in FIG. 5). However, the camera CM is configured to be able to change the shooting angle and shooting range, and can also be configured so that the shooting angle and shooting range are controlled by any robot included in the robot system.

[0097] C6. Alternative Embodiment 6: In the second embodiment, the first camera CM1, while being controlled by one of the control units 100 of the self-propelled robots AMR1, AMR2, can provide images to the other control unit 100 of the self-propelled robots AMR1, AMR2 (see FIG. 3). However, the camera can also be configured to provide images only to an entity that has control of that camera.

[0098] C7. Alternative Embodiment 7: In the first embodiment, the configuration and function of the second camera CM2 are the same as those of the first camera CM1, except for its installation position and shooting range (see the middle left and lower right parts of FIG. 1). However, the second camera CM2 may have a configuration different from that of the first camera CM1. For example, a camera whose shooting range includes the entire working space may have a fixed shooting angle and shooting range, and a camera whose shooting range includes a range that includes the workpiece but does not include part of the working space may be configured to have a changeable shooting angle and shooting range.

[0099] D. Other forms: The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various forms without departing from the spirit of the present disclosure. For example, the present disclosure can be realized in the following forms. The technical features in the above-mentioned embodiments corresponding to the technical features in each form described below can be appropriately replaced or combined in order to solve some or all of the problems of the present disclosure, or to achieve some or all of the effects of the present disclosure. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted.

[0100] (1) According to one embodiment of the present disclosure, a robot system is provided. The robot system includes a plurality of self-propelled robots each having a control unit, and a first camera that is arranged independently of the plurality of self-propelled robots and can capture an image of a workspace in which the plurality of self-propelled robots perform work. The control unit is configured to be able to communicatively connect to the first camera when a predetermined start condition is satisfied, control the operation of a self-propelled robot that is equipped with the control unit among the plurality of self-propelled robots based on an image acquired by the first camera, thereby causing the self-propelled robot to perform the work in the workspace, and to disconnect the connection with the first camera when a predetermined end condition is satisfied. In this embodiment, the following effects can be obtained by setting in advance the orientation and imaging range of the first camera, which is disposed independently of the multiple self-propelled robots, so that an image suitable for use in controlling the movement of the robot working in the workspace can be acquired. That is, while using the same first camera for multiple self-propelled robots, the movement of each robot can be controlled based on images acquired by the first camera without being limited by the position or angle of the robot's hand during work. Therefore, the movement of the robot during work can be controlled based on images with an angle and imaging range suitable for controlling the movement of the robot during work.

[0101] (2) In the robot system of the above aspect, the start condition may be that the work to be performed in the workspace by the self-propelled robot equipped with the control unit is started. By adopting this configuration, the communication bandwidth between each control unit of multiple self-propelled robots and the first camera can be reduced compared to a configuration in which the control unit is communicatively connected to the first camera before work begins.

[0102] (3) In the robot system of the above aspect, the termination condition may be that the work performed in the workspace by the self-propelled robot equipped with the control unit is terminated. By adopting this configuration, the communication bandwidth between each control unit of multiple self-propelled robots and the first camera can be reduced compared to a configuration in which the control unit remains communicatively connected to the first camera even after work is completed.

[0103] (4) In the robot system of the above form, the first camera can be configured to change at least one of the shooting angle and the shooting range by being controlled by one of the control units of the multiple self-propelled robots, and the control unit can obtain control rights to control the first camera prior to performing the task. By adopting such an embodiment, the movement of the robot during work can be controlled based on images from an angle and imaging range that is more suitable for controlling the movement of the robot during work.

[0104] (5) In the robot system of the above embodiment, the first camera may be configured to be controlled by only one control unit among the control units of the multiple self-propelled robots that has the control right, and the control unit may relinquish the control right after performing the task. With this configuration, the first camera is exclusively controlled by only one control unit. As a result, when multiple self-propelled robots are working in parallel, the robot working to obtain images required for control is not hindered from obtaining images required by the other self-propelled robots. Then, since the control right is relinquished after the work is performed, the first camera can be selectively controlled by the control units of the multiple self-propelled robots.

[0105] (6) In the robot system of the above form, the first camera may be capable of providing images to other control units of the plurality of self-propelled robots while being controlled by one of the control units of the plurality of self-propelled robots. By adopting such an embodiment, while one self-propelled robot is performing a task based on an image from a first camera, another self-propelled robot can perform a task based on the same image from the first camera.

[0106] (7) In the robot system of the above form, a second camera is provided that is arranged independently of the plurality of self-propelled robots and can capture images of a work space in which the plurality of self-propelled robots perform work, and the control unit is configured to be able to connect to the second camera so as to be able to communicate when a predetermined start condition is satisfied, and to control the operation of the self-propelled robot among the plurality of self-propelled robots to which the control unit is provided based on an image acquired by the second camera, thereby causing the self-propelled robot to perform the work in the work space, and to disconnect the connection to the second camera when a predetermined end condition is satisfied, and the control unit can be configured to select the camera to which the control unit will connect so as to be able to communicate prior to performing the work, based on information that corresponds the first camera or the second camera to the work. By adopting such an embodiment, images can be obtained from a camera that can provide images with an angle and imaging range that is more suitable for controlling the movement of the robot while it is working, and the movement of the robot while it is working can be controlled.

[0107] (8) According to another aspect of the present disclosure, there is provided a self-propelled robot that performs a task in a workspace, the self-propelled robot communicatively connects to a first camera that is disposed at a position different from the self-propelled robot and captures an image of the workspace when a predetermined start condition is satisfied, performs the task based on an image acquired by the first camera, and disconnects the connection to the first camera when a predetermined end condition is satisfied.

[0108] (9) According to another aspect of the present disclosure, there is provided a method for controlling a self-propelled robot that performs a task in a workspace, the method for controlling the self-propelled robot including the steps of: when a predetermined start condition is satisfied, establishing a communicative connection with a first camera that is disposed at a position different from the autonomous robot and captures an image of the workspace, executing the task based on an image acquired by the first camera, and when a predetermined end condition is satisfied, disconnecting the connection with the first camera.

[0109] The present disclosure may be realized in various forms other than a robot system, a self-propelled robot, and a control method for a self-propelled robot, for example, a robot, a manufacturing method for a robot, a control method for a robot, a computer program for implementing these control methods, a non-transitory recording medium on which the computer program is recorded, etc. [Explanation of symbols]

[0110] 1...robot system, 100...control unit, 110...CPU, 111...trajectory planning unit, 112...arm control unit, 113...hand control unit, 114...path planning unit, 115...camera control unit, 116...object recognition unit, 120...memory, 140...arm control interface, 150...hand control interface, 170...camera control interface, 200...robot arm, 300...hand, 400...vehicle, AMR...self-propelled robot, AMR1...self-propelled robot, AMR2...self-propelled robot, CM...camera, CM1...first camera, CM2...second camera, CN1...connection between self-propelled robot AMR1 and first camera CM1, CN2...connection between self-propelled robot AMR1 and first camera CM2, IR1...first imaging range, IR2...second imaging range, OS...work space, SV...server, WP...workpiece

Claims

1. A plurality of self-propelled robots each having a control unit; a first camera that is arranged independently of the plurality of self-propelled robots and is capable of capturing an image of a working space in which the plurality of self-propelled robots perform work; The control unit is When a predetermined start condition is satisfied, a communication connection is established with the first camera; controlling the operation of a self-propelled robot including the control unit among the plurality of self-propelled robots based on the image acquired by the first camera, thereby causing the self-propelled robot to perform the task in the working space; disconnecting the connection with the first camera when a predetermined termination condition is met; The robot system is configured to:

2. 2. The robot system according to claim 1, A robot system, wherein the start condition is that the task to be performed in the workspace by the self-propelled robot equipped with the control unit is started.

3. 2. The robot system according to claim 1, A robot system, wherein the termination condition is that the work performed in the workspace by the self-propelled robot equipped with the control unit is completed.

4. 2. The robot system according to claim 1, the first camera is configured to be able to change at least one of an imaging angle and an imaging range by being controlled by one of the control units of the plurality of self-propelled robots, The control unit obtains control authority for controlling the first camera prior to performing the task.

5. The robot system according to claim 4, the first camera is configured to be controlled by only one control unit having the control right among the control units of the plurality of self-propelled robots, The control unit relinquishes control after performing the task.

6. The robot system according to claim 4, A robot system, wherein the first camera, while being controlled by one of the control units of the plurality of self-propelled robots, can provide images to other control units of the plurality of self-propelled robots.

7. 2. The robot system according to claim 1, a second camera that is arranged independently of the plurality of self-propelled robots and is capable of capturing an image of a working space in which the plurality of self-propelled robots perform work; The control unit is When a predetermined start condition is satisfied, a communication connection is established with the second camera; controlling the operation of a self-propelled robot including the control unit among the plurality of self-propelled robots based on the image acquired by the second camera, thereby causing the self-propelled robot to perform the task in the working space; disconnecting the connection with the second camera when a predetermined termination condition is met; It is configured to be able to The control unit is A robot system in which the control unit selects a camera to be communicatively connected prior to performing the task based on information correlating the first camera or the second camera with the task.

8. A self-propelled robot that performs work in a work space, When a predetermined start condition is satisfied, a first camera is communicatively connected to a first camera that is disposed at a position different from the self-propelled robot and captures an image of a working space, Performing the task based on an image acquired by the first camera; The self-propelled robot disconnects the connection with the first camera when a predetermined termination condition is met.

9. A method for controlling a self-propelled robot that performs a task in a working space, comprising: When a predetermined start condition is satisfied, a step of communicatively connecting to a first camera disposed at a position different from the autonomous robot and capturing an image of a working space; Executing the task based on an image acquired by the first camera; and disconnecting the connection to the first camera when a predetermined termination condition is satisfied.

Citation Information

Patent Citations

  • Robot system

    JP2019104093A

Cited By

  • Schedule management application and information processing device

    EP4822575A1