Robot system, self-propelled robot, and method for controlling self-propelled robot

By presetting the camera orientation and image shooting range in the robot system, the problem of inappropriate camera acquisition and control of the robot motion angle and image shooting range in the prior art is solved, and more accurate and efficient robot motion control is achieved, reducing communication Frequency band and parameter setting complexity.

CN119017356APending Publication Date: 2024-11-26SEIKO EPSON CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410641047.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-25
Filing Date
2024-05-22
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the prior art, when using a camera to acquire images to control the movement of a robot, it is difficult to obtain a suitable angle and image capture range, resulting in inaccurate motion control.

Method used

Provided is a robot system that allows the control unit to communicate with the camera under the satisfaction of pre-determined conditions, acquires images and controls the movement of the robot until it is satisfied. Disconnect when the condition is terminated.

Benefits of technology

It realizes that the robot's movement is controlled based on images suitable for the angle and camera range without being limited by the position of the robot's hands and robotic arms, improves the accuracy and efficiency of the movement, reduces the communication frequency band, and simplifies the sharing of parameters and algorithms. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119017356A_ABST
    Figure CN119017356A_ABST
Patent Text Reader

Abstract

The invention relates to a robot system, a self-propelled robot, and a method for controlling the self-propelled robot. An image of which the angle and the imaging range are suitable for controlling the operation of a robot in operation is obtained. The robot system includes: a plurality of self-propelled robots each including a control unit; and a first camera which is assembled independently from the plurality of self-propelled robots and is capable of capturing an image of a work space in which the plurality of self-propelled robots work. The control unit is configured so as to be capable of: communicably connecting to the first camera when a predetermined start condition is satisfied; a control unit that causes the self-propelled robots to execute work in the work space by controlling the operation of the self-propelled robots including a control unit among the plurality of self-propelled robots on the basis of the image acquired by the first camera; and cutting off the connection to the first camera when a predetermined termination condition is satisfied.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a robot system, a self-propelled robot, and a control method of the self-propelled robot. Background Art

[0002] There is a technology related to a camera unit that can be selectively installed on each robot among multiple robots and acquire images of an inspected object. In the technology of Patent Document 1, the camera unit can be installed and removed on multiple robots. Multiple robots are used to inspect the inspected object. The same camera unit is sequentially installed in multiple robots and is used to acquire images. Each image inspection unit connected to each robot uses the image acquired by the same camera unit and processed by aberration correction based on the same optical characteristic data for inspection. Therefore, images of a certain quality can be used in all inspections performed by the image inspection unit, and the inspection accuracy can be stabilized.

[0003] On the other hand, there is a technology that controls the motion of a robot based on an image acquired by a camera when the robot performs some kind of work on an object, rather than inspecting the object.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2019-104093

[0005] The multiple robots in the technology of Patent Document 1 have the purpose of their actions to obtain images of the inspected object in order to perform inspections. Therefore, in order to control the actions of the robots based on the images obtained by the cameras, when the technology of Patent Document 1 is used, it is not always possible to obtain images with angles and camera ranges suitable for controlling the actions of the robots in operation. Summary of the invention

[0006] According to one embodiment of the present disclosure, a robot system is provided. The robot system comprises: a plurality of self-propelled robots, each of which comprises a control unit; and a first camera, which is independently mounted on the plurality of self-propelled robots and is capable of photographing a work space in which the plurality of self-propelled robots perform work. The control unit is configured to be able to: be communicatively connected to the first camera when a predetermined start condition is met; control the action of a self-propelled robot equipped with the control unit among the plurality of self-propelled robots based on an image acquired by the first camera, so that the self-propelled robot performs the work in the work space; and cut off the connection with the first camera when a predetermined end condition is met. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 1 is an explanatory diagram of a robot system 1 as a first embodiment of the present disclosure.

[0008] Figure 2 1 is a block diagram showing the functional parts of the control unit 100 of the self-propelled robot AMR.

[0009] Figure 3 This is an explanatory diagram showing a configuration for realizing mutual communication among the autonomous robot AMR, the camera CM, and the server SV.

[0010] Figure 4 This is a flowchart showing the processing when the self-propelled robot AMR performs work.

[0011] Figure 5 This is a flowchart showing the processing when the self-propelled robot AMR performs work in the second embodiment.

[0012] Description of Reference Numerals

[0013] 1: Robot system; 100: Control unit; 110: CPU; 111: Track planning unit; 112: Arm control unit; 113: Hand control unit; 114: Path planning unit; 115: Camera control unit; 116: Object recognition unit; 120: Memory; 140: Interface for arm control; 150: Interface for hand control; 170: Interface for camera control; 200: Robot arm; 300: Hand; 400: Vehicle; AMR: Self-propelled robot; AMR1: Self-propelled robot; AMR2: Self-propelled robot; CM: Camera; CM1: First camera; CM2: Second camera; CN1: Connection between the self-propelled robot AMR1 and the first camera CM1; CN2: Connection between the self-propelled robot AMR1 and the second camera CM2; IR1: First imaging range; IR2: Second imaging range; OS: Work space; SV: Server; WP: Workpiece. DETAILED DESCRIPTION

[0014] A. First Implementation Method:

[0015] Figure 1 1 is an explanatory diagram of a robot system 1 as a first embodiment of the present disclosure. The robot system 1 is a system that performs work on a workpiece WP in a work space OS by a plurality of robots. The robot system 1 includes self-propelled robots AMR1 and AMR2, a first camera CM1, a second camera CM2, and a server SV.

[0016] The self-propelled robots AMR1 and AMR2 autonomously move in a factory where the robot system 1 is installed based on the work instructions received from the server SV, and perform work on the workpiece WP in the work space OS (see Figure 1In this specification, a "self-propelled robot" means a robot that can move around obstacles without external instructions. Hereinafter, when referring to self-propelled robots AMR1 and AMR2 without distinguishing between them, they are referred to as "self-propelled robots AMR".

[0017] The self-propelled robot AMR can obtain images from the first camera CM1 and the second camera CM2. The self-propelled robot AMR works on the workpiece WP based on the images obtained by the first camera CM1 and the second camera CM2. The self-propelled robot AMR includes a control unit 100, a robot arm 200, a hand 300, and a vehicle 400.

[0018] The robot arm 200 has six joints. Three of the six joints are bending joints. The other three of the six joints are torsion joints. A servo motor and a position sensor are provided in each joint. The servo motor generates a rotation output for driving each joint. The position sensor detects the angular position of the output shaft of the servo motor. The robot arm 200 is controlled by the control unit 100 to drive the servo motor of each joint, thereby being able to assemble the hand 300 connected to the front end of the robot arm 200 at a target position in a three-dimensional space in a target posture.

[0019] The hand 300 is installed at the front end of the robot arm 200. The hand 300 is a so-called end effector. The hand 300 is controlled by the control unit 100, and can grasp the workpiece WP as the object of operation, and can also release the grasped workpiece. The hand 300 and the robot arm 200 are controlled by the control unit 100, and can grasp the workpiece to move the workpiece to a target position in a three-dimensional space.

[0020] The vehicle 400 supports the robot arm 200. The vehicle 400 is controlled by the control unit 100, and can move the self-propelled robot AMR to an arbitrary position on the floor surface.

[0021] The control unit 100 controls the robot arm 200, the hand 300, and the vehicle 400. The configuration and function of the control unit 100 will be described later.

[0022] The first camera CM1 is independently assembled with the self-propelled robot AMR (refer to Figure 1). It should be noted that the camera "independently assembled with the robot" means that the camera is not fixed to the robot, but is fixed at a certain position and does not move with the robot. The first camera CM1 is capable of photographing the working space OS where the self-propelled robot AMR operates. The first camera CM1 is connected to the self-propelled robot AMR, and is capable of sending image data obtained by photographing to the self-propelled robot AMR. The first camera CM1 is a monocular digital camera. The first camera CM1 corresponds to ROS2 (Robot Operating System 2).

[0023] The first camera CM1 can be connected to the self-propelled robot AMR without going through the server SV. Therefore, compared with a method in which the first camera CM1 and the self-propelled robot AMR communicate through the server SV, the time delay of the communication is less. As a result, for example, even when image feedback control is performed in the self-propelled robot AMR based on the image obtained by the first camera CM1, accurate control can be performed without reducing the movement speed of the self-propelled robot AMR.

[0024] The second camera CM2 is independently assembled with the self-propelled robot AMR (refer to Figure 1 The second camera CM2 is capable of photographing the workspace OS where the self-propelled robot AMR works. The hardware configuration of the second camera CM2 is the same as that of the first camera CM1. However, the angle and range of the second camera CM2 photographing the workspace OS are different from those of the first camera CM1. Figure 1 The first imaging range IR1 is shown in FIG. Figure 1 The range in which the second camera CM2 takes images of the work space OS is within Figure 1 The second imaging range IR2 is shown in FIG. Figure 1 ).

[0025] The first imaging range IR1 includes the entire work space OS. On the other hand, the second imaging range IR2 includes the workpiece WP assembled in the work space OS, but does not include part of the work space OS. The second camera CM2 further magnifies the vicinity of the workpiece WP for photography. Therefore, compared with the first camera CM1, the second camera CM2 can photograph a smaller structure. The self-propelled robot AMR can perform smaller operations based on the images received from the second camera CM2.

[0026] The other points of the second camera CM2 are the same as those of the first camera CM1. Hereinafter, when referring to a camera without distinguishing between the first camera CM1 and the second camera CM2, it is described as a camera CM.

[0027] The server SV is communicatively connected to the self-propelled robots AMR1, AMR2, the first camera CM1, and the second camera CM2. More specifically, the server SV is connected to the self-propelled robots AMR1, AMR2, the first camera CM1, and the second camera CM2 via a wireless LAN (Local Area Network). The server SV can control the self-propelled robots AMR1, AMR2, the first camera CM1, and the second camera CM2.

[0028] The server SV has: a CPU (Central Processing Unit) as a processor, a RAM (Random Access Memory) and a ROM (Read-Only Memory). The RAM includes a main memory as a semiconductor memory and a hard disk as an auxiliary storage device. The hard disk stores an operating system, a program for controlling the self-propelled robot AMR, a program for controlling the camera CM, etc. The CPU loads the computer program stored in the hard disk into the main memory and executes it, thereby realizing various functions such as sending job instructions to the self-propelled robot AMR.

[0029] Figure 2 1 is a block diagram showing the functional parts of the control unit 100 of the self-propelled robot AMR. The control unit 100 includes a CPU 110 as a processor, a memory 120 , an arm control interface 140 , a hand control interface 150 , and a camera control interface 170 .

[0030] The memory 120 includes a RAM and a ROM. The RAM includes a main memory as a semiconductor memory and a hard disk as an auxiliary storage device. The hard disk stores an operating system, middleware, a robot control program, etc. More specifically, the operating system is Linux (registered trademark). In addition, more specifically, the middleware is ROS2 (Robot Operating System 2). The CPU 110 realizes real-time control of the self-propelled robot AMR by loading the computer program stored in the hard disk into the main memory and executing it.

[0031] The CPU 110 controls the robot arm 200 through the arm control interface 140. Figure 2The arm control unit 112 is shown in FIG. 1 . The CPU 110 controls the hand 300 through the hand control interface 150. The functional unit of the CPU 110 for controlling the hand 300 is shown in FIG. Figure 2 The CPU 110 communicates with the first camera CM1 and the second camera CM2 via the camera control interface 170. The functional part of the CPU 110 that communicates with the first camera CM1 and the second camera CM2 is shown in FIG. Figure 2 The camera control unit 115 is shown in FIG.

[0032] The CPU 110 recognizes the position and posture of the object included in the image data based on the image data received from the first camera CM1 and the second camera CM2. The CPU 110 can recognize the current position of the self-propelled robot AMR based on the recognized object. Figure 2 The object recognition unit 116 is shown in FIG.

[0033] CPU 110 determines the movement path of the self-propelled robot AMR on the floor surface in the factory based on the position of the self-propelled robot AMR in the factory, the position of the work space OS in the factory, and the position and volume of obstacles that become obstacles to the movement of the self-propelled robot AMR. The position of the work space OS in the factory, the position and volume of obstacles that become obstacles to the movement of the self-propelled robot AMR are received from the server SV (refer to Figure 1 The moving path of the self-propelled robot AMR on the floor surface in the factory is determined in a manner that does not collide with obstacles. The function unit of the CPU 110 that determines the moving path of the self-propelled robot AMR on the floor surface is Figure 2 Shown in the figure is a track planning unit 111.

[0034] The CPU 110 determines the movement path of the hand 300 based on the angles of the joints of the robot arm 200, the position of the workpiece WP in the work space OS, and the position and volume of obstacles that hinder the movement of the robot arm 200 and the hand 300. The movement path of the hand 300 is determined so that the hand 300 does not collide with obstacles or the self-propelled robot AMR itself. The function unit of the CPU 110 for determining the movement path of the hand 300 is placed in Figure 2 The path planning unit 114 is shown in FIG.

[0035] Figure 3 This is an explanatory diagram showing a configuration for realizing mutual communication among the self-propelled robot AMR, the camera CM, and the server SV. The camera CM includes an imaging system and a communication layer (see Figure 3The camera system is a hardware including an optical system and a CMOS (Complementary Metal Oxide Semiconductor) image sensor. Communication layer: It has a wireless LAN layer, a UDP (User Datagram Protocol) layer, and a DDS (Data Distribution Service) layer. Through these structures, publish / subscribe type communication is realized in the camera CM. Specifically, the camera CM repeatedly publishes the images obtained by the camera to the self-propelled robot AMR and the server SV at a certain period through the above structure.

[0036] The server SV includes a camera control unit (see Figure 3 The camera control unit of the server SV has a wireless LAN layer, a UDP layer, and a DDS layer, similar to the camera CM. The camera control unit 115 of the self-propelled robot AMR also has a wireless LAN layer, a UDP layer, and a DDS layer. Through these structures, publish / subscribe type communication is implemented between the server SV and the self-propelled robot AMR. Specifically, the server SV and the self-propelled robot AMR subscribe to the images obtained by the camera CM from the camera CM. As a result, the images obtained by the camera CM are repeatedly sent to the self-propelled robot AMR and the server SV.

[0037] Figure 4 This is a flowchart showing the processing when the self-propelled robot AMR performs work. Figure 4 The processing is performed by the control unit 100 of the self-propelled robot AMR (refer to Figure 1 The following description will take the case where the self-propelled robot AMR1 performs an operation among the self-propelled robots AMR1 and AMR2 as an example.

[0038] In step S100, the control unit 100 determines whether an instruction to start a job has been received from the server SV. The instruction to start a job includes: (i) information uniquely indicating the job to be started, (ii) information indicating one or more job contents included in the job to be started, and (iii) information associating at least one of the first camera CM1 and the second camera CM2 with the job to be started. The control unit 100 stands by until an instruction to start a job is received from the server SV. When an instruction to start a job is received from the server SV, the processing proceeds to step S200. That is, the following processing is performed on the condition that the self-propelled robot AMR1 having the control unit 100 starts the job executed in the work space OS.

[0039] In step S200, the control unit 100 selects a camera CM to be used in a job to be performed, that is, one or more cameras CM to which the control unit 100 is communicatively connected. The camera CM is selected based on information associating one or both of the first camera CM1 and the second camera CM2 with the job, which is included in the instruction to start the job received from the server SV.

[0040] By performing such processing, the self-propelled robot AMR1 can be controlled by acquiring images from a camera that can provide images with suitable angles and imaging ranges. The following description will be made by selecting the first camera CM1.

[0041] In step S300, the control unit 100 is connected to the selected camera CM via the camera control unit 115 so as to be communicable (see Figure 2 As a result, the self-propelled robot AMR starts to subscribe to the image captured by the camera CM. This process is also called "subscription start process". The connection between the self-propelled robot AMR1 and the first camera CM1 is Figure 1 The connection between the self-propelled robot AMR1 and the second camera CM2 is shown as CN1. Figure 1 It should be noted that the connection that is connected when a predetermined condition is satisfied during processing by the robot system 1 and disconnected when a predetermined condition is satisfied is shown in FIG. Figure 1 Indicated by dotted lines.

[0042] In this embodiment, the control unit 100 is connected to the camera CM in a communicative manner under the condition that the operation starts. Therefore, compared with a method in which the control unit 100 is connected to the camera in a communicative manner before receiving an instruction to start the operation, the frequency band of the communication CM1 and CM2 between the control units 100 of the self-propelled robots AMR1 and AMR2 and the camera can be reduced.

[0043] In steps S400 to S600 , the control unit 100 controls the operation of the self-propelled robot AMR1 included in the control unit 100 based on the image acquired by the connected camera CM, thereby causing the self-propelled robot AMR1 to perform work in the work space OS.

[0044] In step S400 , the control unit 100 receives an image acquired by the connected camera CM from the camera CM.

[0045] In step S500, the object recognition unit 116 of the control unit 100 recognizes a target position based on the image received from the connected camera CM. For example, the target position where the workpiece WP is placed in the work space OS is recognized.

[0046] The path planning unit 114 of the control unit 100 determines the movement path of the hand 300 based on the angles of the joints of the robot 200, the position of the workpiece WP in the work space OS, and the position and volume of obstacles that hinder the movement of the robot 200 and the hand 300 (see Figure 2 left middle section).

[0047] In step S600 , the control unit 100 executes a work command included in the work. According to the work command, the control unit 100 moves the hand 300 to the next relay point on the moving path of the hand 300 , or makes the hand 300 grasp the workpiece, or makes the hand 300 release the workpiece.

[0048] Specifically, the arm control unit 112 of the control unit 100 controls the robot arm 200 through the arm control interface 140 (see Figure 2 The hand control unit 113 of the control unit 100 controls the hand 300 via the hand control interface 150 (see Figure 2 left middle section).

[0049] In step S700, the control unit 100 determines whether there are any remaining job commands among the job commands included in the job. If there are any remaining job commands, the process returns to step S600 and the next job command is executed. If there are no remaining job commands, the process proceeds to step S800.

[0050] In step S800, the control unit 100 disconnects the camera CM. That is, the connection between the control unit 100 of the self-propelled robot AMR1 and the camera CM is disconnected on the condition that the work performed by the self-propelled robot AMR1 having the control unit 100 in the work space OS is completed. As a result, the subscription of the self-propelled robot AMR to the images captured by the camera CM ends. After that, the images are not sent from the camera CM to the self-propelled robot AMR1. Then, Figure 4 The processing is finished.

[0051] By performing such processing, compared with a method in which the control unit 100 is communicatively connected to the camera CM after the work is completed, the frequency bands of the communications CN1 and CN2 between the control units 100 of the self-propelled robots AMR1 and AMR2 and the camera CM can be reduced.

[0052] In the above, the processing in the robot system 1 is described by taking the case where the self-propelled robot AMR1 performs work using the image of the first camera CM1 as an example. However, the processing is performed similarly when the self-propelled robot AMR1 performs work using the image of the second camera CM2. In addition, the same processing is performed even in the self-propelled robot AMR2.

[0053] In this embodiment, the orientation and imaging range of the camera independently mounted on the self-propelled robots AMR1 and AMR2 are pre-set so that an image suitable for controlling the movement of the robots in operation in the operating space OS can be obtained, thereby achieving the following effects (see Figure 1 That is, when the same camera CM is used in the self-propelled robots AMR1 and AMR2, the actions of the self-propelled robots AMR1 and AMR2 can be controlled based on the images acquired by the camera CM without being limited by the positions and angles of the hands 300 of the self-propelled robots AMR1 and AMR2 in operation. Therefore, the actions of the self-propelled robot AMR1 in operation can be appropriately controlled based on the images whose angles and imaging ranges are suitable for controlling the actions of the self-propelled robot AMR1 in operation.

[0054] In addition, since the same camera CM can be used in the self-propelled robots AMR1 and AMR2, the work performed by the self-propelled robots AMR1 and AMR2 can be performed by a smaller number of cameras compared to a method in which the self-propelled robots AMR1 and AMR2 are provided with two cameras at positions where the photographic range is not blocked by the hand 300 and the robot arm 200 during the work.

[0055] Furthermore, since the same camera CM can be used in the self-propelled robots AMR1 and AMR2, common values ​​can be used as action parameters in the control unit 100 of the self-propelled robots AMR1 and AMR2, such as parameters for image recognition based on the image of the camera CM, parameters for feedback control based on the image of the camera CM, parameters for machine learning for operations, and the like. Therefore, there is no need to set these parameters for each of the self-propelled robots AMR1 and AMR2. In addition, a common algorithm can be used between the self-propelled robots AMR1 and AMR2 as an algorithm for using the image of the camera CM. Even if the self-propelled robots AMR1 and AMR2 are replaced with other self-propelled robots due to maintenance or failure, the parameters and algorithms that have been used until then can be applied to the new robot. As a result, the startup time after maintenance and repair can be shortened.

[0056] In this embodiment, the condition for determining whether or not the job is successful in step S100, i.e., receiving an instruction to start the job from the server SV, is also referred to as a "start condition". In this embodiment, the condition for determining whether or not the job is successful in step S700, i.e., there are no remaining job commands among the job commands included in the job, is also referred to as an "end condition".

[0057] B. Second Implementation Method:

[0058] The robot system of the second embodiment differs from the first embodiment in the configuration and function of the first camera CM1 and the second camera CM2 and in some processing of the control unit 100 of the autonomous robots AMR1 and AMR2 during operation. The other points of the robot system of the second embodiment are the same as the robot system 1 of the first embodiment.

[0059] In the second embodiment, the first camera CM1 is configured to be able to change the photographic angle and the photographic range. That is, the first camera CM1 has the functions of panning, pitching and zooming. The first camera CM1 changes the photographic angle and the photographic range by being controlled by the control unit 100 having the control right for controlling the first camera CM1 in the control unit 100 of the self-propelled robots AMR1 and AMR2. The "control right" of a certain camera CM is the right to change at least one of the photographic angle and the photographic range of the camera CM according to instructions. In the present embodiment, the control unit 100 of the self-propelled robot AMR having the control right of the first camera CM1 can change both the photographic angle and the photographic range of the first camera CM1 according to instructions. The control right of the first camera CM1 cannot be maintained by multiple control units 100 at the same time. The first camera CM1 is configured to be controlled by only one control unit 100 having the control right in the control units 100 of the self-propelled robots AMR1 and AMR2.

[0060] On the other hand, while being controlled by one of the control units 100 of the self-propelled robots AMR1 and AMR2, the first camera CM1 can provide images to the other control unit 100 of the self-propelled robots AMR1 and AMR2 (see Figure 3 ). That is, the image is also released to the control unit 100 of the self-propelled robot AMR and the server SV other than the control unit 100 of the self-propelled robot AMR having control authority (refer to Figure 3 ).

[0061] With such a configuration, while a certain self-propelled robot AMR1 is performing work based on the image of the first camera CM1 , another self-propelled robot AMR2 can perform work based on the same image of the first camera CM1 .

[0062] In the second embodiment, the second camera CM2 also has the same configuration and function as the first camera CM1 .

[0063] Figure 5 This is a flowchart showing the processing when the self-propelled robot AMR performs work in the second embodiment. Figure 5 The processing is performed by the control unit 100 of the self-propelled robot AMR. Figure 5 In the process of, steps S300B and S800B are respectively executed to replace Figure 4Steps S300 and S800 in the process of Figure 5 During the processing, steps S250, S300C, S350, S400C, S450, and S470 are also executed. Figure 5 The other points of the treatment are Figure 4 The following description will be made by taking the case where the self-propelled robot AMR1 performs the operation among the self-propelled robots AMR1 and AMR2 as an example.

[0064] exist Figure 5 In the process, the process of step S250 is performed after step S200. In step S250, the control unit 100 determines whether the other self-propelled robot AMR has the control right of the camera CM to be used in the executed work. When the other self-propelled robot AMR does not have the control right of the camera CM to be used in the executed work, the process proceeds to step S300B. When the other self-propelled robot AMR has the control right of the camera CM to be used in the executed work, the process proceeds to step S300C.

[0065] In step S300B, the control section 100 is connected to the selected camera CM via the camera control section 115 so as to be communicable. Figure 4 The control unit 100 also obtains the control right of the camera CM to be used in the executed operation before executing the operation. Here, it is assumed that the control right of the first camera CM1 is obtained by the control unit 100 of the self-propelled robot AMR1. As a result, the control of the photographing angle and the photographing range of the first camera CM1 cannot be performed by the control unit 100 of the self-propelled robot AMR2, but can only be performed by the control unit 100 of the self-propelled robot AMR1.

[0066] With such a configuration, the control unit 100 can control the operation of the self-propelled robot AMR1 during operation based on an image having an appropriate angle and imaging range by controlling the operation of the self-propelled robot AMR1 during operation.

[0067] In the second embodiment, the camera CM is exclusively controlled by only one control unit 100. Therefore, when the autonomous robots AMR1 and AMR2 are operating in parallel, the autonomous robot AMR1 in operation does not prevent the autonomous robot AMR2 from acquiring images required for control.

[0068] Figure 5 The processing of steps S400, S500, S600, and S700 is Figure 4The processing of steps S400, S500, S600, and S700 is the same. In step S700, if there is no remaining job command, the processing proceeds to step S800B.

[0069] In step S800B, the control unit 100 disconnects the camera CM. Figure 4 The control unit 100 also gives up the control right of the camera CM. Then, Figure 5 The processing is finished.

[0070] In this embodiment, after executing the job, the control right is abandoned (refer to Figure 5 Therefore, the camera CM can be selectively controlled by the control unit 100 of the self-propelled robots AMR1 and AMR2.

[0071] On the other hand, in step S300C, the control section 100 is connected to the selected camera CM via the camera control section 115 so as to be communicable. Figure 4 However, unlike the process in step S300B, in step S300C, the control unit 100 does not acquire the control right of the camera CM to be used in the executed job (refer to Figure 5 S300B).

[0072] In step S400C, the control unit 100 receives the image acquired by the connected camera CM from the camera CM. Figure 4 However, in the second embodiment, the self-propelled robot AMR having control over the camera CM receives images from the camera CM in step S400 and the self-propelled robot AMR not having control over the camera CM receives images from the camera CM in step S400C at the same time.

[0073] In step S450, the object recognition unit 116 of the control unit 100 determines whether the target position can be recognized based on the image received from the connected camera CM. For example, it is determined whether the target position where the workpiece WP should be placed in the work space OS can be recognized. In the case where the target position cannot be recognized based on the image received from the connected camera CM, the process proceeds to step S470. In the case where the target position can be recognized based on the image received from the connected camera CM, the process proceeds to step S500. Then, the robot performs the work based on the image obtained from the camera CM without having the control right of the camera CM used in the work.

[0074] By performing such processing, the self-propelled robot AMR that has not been able to obtain control rights can be effectively and flexibly used to advance the work.

[0075] In step S470, the control unit 100 waits for a predetermined time. Then, the process returns to step S250. By adopting such a configuration, the self-propelled robots AMR1 and AMR2 can sequentially acquire control rights of the cameras used in the operation when necessary, so that the operation can be performed based on the preferred image.

[0076] C. Other implementations:

[0077] C1. Other implementations 1:

[0078] (1) In the first embodiment, the camera CM is a monocular digital camera, but the camera may be a stereo camera or an RGBD camera capable of acquiring depth information in addition to RGB images.

[0079] (2) In the first embodiment described above, the CPU 110 functions as the trajectory planning unit 111, the path planning unit 114, and the object recognition unit 116 (see Figure 2 However, the control unit 100 includes a GPU (Graphics Processing Unit), and the GPU can also realize part or all of the functions of the track planning unit 111, the path planning unit 114, and the object recognition unit 116.

[0080] (3) In the first embodiment described above, the workspace OS is fixed (see Figure 1 However, the operating space OS may be defined according to the operation. In addition, the operating space OS may be defined according to the combination of the operation and the robot performing the operation.

[0081] (4) In the first embodiment, the camera CM, the self-propelled robot AMR, and the server SV perform publish / subscribe type communication via DDS, a communication framework of ROS2 (see Figure 3 ). However, the camera CM, the autonomous robot AMR, and the server SV may also transmit and receive image data using other methods such as a communication framework of ROS (Robot Operating System).

[0082] (5) In the first embodiment described above, the camera CM included in the robot system publishes to the self-propelled robot AMR and the server SV (see Figure 3However, it is also possible to adopt a method in which no publish / subscribe type communication is performed between the camera CM and the self-propelled robot AMR and the server SV. In such a method, the self-propelled robot AMR and / or the server SV repeatedly send instructions for filming, and the camera CM repeatedly films and sends images according to the instructions.

[0083] (6) In the first embodiment described above, the camera CM is connected to the self-propelled robot AMR and the server SV via a wireless LAN (see Figure 3 ). However, the camera CM and the self-propelled robot AMR may be connected by a communication method based on other communication standards such as Bluetooth (trademark).

[0084] (7) In the first embodiment described above, the control unit 100 selects the camera CM based on the information associating one or both of the first camera CM1 and the second camera CM2 with the job, which is included in the instruction to start the job received from the server SV (see Figure 2 However, the control unit may select the camera in other ways.

[0085] For example, information associating the camera CM with the job may be sent from the server SV to the self-propelled robot AMR separately from the instruction to start the job. In addition, information associating the camera CM with the job may be stored in advance in the memory 120 of the self-propelled robot AMR. The information specifying the camera may be either an ID assigned to the camera or an IP address assigned to the camera. In the case where the information specifying the camera is an IP address, the control unit 100 can connect to the IP address according to the job so that the image is sent.

[0086] Furthermore, for example, the data issued by the camera CM includes information indicating a task, and the control unit 100 of the self-propelled robot AMR can also perform the task using an image of data including information indicating a task to be performed by the self-propelled robot AMR among the data sent by the plurality of cameras CM.

[0087] Furthermore, when performing a certain operation using images from a plurality of cameras, the following method can be used. That is, the data issued by the camera CM includes information indicating the camera CM that photographed the image and information indicating the operation to be performed based on the image. The control unit 100 of the self-propelled robot AMR executes the operation command using the images of the plurality of data including the information indicating the operation to be performed by the self-propelled robot AMR among the data sent from the plurality of cameras CM.

[0088] In addition, the following method can be used. The data issued by the camera CM includes information indicating a work command to be executed based on the image. The control unit 100 of the self-propelled robot AMR executes the work command using the image including the data indicating the work command executed by the self-propelled robot AMR. At this time, the camera CM that sends the data used in executing the work command can be changed according to the work command.

[0089] In addition, for example, a method can be adopted in which a robot camera is provided in each self-propelled robot, and an external camera within the field of view of the robot camera is selected as a camera used in the operation. When the condition for connecting to the external camera is that the external camera enters the field of view of the robot camera, for example, it is preferable to mark the external camera with a mark for distinguishing each external camera. In such a method, the self-propelled robot and the external camera can be connected based on the mark existing in the image acquired by the robot camera.

[0090] (8) In the first embodiment described above, the movement path of the hand 300 is determined based on the image from the camera CM (see Figure 4 S500 and Figure 2 Then, the robot arm 200 and the hand 300 are controlled along the determined path (refer to Figure 4 S600). However, in controlling the robot arm 200 and the hand 300, the control unit 100 may also perform image feedback control of the robot arm 200 and the hand 300 in real time based on the image received from the connected camera CM. For example, the control unit 100 performs image feedback control so that the distance between the designated part of the workpiece WP and the target position becomes zero. In addition, the control unit 100 may also determine the end of the work when recognizing that the distance between the designated part of the workpiece WP and the target position becomes zero.

[0091] When a plurality of robots are included in the field of view of the camera and image feedback control is performed on one or more designated robots, for example, it is preferable to mark the plurality of robots with a QR code or a mark for distinguishing each robot. In such a manner, image feedback control of each robot can be performed based on the QR code or mark present in the image acquired by the camera.

[0092] (9) In the second embodiment, when the control unit 100 of the self-propelled robot AMR recognizes that another robot has the control right of the camera CM used in the operation after the operation starts, the control unit 100 performs the operation or waits in the state where the other robot has the control right of the camera CM (see Figure 5However, for example, it may be configured such that when a robot having control over the camera CM completes a task, it is determined which of the multiple robots that are using the camera to perform the task should obtain control, so that the determined robot obtains control.

[0093] (10) In the above-mentioned second embodiment, the first camera CM1 is controlled by the control unit 100 having the control right for controlling the first camera CM1 in the control unit 100 of the self-propelled robots AMR1 and AMR2 to change the photographic angle and the photographic range. However, the control right of the camera may be held by a server instead of the robot. In that manner, the camera changes its photographic angle and the photographic range by being controlled by the server. The server is able to set the camera within the photographic angle and the photographic range suitable for the operation of the robot performing the operation so that an image is acquired and the image is sent to the robot performing the operation.

[0094] (11) In the first embodiment described above, the case where the self-propelled robot AMR1 performs work on the workpiece WP in the work space OS independently has been described as an example (see Figure 1 However, the technology disclosed in the present invention can also be applied to the case where two or more robots are coordinated to perform operations. In such a case, it is possible to appropriately perform coordinated control by performing image feedback control on each robot based on images released from the same camera (see Figure 1 AMR1, AMR2 and IR1).

[0095] (12) In the first embodiment described above, after a camera is selected, the work performed by the self-propelled robot AMR begins, and the image of the selected camera is used until the work is completed (see Figure 4 However, the camera that issues the image used for the operation can be switched during the operation.

[0096] (13) In the first embodiment or the second embodiment described above, the server SV may store images received from the camera CM (see Figure 1 The data accumulated in this way can be used flexibly as data for recording the progress of work, analyzing work, and learning the control of the robot.

[0097] (14) In the first embodiment or the second embodiment, after the work performed by the self-propelled robots AMR1 and AMR2 is completed, another robot that performs work on the workpiece WP receives the image from the camera CM. If such a method is used, the robot that performs the next process can confirm the progress of the work performed by the self-propelled robots AMR1 and AMR2, estimate the waiting time, and plan the work.

[0098] (15) In the first embodiment described above, the robot system 1 includes the self-propelled robots AMR1 and AMR2 (see Figure 1 ). However, the robot system may also include more than three self-propelled robots. In addition, the technology disclosed in the present invention can also be applied to a robot system including multiple immobile robots. Multiple immobile robots can alternately use a common camera to perform operations in different time periods. In addition, multiple immobile robots can simultaneously receive images from a common camera, and perform operations based on the images respectively.

[0099] C2. Other implementation methods 2:

[0100] In the first embodiment, the control unit 100 and the camera CM are connected to each other so as to be communicable with each other under the condition that the operation starts. More specifically, under the condition that an instruction to start the operation is received from the server SV, the control unit 100 and the camera CM are connected to each other so as to be communicable with each other (see Figure 4 However, as a condition for the control unit of the self-propelled robot to be communicatively connected to the camera CM used in the operation, other conditions such as the self-propelled robot or a designated part of the self-propelled robot entering the field of view of the camera can also be used.

[0101] When the condition for connecting to the camera is that the self-propelled robot or a designated part of the self-propelled robot enters the field of view of the camera, for example, it is preferable to mark each self-propelled robot with a QR code or a mark for distinguishing each self-propelled robot. In such a manner, the self-propelled robot can be connected to the camera based on the QR code or mark present in the image captured by the camera.

[0102] The robot may use one camera or multiple cameras for operation. When only one camera is used for operation, the start condition for the control unit to be communicably connected to the camera used for operation is different from the start condition for the control unit to be communicably connected to other cameras. On the other hand, when multiple cameras are used for operation, the start condition for the control unit to be communicably connected to the multiple cameras used for operation may be set as a common condition for the cameras used in these operations.

[0103] C3. Other implementation methods 3:

[0104] In the first embodiment, the control unit 100 disconnects the camera CM when the job is completed. More specifically, the control unit 100 disconnects the camera CM when there are no remaining job commands in the job commands included in the job (see Figure 4 However, as a condition for the control unit of the self-propelled robot to cut off the connection with the camera used in the operation, other conditions such as the self-propelled robot or a designated part of the self-propelled robot disappearing from the field of view of the camera can also be used.

[0105] When only one camera is used for the operation, the termination condition for the control unit to cut off the communication with the camera in use is different from the termination condition for the control unit to cut off the communication with other cameras. On the other hand, when multiple cameras are used for the operation, the termination condition for the control unit to cut off the communication with the multiple cameras used in the operation can be made common to the cameras used in the operation.

[0106] In the case where the self-propelled robot or a designated part of the self-propelled robot disappears from the camera's field of view as a condition for disconnecting the connection with the camera, for example, it is preferable to mark each self-propelled robot with a QR code or a mark for distinguishing each self-propelled robot. In such a method, the connection and disconnection of the self-propelled robot with the camera can be performed based on the presence or absence of the QR code or the mark in the image acquired by the camera. In addition, even in a method where such a QR code or mark is not set for the robot, the robot in the camera's field of view can be specified by comparing the operation commands sent to the robot arm 200 and / or hand 300 of each robot and the activities of the robot arm 200 and / or hand 300 in the camera's field of view.

[0107] C4. Other implementation methods 4:

[0108] In the second embodiment, the first camera CM1 and the second camera CM2 are configured to be able to change the photographing angle and the photographing range (see Figure 5 However, as shown in the first embodiment, the first camera CM1 and the second camera CM2 may be configured to have fixed photographic angles and photographic ranges. In addition, the following method may be used: a portion of the plurality of cameras included in the robot system may be configured to be able to change photographic angles and photographic ranges, while the photographic angles and photographic ranges of the other portions may be fixed.

[0109] C5. Other implementation methods 5:

[0110] In the second embodiment, the camera CM is controlled by only one of the control units 100 of the self-propelled robots AMR1 and AMR2 that has control authority (see Figure 5 However, the camera CM is configured to be able to change the photographing angle and the photographing range, and can also be configured to be able to control the photographing angle and the photographing range by any robot included in the robot system.

[0111] C6. Other implementations 6:

[0112] In the second embodiment, the first camera CM1 can provide images to the other control unit 100 of the self-propelled robots AMR1 and AMR2 while being controlled by one of the control units 100 of the self-propelled robots AMR1 and AMR2 (see Figure 3 ). However, it is also possible to configure the camera to provide images only to the object that has control over the camera.

[0113] C7. Other implementations 7:

[0114] In the first embodiment, the configuration and function of the second camera CM2 are the same as those of the first camera CM1 except for its installation position and imaging range (see Figure 1 The left part of the middle section and the right part of the lower section). However, the second camera CM2 may have a different configuration from the first camera CM1. For example, the second camera CM2 may have a fixed imaging angle and imaging range for a camera that includes the entire workspace within its imaging range, and a variable imaging angle and imaging range for a camera that includes a workpiece but excludes part of the workspace as its imaging range.

[0115] D.Other methods:

[0116] The present disclosure is not limited to the above-mentioned embodiments, and can be implemented in various ways within the scope of the main purpose. For example, the present disclosure can be implemented even in the following manner. In order to solve part or all of the technical problems of the present disclosure, or to achieve part or all of the effects of the present disclosure, the technical features in the above-mentioned embodiments corresponding to the technical features in the various methods described below can be appropriately replaced or combined. In addition, as long as the technical features are not described as necessary in this specification, they can be appropriately deleted.

[0117] (1) According to one embodiment of the present disclosure, a robot system is provided. The robot system comprises: a plurality of self-propelled robots, each of which comprises a control unit; and a first camera, which is independently mounted on the plurality of self-propelled robots and is capable of photographing a work space in which the plurality of self-propelled robots perform work. The control unit is configured to: be communicatively connected to the first camera when a predetermined start condition is satisfied; control the movement of a self-propelled robot equipped with the control unit among the plurality of self-propelled robots based on an image acquired by the first camera, so that the self-propelled robot performs the work in the work space; and disconnect the connection with the first camera when a predetermined end condition is satisfied.

[0118] In such a manner, by presetting the orientation and imaging range of the first camera independently mounted on a plurality of self-propelled robots so as to acquire an image suitable for controlling the motion of the robots in operation in the working space, the following effects can be obtained. That is, when the same first camera is used for a plurality of self-propelled robots, the motion of each robot can be controlled based on the image acquired by the first camera without being limited by the position and angle of the fingers of the robots in operation. Therefore, the motion of the robots in operation can be controlled based on an image suitable for controlling the motion of the robots in operation in terms of angle and imaging range.

[0119] (2) In the robot system of the above aspect, the start condition may be a start of the work performed by the self-propelled robot including the control unit in the work space.

[0120] By adopting such a mode, compared with a mode in which the control unit is communicably connected to the first camera before the start of the work, it is possible to reduce the communication band between each control unit of the plurality of self-propelled robots and the first camera.

[0121] (3) In the robot system of the above aspect, the termination condition may be a termination of the work performed by the self-propelled robot including the control unit in the work space.

[0122] By adopting such a configuration, compared with a configuration in which the control unit is still communicatively connected to the first camera after the work is completed, it is possible to reduce the communication band between each control unit of the plurality of self-propelled robots and the first camera.

[0123] (4) In the robot system of the above-mentioned method, it can be set as follows: the first camera is configured to be able to change at least one of the photographing angle and the photographing range by being controlled by any one of the control units of the multiple self-propelled robots, and the control unit obtains control rights for controlling the first camera before performing the operation.

[0124] By adopting such a mode, the operation of the robot during operation can be controlled based on an image having a suitable angle and imaging range.

[0125] (5) In the robot system of the above aspect, the first phase may be controlled by only one control unit having the control right among the control units of the plurality of self-propelled robots, and the control unit may relinquish the control right after executing the task.

[0126] By setting it in this way, the first camera is exclusively controlled by only one control unit. As a result, when multiple self-propelled robots are working in parallel, in order to obtain images required by other self-propelled robots, it will not hinder the acquisition of images required by the robot in operation during control. In addition, since the control right is abandoned after the operation is performed, the first camera can be selectively controlled by the control units of multiple self-propelled robots.

[0127] (6) In the robot system of the above aspect, the first camera may be configured to provide images to other control units of the plurality of self-propelled robots while being controlled by one of the control units of the plurality of self-propelled robots.

[0128] By adopting such a configuration, while a certain self-propelled robot is performing an operation based on an image of the first camera, another self-propelled robot can perform an operation based on the same image of the first camera.

[0129] (7) In the robot system of the above-mentioned manner, it can be set as follows: the robot system includes a second camera, the second camera is independently assembled with the multiple self-propelled robots, and can photograph the working space for the multiple self-propelled robots to work, and the control unit is configured to: be communicatively connected to the second camera when a predetermined starting condition is met; control the movement of a self-propelled robot equipped with the control unit among the multiple self-propelled robots based on an image obtained by the second camera, so that the self-propelled robot performs the work in the working space; and cut off the connection with the second camera when a predetermined ending condition is met, and the control unit selects the camera to which the control unit is communicatively connected before performing the work based on information associating the first camera or the second camera with the work.

[0130] By adopting such a configuration, the operation of the robot during operation can be controlled by acquiring an image from a camera that can provide an image with a suitable angle and imaging range, thereby controlling the operation of the robot during operation.

[0131] (8) According to another embodiment of the present disclosure, there is provided a self-propelled robot for performing work in a work space. The self-propelled robot is communicatively connected to a first camera installed at a position different from the self-propelled robot and photographing the work space when a predetermined start condition is satisfied; performs the work based on the image acquired by the first camera; and disconnects the connection with the first camera when a predetermined end condition is satisfied.

[0132] (9) According to another aspect of the present disclosure, a control method for a self-propelled robot that performs work in a work space is provided. The control method for the self-propelled robot includes: a process of communicatively connecting to a first camera that is mounted at a position different from the self-propelled robot and that photographs the work space when a predetermined start condition is satisfied; a process of allowing the work to be performed based on an image acquired by the first camera; and a process of disconnecting the connection with the first camera when a predetermined end condition is satisfied.

[0133] The present disclosure can also be implemented in various ways other than the robot system, the self-propelled robot, and the control method of the self-propelled robot. For example, it can be implemented in ways such as a robot, a method for manufacturing a robot, a method for controlling a robot, a computer program for implementing these control methods, and a non-temporary storage medium storing the computer program.

Claims

1. A robot system, characterized in that: have: A plurality of self-propelled robots, each of which has a control unit; and a first camera, which is independently mounted on the plurality of self-propelled robots and is capable of photographing a working space in which the plurality of self-propelled robots operate; The control unit is configured to: When a predetermined starting condition is satisfied, connecting to the first camera in a communicable manner; controlling the movement of a self-propelled robot provided with the control unit among the plurality of self-propelled robots based on the image acquired by the first camera, so that the self-propelled robot performs the work in the work space; as well as When a predetermined termination condition is satisfied, the connection with the first camera is disconnected.

2. The robot system according to claim 1, wherein: The start condition is that the work performed by the self-propelled robot including the control unit in the work space starts.

3. The robot system according to claim 1, wherein: The termination condition is that the work performed by the self-propelled robot including the control unit in the work space is terminated.

4. The robot system according to claim 1, wherein: The first camera is configured to be capable of changing at least one of a photographing angle and a photographing range by being controlled by any one of the control units of the plurality of self-propelled robots. The control unit acquires control authority for controlling the first camera before executing the task.

5. The robot system according to claim 4, wherein: The first phase is controlled by only one control unit having the control authority among the control units of the plurality of self-propelled robots. The control unit relinquishes the control right after executing the job.

6. The robot system according to claim 4, wherein: The first camera, while being controlled by one of the control units of the plurality of self-propelled robots, can provide images to the other control units of the plurality of self-propelled robots.

7. The robot system according to claim 1, wherein: The robot system includes a second camera, which is independently mounted on the plurality of self-propelled robots and is capable of photographing a work space in which the plurality of self-propelled robots perform work. The control unit is configured to: When a predetermined starting condition is satisfied, connecting to the second camera in a communicable manner; controlling the movement of a self-propelled robot provided with the control unit among the plurality of self-propelled robots based on the image acquired by the second camera, so that the self-propelled robot performs the work in the work space; as well as When a predetermined termination condition is met, disconnecting the connection with the second camera; The control unit Based on the information associating the first camera or the second camera with the job, a camera to which the control section is communicably connected is selected before executing the job.

8. A self-propelled robot, characterized in that: Working in the work space, The self-propelled robot When a predetermined start condition is satisfied, the robot is communicatively connected to a first camera installed at a position different from the self-propelled robot and capturing an image of a work space. Execute the operation based on the image acquired by the first camera; as well as When a predetermined termination condition is satisfied, the connection with the first camera is disconnected.

9. A control method for a self-propelled robot, characterized in that: The self-propelled robot performs operations in a working space. The control method comprises: A step of connecting to a first camera that is mounted at a position different from the self-propelled robot and captures an image of a work space in a communicable manner when a predetermined start condition is satisfied; a process of causing the work to be performed based on the image acquired by the first camera; and The step of disconnecting the connection with the first camera when a predetermined termination condition is satisfied.