Robot control system and control device

Abstract

To provide a technique for reducing defects that a photographed image of a portion to be imaged cannot be obtained.SOLUTION: A robot control system 1 comprises: a robot 200; at least one camera 300 mounted on or fixed to the robot; a second camera 400 that is mounted on an operator; and a control device 100. The control device controls the robot by identifying a work-piece W or a position of a component or a skeleton posture or a state of the operator, on the basis of images photographed by the first camera and images photographed by the second camera.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technology for a control system of various robots, and more particularly to a technology for controlling a robot based on images captured by a camera. [Background technology]

[0002] A technique for controlling a robot based on an image captured by a camera has been known for some time. For example, Japanese Patent Application Laid-Open No. 2018-58178 (Patent Document 1) discloses a work assistance system equipped with a machine learning unit. According to Patent Document 1, a work assistance system is provided that includes a sensor unit that detects the position and orientation of at least one part of a worker's body, a supply unit that supplies parts or tools to the worker, and a cell controller that controls the supply unit, and the cell controller includes a machine learning unit that learns the work state of the worker based on the detected position and orientation and constructs a model, and a work state determination unit that determines the work state of the worker using the constructed model, and the supply unit selects a part or tool based on the determined work state, and changes the position and orientation of the part or tool based on the position and orientation of at least one part. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-58178 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a technique for reducing the problem of not being able to capture an image of a desired location. [Means for solving the problem]

[0005] According to one aspect of the present invention, there is provided a robot control system including a robot, at least one first camera attached or fixed to the robot, a second camera attached to a worker, and a control device. The control device determines the work situation by identifying the positions, postures, and conditions of the workpiece and parts and the skeletal posture and condition of the worker based on the images from the first and second cameras, and controls the robot according to the situation. Effect of the Invention

[0006] As described above, according to the present invention, it is possible to reduce the problem that a photographed image of a necessary portion cannot be acquired. [Brief description of the drawings]

[0007] [Figure 1] 1 is a block diagram showing an overall configuration of a robot control system according to a first embodiment. [Diagram 2] FIG. 2 is a conceptual diagram showing photography by a robot camera and a wearable camera according to the first embodiment. [Diagram 3] 5 is a flowchart showing information processing for robot control according to the first embodiment. [Figure 4] 13 is a flowchart showing a merge process according to the first embodiment; [Diagram 5] FIG. 2 is a conceptual diagram showing a merge processing method according to the first embodiment. [Figure 6] 10 is a flowchart showing information processing for robot control according to a second embodiment. [Figure 7] 13 is a flowchart showing information processing for robot control according to a third embodiment. [Figure 8] 13 is a flowchart showing information processing for robot control according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated. <First embodiment> <Overall configuration of the robot control system>

[0009] First, the overall configuration of a robot control system 1 according to the present embodiment will be described with reference to Fig. 1. The robot control system 1 includes, as main devices, a robot 200, a first camera 300, a second camera 400, and a control device 100 for controlling the operation of the robot 200 based on the captured images.

[0010] The robot control system 1 according to this embodiment is applied to, for example, a production site in a factory, and is configured to cause the robot 200 to execute a predetermined task at the production site. Moreover, in the robot control system 1 according to this embodiment, the robot 200 is not separated by a fence or the like, humans can access the working area of ​​the robot 200, and humans and the robot 200 work together to advance the work.

[0011] 2, the first camera 300 may be a camera attached to the robot 200 or a camera fixed to a workbench or a ceiling. The first camera 300 captures an image of an object placed on the workbench, i.e., a workpiece W, a part, and / or a worker working there.

[0012] 2, the second camera 400 is attached to the worker's body, work clothes, glasses, hat, helmet, etc., and is a so-called wearable camera. It is preferable that the workpiece W, parts, and / or parts of the worker's own body, such as the hands and arms, are within the shooting range of the second camera 400 while the worker is working.

[0013] Returning to Fig. 1, the control device 100 causes the robot 200 to execute various tasks based on images captured by the first camera 300 and the second camera 400. A task may be, for example, a process of moving a workpiece from one location to another location, or a process of handing over a tool appropriate for the workpiece W to a worker.

[0014] The control device 100 mainly includes a CPU 110, a memory 120, a display 130, an operation unit 140, a speaker 150, and a communication interface 160. The CPU 110 controls the robot 200 and each unit of the control device 100 by executing a program stored in the memory 120. For example, the CPU 110 executes a program stored in the memory 120 and performs various information processing, which will be described later, by referring to various types of data.

[0015] The memory 120 is realized by various RAMs, various ROMs, etc. The memory 120 stores programs such as tasks of the robot 200 executed by the CPU 110, and data generated by the execution of the programs by the CPU 110, such as the operating state, current position, posture, and target position of the robot 200.

[0016] The display 130 displays text and images based on signals from the CPU 110 .

[0017] The operation unit 140 receives instructions from the user and inputs them to the CPU 110 .

[0018] The speaker 150 outputs various sounds based on signals from the CPU 110 .

[0019] The communication interface 160 is realized by a connector, an antenna, etc., and exchanges various data with other devices such as the robot 200, the first camera 300, and the second camera 400 via a communication cable, a wireless LAN, etc. In this embodiment, the CPU 110 acquires an image captured by the first camera 300 using a communication cable via a connector serving as the communication interface 160. The CPU 110 acquires an image captured by the second camera 400 via a wireless antenna serving as the communication interface 160.

[0020] In this embodiment, the second camera 400 is equipped with a wireless antenna 460 and a battery 450. The first camera 300 has connectors for a signal cable and a power cable, and obtains power from the robot 200 and from a power cable.

[0021] In this way, the CPU 110 of the control device 100 causes the robot 200 to perform various operations via the communication interface 160 based on images acquired from the first camera 300 and the second camera 400 via the communication interface 160 in accordance with the robot control program in the memory 120. <Operational Overview of Robot Control System>

[0022] The control device 100 determines the working status of the worker in the work space and the status of the workpieces and parts on the workbench based on the image captured by the first camera 300, and controls the robot 200, but the worker's body or head may get in the way and the first camera 300 may not be able to capture the worker's hands or all of the workpieces W and parts on the workbench well. Therefore, in this embodiment, the image captured by the second camera 400 worn by the worker is used to complement information that cannot be obtained from the image captured by the first camera 300, making it possible to grasp the exact status of the workpieces. <Information processing by the control device 100>

[0023] The information processing of the control device 100 in this embodiment will be described in detail below with reference to Figures 3, 4, and 5. The CPU 110 of the control device 100 reads out, for example, a program for causing the robot 200 to execute a task in accordance with the program in the memory 120, and executes the following processes.

[0024] First, CPU 110 starts capturing an image with first camera 300 (step S110). Fig. 5(A) shows an image captured by first camera 300 attached to the ceiling.

[0025] CPU 110 starts capturing an image with second camera 400 (step S120).

[0026] CPU 110 can analyze the status of the work space from the images captured by first camera 300 and second camera 400, and by merging the analysis results, it is possible to obtain analysis results with less loss of information from multiple viewpoints (step S130). This allows CPU 110 to recognize information necessary for executing a task, such as the positions and postures of workpiece components, and / or the position and posture of the workpiece itself, and / or the posture of the worker.

[0027] The merge process (step S130) according to this embodiment will now be described with reference to FIG.

[0028] CPU 110 acquires where in the world coordinate system the position of first camera 300 is set (step S131). In the case of a camera fixed to a workbench, the position and orientation of the camera in the world coordinate system are determined in advance. In the case of a camera attached to robot 200, CPU 110 calculates the position, orientation, angle of view, etc. of first camera 300 in the world coordinate system based on the position and orientation of robot 200 in the world coordinate system and the position and orientation of first camera 300 relative to robot 200.

[0029] CPU 110 calculates a first transformation matrix from the coordinates seen from first camera 300 to the world coordinate system, based on information such as the camera's angle of view in the world coordinate system (step S132).

[0030] As shown in FIG. 5(B), based on the image of the first camera 300, the CPU 110 detects the workpiece W or parts, detects the skeletal posture of the worker, and so on from the two-dimensional image from the viewpoint of the first camera 300 (step S133).

[0031] For example, when using a three-dimensional camera such as an RGB-D camera, the CPU 110 adds depth information to the two-dimensional coordinates obtained above, thereby obtaining three-dimensional coordinates of the workpiece W, parts, and skeletal posture of the worker as the coordinates of the first camera 300.

[0032] When a two-dimensional camera is used, the CPU 110 can detect the same point using multiple first cameras 300, 300, etc., and can obtain the three-dimensional coordinates of the workpiece W, parts, and skeletal posture of the worker by triangulation or the like.

[0033] The CPU 110 uses the first transformation matrix to transform the coordinates of the workpiece W, parts, and skeleton posture of the worker from the viewpoint of the first camera 300 into world coordinates (step S134).

[0034] For example, as shown in FIG. 5C, the CPU 110 recognizes the AR marker M from the image captured by the first camera 300 (step S136).

[0035] The CPU 110 calculates the position and orientation of the first camera 300 based on the position and shape of the AR marker M as seen from the first camera 300 (step S137).

[0036] Then, the CPU 110 recognizes the AR marker M from the image captured by the second camera 400, as shown in FIG. 5(D) (step S138).

[0037] The CPU 110 calculates the position and orientation of the second camera 400 relative to the AR marker based on the position and shape of the AR marker M as seen from the second camera 400 (step S139). Note that the present invention is not limited to a form in which an AR marker is used, and the position and orientation of the second camera 400 relative to the reference may be calculated via the outer frame or parts of a workbench.

[0038] The CPU 110 calculates a second transformation matrix from the coordinates seen by the second camera 400 to the world coordinates (step S140). More specifically, the CPU 110 calculates the second transformation matrix from the first transformation matrix based on the position and orientation of the second camera 400 relative to the position and orientation of the first camera 300.

[0039] As shown in FIG. 5(D), based on the image of the second camera 400, the CPU 110 detects the workpiece W or parts, detects the skeletal posture of the worker, and so on from the two-dimensional image from the viewpoint of the second camera 400 (step S141).

[0040] For example, when an RGB-D camera is used, the CPU 110 adds depth information to the two-dimensional coordinates obtained above to obtain three-dimensional coordinates of the workpiece W, parts, and the skeletal posture of the worker as the coordinates of the second camera 400.

[0041] When a two-dimensional camera is used, the CPU 110 can detect the same point using multiple second cameras 400, 400, etc., and can obtain the three-dimensional coordinates of the workpiece W, parts, and skeletal posture of the worker by triangulation or the like.

[0042] CPU 110 uses the second transformation matrix to transform the coordinates of each detection result from the viewpoint of second camera 400 into world coordinates (step S142).

[0043] Based on the data on the position of the workpiece W or part or the skeletal posture or state of the worker acquired from the image from the first camera 300, and the data on the position of the workpiece W or part or the skeletal posture or state of the worker acquired from the image from the second camera 400, the CPU 110 creates three-dimensional data indicating all of these positions, postures, and states (step S143). Note that when different positions, postures, and states of the same part are obtained from the image from the first camera 300 and the image from the second camera 400, it is preferable to give priority to the position, posture, and state obtained from the image from the first camera 300 that is closer to the workpiece W, part, or worker's hand, which is more important in understanding the situation.

[0044] Returning to FIG. 3, the CPU 110 determines the next process for the workpiece W or part based on the position of the workpiece W or part or the data on the skeletal posture or state of the worker, in accordance with the control data for various tasks (step S150).

[0045] The CPU 110 transmits a control command to the robot 200 via the communication interface 160 (step S152).

[0046] Note that the process of step S136 is unnecessary if the first camera 300 has been calibrated (transformation matrix calculation) in advance and the position and orientation of the camera do not change. Even when the first camera 300 is attached to a robot 200 or the like and moves, if the position and orientation of the first camera 300 can be accurately calculated, the transformation matrix acquired by calibration can be corrected according to the position and orientation of the first camera 300, and thus the process of step S136 is also unnecessary.

[0047] Furthermore, in the second camera 400, if the transformation matrix has been calculated even once, even if the position and orientation of the first camera 300 changes thereafter, it is possible to estimate the position and orientation of the first camera 300 by analyzing an acceleration sensor or image change, and correct the already acquired transformation matrix. Therefore, when the AR marker can be detected, the transformation matrix may be calculated in step S138, and when the AR marker cannot be detected, the already acquired transformation matrix may be corrected by self-position estimation instead of the process of step S138. <Second embodiment>

[0048] In the above embodiment, the data from the first camera 300 and the data from the second camera 400 are always merged. In the present embodiment, when the position and posture of the second camera 400 are stable, the data from the first camera 300 and the data from the second camera 400 are used. Conversely, when the second camera 400 is moving or vibrating, the reliability of the data obtained using both images may be low, so the position of the workpiece W or the part, or the skeletal posture or state of the worker is determined based only on the image of the first camera 300.

[0049] In this embodiment, the second camera is equipped with an acceleration sensor 470, a vibration sensor, and the like.

[0050] As shown in FIG. 6, when the second camera 400 is vibrating or when the position or posture of the second camera 400 is changing (YES in step S212), the CPU 110 uses the acceleration sensor 470 to identify the position of the workpiece W or part, or the skeletal posture or condition of the worker, based on images from one or more first cameras 300, 300, ... (steps S131 to S134).

[0051] If the vibration of second camera 400 is small and the change in position and attitude is also small (NO in step S212), CPU 110 executes the process from step S120.

[0052] However, regardless of the form in which acceleration sensor 470 is used, CPU 110 may execute processing from step S131 when there is a large change in the image captured by second camera 400 (YES in step S212), and execute processing from step S120 when there is a small change in the image captured by second camera 400 (NO in step S212).

[0053] Alternatively, if the change in the first camera 300 is greater than that in the second camera 400 (YES in step S212), the CPU 110 may execute processing from step S120, and if the change in the first camera 300 is less than that in the second camera 400 (NO in step S212), the CPU 110 may execute processing from step S131.

[0054] In this embodiment, if there is a high possibility that the accuracy of the second camera 400 is low (YES in step S212), the CPU 110 does not start up the second camera 400, does not allow it to take pictures, or does not allow it to transmit images, thereby reducing consumption of the battery 450. <Third embodiment>

[0055] Alternatively, referring to FIG. 7, when the area of ​​the image captured by the first camera 300 that shows parts of the worker's head, body, etc. is larger than a predetermined ratio and the area of ​​the image captured by the first camera 300 that shows the hands, arms, etc. is small (YES in step S312), CPU 110 may cause the second camera 400 to capture an image or merge the data from the first camera 300 and the data from the second camera 400.

[0056] Alternatively, instead of or in addition to determining the vibration, the CPU 110 may cause the second camera 400 to take an image when the second camera 400 is located in a predetermined position, such as between the position of the workbench and the first camera 300, or may merge the detection results of the first camera 300 and the second camera 400.

[0057] In other words, when the position of the second camera 400 is lower than the position of the first camera 300, the CPU 110 may have the second camera 400 take an image, or may merge the detection results of the first camera 300 and the detection results of the second camera 400, and may not use the detection results of the second camera 400 when the position of the second camera 400 is higher than the position of the first camera 300.

[0058] Alternatively, when the orientation of the second camera 400 is in a predetermined direction, for example when the second camera 400 is facing downward, the CPU 110 may cause the second camera 400 to take an image, or may merge the detection results of the image from the first camera 300 and the image from the second camera 400.

[0059] In addition, the CPU 110 may decide whether or not to merge the detection results of the first camera 300 and the second camera 400 based on the coverage range of the target area (hidden, cut off), the reliability of the self-position estimation of the second camera 400 (such as a large amount of change in position and orientation since the marker was last detected), and image quality (blur, over / under exposure), etc. <Fourth embodiment>

[0060] In the above embodiment, the data from the first camera 300 and the data from the second camera 400 are merged. However, when a predetermined condition is satisfied, data on the position of the workpiece W or a part, or the skeletal posture or state of the worker may be acquired by the second camera 400, and when the predetermined condition is not satisfied, data on the position of the workpiece W or a part, or the skeletal posture or state of the worker may be acquired by the first camera 300.

[0061] For example, referring to FIG. 8, CPU 110 starts shooting with first camera 300 (step S110).

[0062] Based on the image from the first camera 300, the CPU 110 determines whether the area showing the worker's head, body, and other parts of the body in the image captured by the first camera 300 is larger than a predetermined ratio, or whether the area showing the hands, arms, and other parts of the body is small (step S312).

[0063] If the area showing the worker's head, body, and other parts is smaller than a predetermined ratio, or if the area showing the hands, arms, and other parts is large (NO in step S312), CPU 110 acquires the position of first camera 300 set in the world coordinate system (step S131). In the case of a camera fixed to a workbench, the position and orientation of the camera in the world coordinate system are determined in advance. In the case of a camera attached to robot 200, CPU 110 calculates the position, orientation, and angle of view of first camera 300 in the world coordinate system based on the position and orientation of robot 200 in the world coordinate system and the position and orientation of first camera 300 relative to robot 200.

[0064] CPU 110 calculates a first transformation matrix from the coordinates seen from first camera 300 to the world coordinate system, based on information such as the camera's angle of view in the world coordinate system (step S132).

[0065] Based on the image from the first camera 300, the CPU 110 detects the workpiece W or parts, detects the skeletal posture of the worker, and so on from the two-dimensional image from the viewpoint of the first camera 300 (step S133).

[0066] For example, when a three-dimensional camera is used, the CPU 110 adds depth information to the two-dimensional coordinates obtained above, thereby obtaining three-dimensional coordinates of the workpiece W, parts, and skeletal posture of the worker as the coordinates of the first camera 300.

[0067] When a two-dimensional camera is used, the CPU 110 can detect the same point using multiple first cameras 300, 300, etc., and can obtain the three-dimensional coordinates of the workpiece W, parts, and skeletal posture of the worker by triangulation or the like.

[0068] The CPU 110 uses the first transformation matrix to transform the coordinates of the workpiece W, parts, and skeleton posture of the worker from the viewpoint of the first camera 300 into world coordinates (step S134).

[0069] The CPU 110 determines the next process for the workpiece W or the part, based on the state of the workpiece W or the part and the skeletal posture and state of the worker, in accordance with the control data for various tasks (step S150).

[0070] The CPU 110 transmits a control command to the robot 200 via the communication interface 160 (step S152).

[0071] On the other hand, if the area covering the worker's head, body, etc. is larger than a predetermined ratio and the area covering the hands, arms, etc. is small (YES in step S312), CPU 110 determines whether second camera 400 is vibrating or its position or posture is changing (step S212).

[0072] If second camera 400 has vibrated or its position or orientation has changed (YES in step S212), the process goes on to step S131.

[0073] If the second camera 400 is stable (NO in step S212), the CPU 110 recognizes the AR marker M from the image from the first camera 300 (step S136). Note that if calibration (calculation of the transformation matrix) has been performed in advance for the first camera 300, the processes of steps S136 and S138 are not necessary.

[0074] The CPU 110 calculates the position and orientation of the first camera 300 based on the position and shape of the AR marker M as seen from the first camera 300 (step S137).

[0075] Then, the CPU 110 recognizes the AR marker M from the image captured by the second camera 400 (step S138).

[0076] The CPU 110 calculates the position and orientation of the second camera 400 based on the position and shape of the AR marker M as seen from the second camera 400 (step S139).

[0077] CPU 110 calculates a second transformation matrix from the coordinates seen by second camera 400 to world coordinates (step S140). More specifically, in this embodiment, CPU 110 calculates a first transformation matrix by executing steps S131 and S132, and then calculates a second transformation matrix from the first transformation matrix based on the position and orientation of second camera 400 relative to the position and orientation of first camera 300.

[0078] Based on the image from the second camera 400, the CPU 110 detects the workpiece W or parts, detects the skeletal posture of the worker, and so on from the two-dimensional image from the viewpoint of the second camera 400 (step S141).

[0079] For example, when a three-dimensional camera is used, the CPU 110 adds depth information to the two-dimensional coordinates obtained above to obtain three-dimensional coordinates of the workpiece W, parts, and the skeletal posture of the worker as the coordinates of the second camera 400.

[0080] When a two-dimensional camera is used, the CPU 110 can detect the same point using multiple second cameras 400, 400, etc., and can obtain the three-dimensional coordinates of the workpiece W, parts, and skeletal posture of the worker by triangulation or the like.

[0081] The CPU 110 uses the second transformation matrix to transform the coordinates of the workpiece W, parts, and skeleton posture of the worker from the viewpoint of the second camera 400 into world coordinates (step S142).

[0082] CPU 110 executes the process from step S150.

[0083] In this embodiment as well, in addition to using acceleration sensor 470 to determine the presence or absence of vibration, CPU 110 may execute processing from step S131 when there is a large change in the image captured by second camera 400 (YES in step S212), and execute processing from step S120 when there is a small change in the image captured by second camera 400 (NO in step S212).

[0084] Alternatively, if the change in the first camera 300 is greater than that in the second camera 400 (YES in step S212), the CPU 110 may execute processing from step S120, and if the change in the first camera 300 is less than that in the second camera 400 (NO in step S212), the CPU 110 may execute processing from step S131.

[0085] Alternatively, instead of or in addition to the above determination, the CPU 110 may execute processing from step S120 if the position of the second camera 400 is in a predetermined position, such as between the position of the workbench and the first camera 300, and execute processing from step S131 if the position of the second camera 400 is not in a predetermined position, such as between the position of the workbench and the first camera 300.

[0086] In other words, if the position of second camera 400 is lower than the position of first camera 300, CPU 110 may execute the process from step S120, and if not, execute the process from step S131.

[0087] Alternatively, CPU 110 may execute processing from step S120 when the orientation of second camera 400 is in a predetermined direction, for example when second camera 400 is facing downward, and may execute processing from step S131 if the orientation of second camera 400 is not in a predetermined direction.

[0088] In addition, CPU 110 may determine whether to execute processing from step S120 or processing from step S131 based on the coverage range of the target area (hidden, cut off), the reliability of the self-position estimation of second camera 400 (such as a large amount of change in position or attitude since the marker was last detected), and image quality (blur, over / under exposure), etc. Sixth embodiment

[0089] Some or all of the roles of the devices such as the control device 100 and the robot 200 in the robot control system 1 of the above embodiment may be performed by other devices. For example, some of the roles of the control device 100 may be performed by the robot 200, the role of the control device 100 may be performed by multiple personal computers, or the information processing of the control device 100 may be performed by a server on a cloud. <Summary>

[0090] In the above embodiment, a robot control system is provided that includes a robot, at least one first camera attached or fixed to the robot, a second camera attached to a worker, and a control device. The control device controls the robot by identifying the position of a workpiece or part or the skeletal posture or state of the worker based on an image from the first camera and an image from the second camera.

[0091] Preferably, the control device specifies a position of a workpiece or part or a skeletal posture or state of a worker based on an image of the at least one first camera and an image of the second camera when a predetermined condition is satisfied, and specifies a position of a workpiece or part or a skeletal posture or state of a worker based on an image of the at least one first camera when the predetermined condition is not satisfied.

[0092] Preferably, the second camera includes an acceleration sensor. When the change in position or attitude of the second camera is smaller than a predetermined degree as the predetermined condition, the control device identifies a position of a workpiece or part or a skeletal attitude or state of a worker based on at least one image of the first camera and an image of the second camera, and when the movement of the second camera is larger than a predetermined degree, the control device identifies a position of a workpiece or part or a skeletal attitude or state of a worker based on at least one image of the first camera.

[0093] Preferably, the control device determines the position of the work or part or the skeletal posture or condition of the worker based on an image from at least one first camera and an image from the second camera when the specified condition is such that part of the work object cannot be captured by at least one first camera due to the worker's body or head, and determines the position of the work or part or the skeletal posture or condition of the worker based on an image from the at least one first camera when the entire work object can be captured by the at least one first camera.

[0094] In the above embodiment, a control device is provided that includes a communication interface between a first camera, a second camera, and a robot, a memory, and a processor. The processor controls the robot by identifying the position of a workpiece or part or the skeletal posture or state of a worker based on an image from the first camera and an image from the second camera.

[0095] In the above embodiment, a robot control system is provided that includes a robot, a fixed camera or at least one first camera attached to the robot, a second camera attached to a worker, and a control device. The control device controls the robot by identifying the position of a workpiece or part or the skeletal posture or state of the worker based on an image from the second camera when a predetermined condition is satisfied, and controls the robot by identifying the position of a workpiece or part or the skeletal posture or state of the worker based on an image from the at least one first camera when the predetermined condition is not satisfied.

[0096] In the above embodiment, a robot control system is provided that includes a communication interface between a first camera, a second camera, and a robot, a memory, and a processor. The processor controls the robot by identifying the position of a workpiece or part or the skeletal posture or state of a worker based on an image from the second camera when a predetermined condition is satisfied, and controls the robot by identifying the position of a workpiece or part or the skeletal posture or state of a worker based on an image from at least one first camera when the predetermined condition is satisfied.

[0097] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not the above description, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0098] 1: Robot control system 100: Control device 110: CPU 120:Memory 130: Display 140:Operation unit 150: Speaker 160: Communication interface 200: Robot 300: 1st camera 400: Second camera 450:Battery 460: Radio antenna 470: Acceleration sensor

Claims

1. Robots and At least one first camera attached or fixed to the robot; a second camera attached to the worker; and a control device; The control device includes: When the change in the position or posture of the second camera is smaller than a predetermined level, the robot is controlled by identifying the position of a workpiece or part or the skeletal posture or state of a worker based on the image of the at least one first camera and the image of the second camera; A robot control system that controls the robot by identifying a position of a workpiece or part or a skeletal posture or state of a worker based on an image of the at least one first camera when movement of the second camera is greater than a predetermined degree.

2. Robots and At least one first camera attached or fixed to the robot; a second camera attached to the worker; and a control device; The control device includes: When a part of a work target cannot be photographed by the at least one first camera due to the body of the worker, the robot is controlled by identifying a position of a work or part or a skeletal posture or state of the worker based on an image of the at least one first camera and an image of the second camera; A robot control system that controls the robot by identifying a position of a workpiece or part or a skeletal posture or state of a worker based on an image from the at least one first camera when all of the work objects can be captured by the at least one first camera.

3. a communication interface for communicating with the first camera, the second camera, and the robot; Memory, a processor; The processor, When the change in the position or posture of the second camera is smaller than a predetermined level, the robot is controlled by identifying the position of a workpiece or part or the skeletal posture or state of a worker based on the image of the at least one first camera and the image of the second camera; A control device that controls the robot by identifying a position of a workpiece or part or a skeletal posture or state of a worker based on an image of the at least one first camera when movement of the second camera is greater than a predetermined degree.

4. a communication interface for communicating with the first camera, the second camera, and the robot; Memory, a processor; The processor, When a part of a work target cannot be photographed by the at least one first camera due to the body of the worker, the robot is controlled by identifying a position of a work or part or a skeletal posture or state of the worker based on an image of the at least one first camera and an image of the second camera; A control device that controls the robot by identifying the position of a workpiece or part or the skeletal posture or state of a worker based on an image from the at least one first camera when the entire work object can be captured by the at least one first camera.

5. Robots and At least one first camera, either a fixed camera or attached to the robot; a second camera attached to the worker; and a control device; The control device includes: When the change in the position or posture of the second camera is smaller than a predetermined level, the robot is controlled by identifying the position of a workpiece or part or the skeletal posture or state of a worker based on the image of the second camera; A robot control system that controls the robot by identifying a position of a workpiece or part or a skeletal posture or state of a worker based on an image of the at least one first camera when movement of the second camera is greater than a predetermined degree.

6. Robots and At least one first camera, either a fixed camera or attached to the robot; a second camera attached to the worker; and a control device; The control device includes: When a part of a work target cannot be photographed by the at least one first camera due to the body of the worker, the robot is controlled by identifying a position of a work or part or a skeletal posture or state of the worker based on an image of the second camera; A robot control system that controls the robot by identifying a position of a workpiece or part or a skeletal posture or state of a worker based on an image from the at least one first camera when all of the work objects can be captured by the at least one first camera.

7. a communication interface for communicating with the first camera, the second camera, and the robot; Memory, a processor; The processor, When the change in the position or posture of the second camera is smaller than a predetermined level, the robot is controlled by identifying the position of a workpiece or part or the skeletal posture or state of a worker based on the image of the second camera; A control device that controls the robot by identifying a position of a workpiece or part or a skeletal posture or state of a worker based on an image of the at least one first camera when movement of the second camera is greater than a predetermined degree.

8. a communication interface for communicating with the first camera, the second camera, and the robot; Memory, a processor; The processor, When a part of a work target cannot be photographed by the at least one first camera due to the worker's body, the robot is controlled by identifying a position of a work or part or a skeletal posture or state of the worker based on an image of the second camera; A control device that controls the robot by identifying the position of a workpiece or part or the skeletal posture or state of a worker based on an image from the at least one first camera when the entire work object can be captured by the at least one first camera.

Images