Sensing system, sensing method, and work robot

Abstract

To solve a problem of a conventional robot control method that it is not possible to check action of a work robot and adjust the action to proper action.SOLUTION: A sensing system includes: a learning part which refers to first information acquired by a first sensor to learn predetermined action; an action information generation part which refers to a learning result of the predetermined action by the learning part to generate action control information for providing an instruction for action to a work robot; and an adjustment part which compares first information with second information acquired by a second sensor to adjust the action control information so that robot action of the work robot becomes similar to the predetermined action.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a sensing system, a sensing method, and a working robot. [Background technology]

[0002] In recent years, technology has been considered for making a work robot perform work performed by humans, such as line work in a factory. For example, Patent Document 1 discloses a robot control method in which a two-dimensional code expressing work information is read by a sensor, an image of a work object is captured by a sensor that captures the image of the work object, operation information for making the robot perform the work is generated based on the captured image and the work information, and the robot is made to perform the work based on the operation information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-042867 Summary of the Invention [Problem to be solved by the invention]

[0004] In the robot control method of Patent Document 1, it is difficult to confirm whether the robot is operating according to the work content included in the work information, and it is also difficult to adjust the robot's operation to an appropriate operation when the work content is not performed appropriately.

[0005] Therefore, an object of the present invention is to provide a sensing system, a sensing method, and a working robot that can check the operation of a working robot and adjust the operation appropriately. [Means for solving the problem]

[0006] The sensing system of the present invention comprises a first sensor for sensing a specified movement of a sensing target, a work robot that operates in response to an operation instruction, a second sensor for sensing robot movement of the work robot, and a management control device capable of communicating with the first sensor, the second sensor, and the work robot, wherein the management control device has: a learning unit that learns the specified movement by referring to first information acquired by the first sensor; an operation information generation unit that generates operation control information for giving the operation instruction to the work robot by referring to a learning result of the specified movement by the learning unit; and an adjustment unit that compares the first information with second information acquired by the second sensor, and adjusts the operation control information so that the robot movement of the work robot approximates the specified movement.

[0007] Moreover, the sensing method of the present invention includes a first sensor for sensing a predetermined movement of a sensing object, a work robot that operates in response to an operation instruction, a second sensor for sensing robot movement of the work robot, and a management control device capable of communicating with the first sensor, the second sensor, and the work robot, wherein the management control device learns the predetermined movement by referring to first information acquired by the first sensor, generates movement control information for giving the operation instruction to the work robot by referring to a learning result of the predetermined movement by the learning unit, and compares the first information with second information acquired by the second sensor, and adjusts the movement control information so that the robot movement of the work robot approximates the predetermined movement.

[0008] Moreover, the working robot of the present invention is a work robot that operates in response to operational instructions, and is equipped with a first sensor for sensing a predetermined operation of a sensing target, a second sensor for sensing robot operation of the work robot, and an information processing unit capable of communicating with the first sensor and the second sensor, wherein the information processing unit has: a learning unit that learns the predetermined operation by referring to first information acquired by the first sensor; an operation information generation unit that generates operation control information for giving the operation instructions to the work robot by referring to a learning result of the predetermined operation by the learning unit; and an adjustment unit that compares the first information with second information acquired by the second sensor, and adjusts the operation control information so that the robot operation of the work robot approximates the predetermined operation. Effect of the Invention

[0009] According to the present invention, there are provided a sensing system, a sensing method, and a working robot that are capable of checking the operation of a working robot and adjusting the operation appropriately. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2A is a diagram showing an example of a system configuration in a sensing system according to a first embodiment of the present invention, and FIG. 2B is a diagram showing an example of the humanoid robot shown in FIG. [Diagram 2] 1 is a block diagram showing an example of the configuration and functions of a sensing system according to a first embodiment of the present invention. [Diagram 3] 2 is a block diagram showing an example of the functions of a management control device in the sensing system of the first embodiment of the present invention. FIG. [Figure 4] FIG. 2 is a diagram showing an example of each sensing period in the sensing system according to the first embodiment of the present invention. [Diagram 5] 4 is an example of a flowchart showing processing of the sensing system according to the first embodiment of the present invention. [Figure 6] 6 is an example of a flowchart showing more detailed processing of the operation control information generating processing shown in step S103 of FIG. 5. [Figure 7] 7 is an example of a flowchart showing more detailed processing of the action control information adjustment processing shown in step S106 of FIG. 5. [Figure 8] FIG. 2A is a diagram showing an example of a system configuration in a sensing system according to a first modified example of the first embodiment of the present invention, and FIG. 2B is a diagram showing an example of the humanoid robot shown in FIG. [Figure 9] FIG. 2 is a block diagram showing an example of functions of a humanoid robot in a sensing system according to a first modified example of the first embodiment of the present invention. [Figure 10] FIG. 11 is a diagram showing an example of a system configuration of a sensing system according to a second modification of the first embodiment of the present invention. [Figure 11] FIG. 13A is a diagram showing an example of a system configuration in a sensing system according to a third modification of the first embodiment of the present invention, and FIG. 13B is a diagram showing an example of a sensor mounting member shown in FIG. [Figure 12] FIG. 11 is a diagram showing an example of the configuration and functions of a sensing system according to a third modified example of the first embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The sensing system, the sensing method, and the mobile robot will be described below with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to the embodiments, but extends to the inventions described in the claims and their equivalents.

[0012] (Embodiment 1) FIG. 1 is a diagram for explaining the sensing system.

[0013] 1(a) is a diagram showing an example of a system configuration of a sensing system according to a first embodiment of the present invention. The sensing system includes a first humanoid robot 20a and a second humanoid robot 20b functioning as mobile robots, and a third humanoid robot 20c functioning as a working robot. Note that the number of mobile robots and humanoid robots functioning as working robots is not limited to this.

[0014] The first humanoid robot 20a moves to the vicinity of the worker 400 working on the work line 201 in the work area 200 by receiving an instruction from a management control device 60 (see FIG. 2) described later or by an instruction from a first information processing device 25a (see FIG. 2) provided in the first humanoid robot 20a. The sensing system senses a predetermined motion of the worker 400 by a first robot sensor 23a (first robot imaging device 24a) provided in the first humanoid robot 20a. The predetermined motion is wide-ranging, and includes, for example, assembling parts, moving parts, painting a product, and the movement of the worker himself. In addition, when sensing the worker 400, a known image recognition technology may be used, or the worker 400 and his predetermined motion may be recognized by learning by a learning unit 663 (see FIG. 3). The same applies to sensing of the working robot described later.

[0015] The sensing system learns a predetermined motion of the worker 400 by referring to the first information acquired by the first robot sensor 23a (first robot imaging device 24a) functioning as a first sensor. The sensing system also generates motion control information for giving motion instructions to the third humanoid robot 20c by referring to the learning result of the predetermined motion.

[0016] The second humanoid robot 20b moves to the vicinity of the third humanoid robot 20c working on the work line 201 in the workplace 200, in response to an instruction from the management control device 60 or an instruction from a second information processing device provided in the second humanoid robot 20b. Similarly, the third humanoid robot 20c moves to the vicinity of the worker 400 in the workplace 200, in response to an instruction from the management control device 60 or an instruction from a third information processing device provided in the third humanoid robot 20c.

[0017] The sensing system operates the third humanoid robot 20c by referring to the motion control information. The sensing system also senses the robot motion of the third humanoid robot 20c using the second robot sensor 23b (second robot imaging device 24b) provided in the second humanoid robot 20b. This allows the sensing system to confirm the robot motion of the third humanoid robot 20c functioning as a work robot.

[0018] The sensing system compares the first information with the second information acquired by the second robot sensor 23b (second robot imaging device 24b) functioning as the second sensor, and adjusts the motion control information so that the robot motion of the third humanoid robot 20c approximates a predetermined motion. This makes it possible to adjust the robot motion of the third humanoid robot 20c to an appropriate motion.

[0019] 1(b) is a diagram showing an example of the humanoid robot 20 shown in (a). The humanoid robot 20, which functions as a mobile robot and a working robot, includes a robot body 21, a robot movement mechanism 22, a robot sensor 23, a robot imaging device 24 included in the robot sensor 23, an information processing device 25, and a robot arm 26.

[0020] The humanoid robot 20 can move using a robot movement mechanism 22 provided below the robot body 21, and moves to the vicinity of the work line 201 in the workplace 200 upon receiving instructions from outside the humanoid robot 20, such as from a management control device 60, or by referring to a program stored in an information processing device 25.

[0021] The robot main body 21 includes a robot torso 211 and a robot head 212. The robot torso 211 and the robot head 212 constitute a torso / head drive mechanism, and are capable of changing the sensing area 230 (imaging area 240) of the robot sensor 23 (robot imaging device 24). The configuration of the drive mechanism is not particularly limited, and may be configured such that, for example, the robot head 212 rotates a predetermined angle relative to the robot torso 211, or the robot torso 211 rotates a predetermined angle relative to the robot movement mechanism 22, by a servo motor (not shown).

[0022] A robot moving mechanism 22 is provided below the robot body 211, a robot arm 26 is provided on each side of the robot body 211, and a robot sensor 23 is provided in the robot head 212. An information processing device 25 is provided inside the robot body 21.

[0023] The robot movement mechanism 22 may have any configuration, for example, it may be provided with a rotating body driven by a motor, or may have legs that resemble the shape of a human leg. As an example, when the robot movement mechanism 22 is configured to resemble the shape of a human leg, a servo motor is provided at the location corresponding to the human joint, and the movement mechanism is configured by rotating the servo motor by a predetermined angle.

[0024] The robot sensor 23, which functions as the first sensor and the second sensor, is preferably provided in the robot head 212 and senses the worker 400 and the working robot. The robot sensor 23 also sequentially acquires information indicating at least the distance and angle between the robot arm 26 and an object around the humanoid robot 20 on which the humanoid robot 20 works. As an example of the robot sensor 23, a highest performance camera, a thermal camera, a high pixel, telephoto, ultra-wide angle, 360 degree, high performance camera, a radar, a solid-state LiDAR, a LiDAR, a multi-color laser coaxial displacement meter, a vision recognition, or a group of various other sensors may be adopted. These are also examples of the robot imaging device 24. Other examples of the robot sensor 23 include a vibration meter, a hardness meter, a micro vibration meter, an ultrasonic measuring instrument, a vibration measuring instrument, an infrared measuring instrument, an ultraviolet measuring instrument, an electromagnetic wave measuring instrument, a thermometer, a hygrometer, a spot AI weather forecast, a high-precision multi-channel GPS, low altitude satellite information, or long tail incident AI data.

[0025] Examples of sensor information acquired from the robot sensor 23 include images, distance, vibration, heat, smell, color, sound, ultrasonic waves, radio waves, ultraviolet rays, infrared rays, humidity, etc., and the image and distance information is preferably acquired by the robot imaging device 24. The robot sensor 23 (robot imaging device 24) performs these sensing operations, for example, every nanosecond. The sensor information is used, for example, for motion capture of the movements of the worker 400, a 3D map of the workplace 200, navigation of the movement and movements of the worker 400 in the workplace 200, analysis of cornering, speed, etc.

[0026] The robot arm 26 includes a right arm 261 and a left arm 262. The right arm 261 includes a right gripping support part 263 and a right gripping part 265, and the left arm 262 includes a left gripping support part 264 and a left gripping part 266. The right gripping support part 263 is a mechanism for supporting the right gripping part 265, and the left gripping support part 264 is a mechanism for supporting the left gripping part 266, and may be shaped like a human arm, for example. The gripping parts 265 and 266 are mechanisms for gripping parts for work, for example, and may be shaped like a human hand, for example.

[0027] The robot arm 26 constitutes an arm drive mechanism. The configuration of the drive mechanism is not particularly limited, and for example, when the robot arm 26 is made to resemble a human shape, a configuration may be adopted in which servo motors are provided at each joint location, such as a location corresponding to a human shoulder, a location corresponding to an elbow, a location corresponding to a wrist, a location corresponding to a finger joint, and the like, and rotated by a predetermined angle.

[0028] The humanoid robot 20 may further be provided with a sensor, for example, on the robot body 211 (see FIG. 8(b)). In this case, the sensor is located at a different height from the robot sensor 23 located on the robot head 212. The different height allows the sensor to sense the movements of the worker 400 from different angles.

[0029] FIG. 2 is a block diagram showing an example of the configuration and functions of the sensing system 100 of this embodiment.

[0030] The sensing system 100 includes a first humanoid robot 20a, a second humanoid robot 20b, a third humanoid robot 20c, and a management control device 60. The first humanoid robot 20a, the second humanoid robot 20b, and the third humanoid robot 20c are each connected to a communication unit 64 of the management control device 60 via wireless or wired communication, and receive instructions from the management control device 60 and transmit information acquired by each sensor. The humanoid robots 20a to c may also be connected to each other via wireless or wired communication, and transmit and receive information acquired by each sensor and instructions.

[0031] The first humanoid robot 20a functioning as a mobile robot includes a first moving mechanism 22a, a first robot sensor 23a functioning as a first sensor, a first robot imaging device 24a included in the first robot sensor 23a, a first information processing device 25a, a first body / head driving mechanism 21a, and a first arm driving mechanism 26a. In this embodiment, the second humanoid robot 20b functioning as a mobile robot and the third humanoid robot 20c functioning as a working robot are also configured in the same manner as the first humanoid robot 20a.

[0032] The first information processing device 25a according to this embodiment includes a central processing unit (CPU) 1212, a random access memory (RAM) 1214, and a graphic controller 1216, which are connected to each other by a host controller 1210. The first information processing device 25a also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The first information processing device 25a also includes input / output units such as a read only memory (ROM) 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0033] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

[0034] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the first information processing device 25a. The storage device 1224 may also store first information and second information. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0035] The ROM 1230 stores therein a boot program, etc., executed by the first information processing device 25a upon activation, and / or a program that depends on the hardware of the first information processing device 25a. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0036] The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the first information processing device 25a, and brings about cooperation between the programs and the above-mentioned various types of hardware resources. An apparatus or method may be configured by realizing an operation or processing of information according to the use of the first information processing device 25a.

[0037] For example, when communication is performed between the first information processing device 25a and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0038] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0039] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may execute various types of processing on data read from the RAM 1214, including various types of operations, information processing, conditional decisions, conditional branches, unconditional branches, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write back the results to the RAM 1214. The CPU 1212 may also search for information in files, databases, etc., in the recording medium.

[0040] The above-described program or software module may be stored in a computer-readable storage medium on the first information processing device 25a or in the vicinity of the first information processing device 25a. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable storage medium, thereby providing the program to the first information processing device 25a via the network.

[0041] The blocks in the flowcharts and diagrams in this embodiment may represent stages of a process in which operations are performed or "parts" of an apparatus responsible for performing the operations. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer readable instructions stored on a computer readable storage medium, and / or a processor provided with computer readable instructions stored on a computer readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0042] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0043] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0044] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0045] The contents described so far also apply to the information processing devices provided in the second humanoid robot 20b and the third humanoid robot 20c.

[0046] The management control device 60 is a control device that gives instructions to the humanoid robots 20a to 20c in order to realize the sensing system 100. In addition, the management control device 60 acquires the sensor information (first information and second information) accumulated in the storage device 1224.

[0047] The management control device 60 is composed of a CPU 60A, a RAM 60B, a ROM 60C, an input / output unit (I / O) 60D, a bus 60E such as a data bus or a control bus connecting these, and a communication unit 68. A storage medium 62 is connected to the I / O 60D.

[0048] In addition, a communication unit 64 is connected to the I / O 60D, which transmits and receives sensor information, work manual information related to the predetermined actions of the worker 400, work schedule information, etc., between the control system of the humanoid robot 20. The work manual information includes, for example, the name and content of each work item, the order of the work items, and information on the standard work time required for each work item. In addition, the work schedule information includes, for example, information indicating the work time and start time / end time of the entire work, information indicating the work time and start time / end time of each work item, information indicating the worker for each work item, etc.

[0049] FIG. 3 is a block diagram showing an example of the functions of the management control device 60 in the sensing system of this embodiment.

[0050] The management control device 60 includes a storage medium 62 , a communication unit 64 , and a processing unit 66 .

[0051] The storage medium 62 includes, for example, at least one of a semiconductor storage device, a magnetic tape device, a magnetic disk device, and an optical disk device. The storage medium 62 stores a driver program, an operating system program, an application program, data, and the like used for processing in the processing unit 66. For example, the storage medium 62 stores first information and second information. The storage medium 62 also stores work manual information for the worker 400. The storage medium 62 may also store process chart information.

[0052] The communication unit 64 has a wireless communication interface circuit such as Wi-Fi (registered trademark) and / or a wired communication interface circuit such as Ethernet (registered trademark). The communication unit 64 transmits and receives various information to and from the humanoid robots 20a to 20c through the interface circuits.

[0053] The processing unit 66 has one or more processors and their peripheral circuits. The processing unit 66 generally controls the overall operation of the sensing system 100, and is, for example, a CPU. The processing unit 66 executes processing by referring to programs (driver programs, operating system programs, application programs, etc.) stored in the storage medium 62. The processing unit 66 can also execute multiple programs (application programs, etc.) in parallel.

[0054] The processing unit 66 includes a determination unit 661, an adjustment unit 662, a learning unit 663, and a motion information generation unit 664. Each of these units is a functional module realized by a program executed by a processor included in the processing unit 66. Alternatively, each of these units may be implemented in the processing unit 66 as firmware.

[0055] The determination unit 661 determines whether or not there is manual information related to the predetermined motion, and if there is work manual information, determines whether or not the learning result of the predetermined motion contradicts the work manual information.

[0056] The adjustment unit 662 compares the first information with the second information and adjusts the motion control information so that the robot motion of the third humanoid robot 20c approximates the predetermined motion. Preferably, the adjustment unit 662 adjusts the motion control information so that the robot motion approximates the predetermined motion during the overlap period. The overlap period is a period during which the first sensing period and the second sensing period overlap. Here, the first sensing period is a period during which the first robot sensor 23a (first robot imaging device 24a) functioning as the first sensor senses the worker 400 and acquires the first information. The second sensing period is a period during which the second robot sensor 23b (second robot imaging device 24b) functioning as the second sensor senses the robot motion of the third humanoid robot 20c and acquires the second information. That is, the overlap period is a period during which the first information and the second information are acquired simultaneously.

[0057] The learning unit 663 learns the predetermined motion of the worker 400 by referring to the first information stored in the storage medium 62 and / or the storage device 1224. This learning is performed by automatic learning, which is learning that automatically creates a trained model and automatically performs judgment / analysis using the trained model, for example.

[0058] The motion information generating unit 664 refers to the learning result of the predetermined motion of the worker 400 by the learning unit 663, and generates motion control information for giving motion instructions to the third humanoid robot 20c functioning as a work robot. The motion information generating unit 664 may refer to work manual information when generating the motion control information. This makes it possible to make the third humanoid robot 20c perform an appropriate motion (work) without reflecting any inappropriate predetermined motion of the worker 400.

[0059] 4 is a diagram showing an example of each sensing period in the sensing system 100. In the sensing system 100, the first sensor performs sensing and the second sensor performs sensing so that an overlap period occurs. Such sensing is performed according to an instruction from the management control device 60.

[0060] In this sensing system 100, the robot operation of the third humanoid robot 20c is started during sensing of the predetermined operation of the worker 400 (i.e., the first sensing period), and during sensing of the robot operation of the third humanoid robot 20c (i.e., the second sensing period), the adjustment unit 662 adjusts the operation control information so that the robot operation approximates the predetermined operation of the worker 400. This makes it possible to adjust the robot operation of the third humanoid robot 20c, which is a working robot, on the spot.

[0061] (Processing of the sensing system according to the first embodiment of the present invention) FIG. 5 is an example of a flowchart showing the processing of the sensing system of this embodiment.

[0062] First, in response to an instruction from the management control device 60 or an instruction to read out a program stored in the storage medium 62 or the storage device 1224, the information processing device of each humanoid robot 20 instructs multiple (three in this embodiment) humanoid robots 20 functioning as mobile robots and working robots to move to the workshop 200 (step S101). The movement is caused by the operation of the robot movement mechanism 22 of each humanoid robot 20.

[0063] During the movement, an instruction is given so that the sensing area 230a (imaging area 240a) of the first robot sensor 23a (first robot imaging device 24a) provided in the first humanoid robot 20a targets the worker 400, while the sensing area 230b (imaging area 240b) of the second robot sensor 23b (second robot imaging device 24b) provided in the second humanoid robot 20b targets the third humanoid robot 20c. The arrangement of such a plurality of humanoid robots 20 is performed, for example, by storing a floor plan of the workplace 200 in the storage device 1224 and / or the storage medium 62 in advance and associating the position of each humanoid robot 20 with the stored floor plan. Alternatively, the arrangement of the humanoid robot 20 may be based on a position optimized through machine learning.

[0064] Next, a predetermined movement of the worker 400 on the work line 201 is sensed by the first robot sensor 23a (first robot imaging device 24a) (step S102). In this embodiment, the processing unit 66 issues an instruction to the first sensor 23a (first imaging device 24a) so that the sensing area 230a (imaging area 240a) targets the predetermined movement of the worker 400, and in response, the first information processing device 25a operates the first moving mechanism 22a and the body / head driving mechanism 21a of the first humanoid robot 20a.

[0065] The first information acquired by the first robot sensor 23a (first robot imaging device 24a) is stored in the storage medium 62 via the storage device 1224 and / or the communication unit 64. The storage device 1224 and the storage medium 62 function as a storage unit.

[0066] The management control device 60 learns the predetermined motion by referring to the first information stored in the memory unit, in other words, stored, and generates motion control information for giving motion instructions to the third humanoid robot 20c functioning as a work robot by referring to the learning result (step S103). S103 is preferably performed during the first sensing period. This enables the sensing system 100 to operate the third humanoid robot 20c from the stage where the worker 400 is performing a predetermined motion for work.

[0067] The management control device 60 operates the third humanoid robot 20c by referring to the motion control information (step S104). The third humanoid robot 20c operates according to the motion instructions given by the motion control information.

[0068] Before and after the robot action of the third humanoid robot 20c based on the action control information (S104), the second robot sensor 23b (second robot imaging device 24b) senses the robot action of the third humanoid robot 20c (step S105). This allows the robot action of the third humanoid robot 20c to be confirmed.

[0069] In this embodiment, the processing unit 66 issues an instruction to set the sensing area 230b (imaging area 240b) of the second robot sensor 23b (second robot imaging device 24b) to target the robot movement of the third humanoid robot 20c, and in response to this, the second information processing device of the second humanoid robot 20b operates the second movement mechanism and the second body / head drive mechanism of the second humanoid robot 20b. The second information acquired by the second sensor 23b (second imaging device 24b) is stored in the storage unit.

[0070] The management control device 60 adjusts the motion control information so that the robot motion approximates the predetermined motion (step S106). Preferably, step S106 is performed during the overlap period. To achieve this, the management control device 60 simultaneously acquires the first information by the first sensor and the second information by the second sensor. This allows the sensing system 100 to adjust the robot motion of the third humanoid robot 20c so that it approximates the predetermined motion of the worker 400 from the stage where the worker 400 is performing the predetermined motion.

[0071] FIG. 6 is an example of a flowchart showing more detailed processing of the action control information generating processing shown in step S103 of FIG.

[0072] The first information is stored in the storage unit (step S201), and the learning unit 663 refers to the first information stored in the storage unit to learn the predetermined motion of the worker 400 (step S202). In the learning, motion capture of the motion of the worker 400, a 3D map of the workplace 200, navigation of the movement and motion of the worker 400 in the workplace 200, cornering, speed, and the like are analyzed, and optimal motion of the humanoid robot 20, which can also function as a working robot, is learned by automatic learning. This makes it possible to analyze the predetermined motion of the worker 400 from multiple perspectives at once, and reduces the time and cost required for analyzing and programming the motion of the worker 400.

[0073] Here, the determination unit 661 determines whether or not there is manual operation information related to the predetermined operation (steps S203, S204). If there is no manual operation information related to the predetermined operation (S204-NO), the operation information generation unit 664 refers to the learning result of the predetermined operation by the learning unit 663 (step S208) and generates operation control information for giving an operation instruction to the third humanoid robot 20c (step S209). Then, the processing unit 66 operates the third humanoid robot 20c by referring to the operation control information (step S210). This enables the third humanoid robot 20c to perform a robot operation corresponding to the work (predetermined operation) of the worker 400.

[0074] On the other hand, if there is manual motion information related to the predetermined motion (S204-YES), the determination unit 661 determines whether the learning result of the predetermined motion is inconsistent with the work manual information (steps S205 and S206). If the determination unit 661 determines that the learning result of the predetermined motion is inconsistent with the work manual information (S206-YES), the motion of the worker 400 may have been inappropriate for the content of the work item included in the work manual information. Therefore, if it is determined that the learning result of the predetermined motion is inconsistent with the work manual information, the motion information generation unit 664 does not adopt the learning result of the predetermined motion when generating motion control information (step S208). In this case, the motion information generation unit 664 refers to the work manual information (step S207) and generates motion control information (step S209). Then, the processing unit 66 operates the third humanoid robot 20c by referring to the motion control information (step S210). This makes it possible to prevent unnecessary or inappropriate predetermined actions by the worker 400 from being reflected in the action control information, and to cause the third humanoid robot 20c to perform appropriate actions.

[0075] FIG. 7 is an example of a flowchart showing more detailed processing of the action control information adjustment processing shown in step S106 of FIG.

[0076] The second information is stored in the storage unit (step S301), and the adjustment unit 662 compares the first information with the second information (step S302) and adjusts the motion control information so that the robot motion approximates the predetermined motion (step S303). That is, the robot motion of the third humanoid robot 20c is adjusted so as to approximate the predetermined motion of the worker 400.

[0077] (Operation and effect of the sensing system according to the first embodiment) According to the sensing system 100 of this embodiment, first information acquired by a first sensor that senses a predetermined movement of the worker 400 is compared with second information acquired by a second sensor that senses the work robot, and the movement control information is adjusted so that the robot movement of the work robot approximates the predetermined movement. This makes it possible to check the robot movement of the work robot while adjusting the movement of the work robot to an appropriate one.

[0078] As an example, in the case of a task in which the worker 400 repeatedly performs the same predetermined motion, the worker 400 first performs the first predetermined motion, and the sensing system 100 acquires the first information. Then, while the worker 400 is performing the second predetermined motion, the sensing system 100 generates motion control information and operates the working robot. Since the worker 400 is also performing the second predetermined motion during this, the sensing system 100 generates motion control information based on the first information from the second predetermined motion, and adjusts the motion control information so that the robot motion of the working robot approximates the predetermined motion of the worker 400. By repeating this, the sensing system 100 can adjust the robot motion so that the robot motion of the working robot approximates the predetermined motion of the worker 400.

[0079] Furthermore, according to the sensing system 100 of this embodiment, the management control device 60 acquires the first information and the second information simultaneously, making it possible to provide a system that can adjust the operation of the work robot on the spot while comparing the specified operations of the work robot and the worker 400.

[0080] Furthermore, according to the sensing system 100 of this embodiment, the operation control information is generated after referring to the work manual information. The worker 400 does not always perform the operations faithfully, and in some cases may perform unnecessary operations or omit necessary operations. Therefore, by referring to the work manual information, appropriate operation information can be reflected in the operation control information, making it possible to adjust the working robot to operate more appropriately.

[0081] Furthermore, according to the sensing system 100 of this embodiment, the learning result of the learning unit 663 on the predetermined motion that contradicts the work manual information is not adopted when generating the motion control information. This makes it possible to prevent unnecessary or inappropriate predetermined motions by the worker 400 from being reflected in the motion control information.

[0082] (Modification 1 of the first embodiment) FIG. 8 is a diagram illustrating an example of a sensing system according to the first modification of the present embodiment.

[0083] Fig. 8(a) is a diagram showing an example of a system configuration of a sensing system according to a first modification of the first embodiment of the present invention. This sensing system is characterized in that in a humanoid robot 20' functioning as a working robot, a head sensor 23' (head image capture device 24'') functions as a first sensor, and a torso sensor 23'' (torso image capture device 24'') functioning as a second sensor is provided in the humanoid robot 20'. Also, in this sensing system, the management control device 60 is not necessarily required, and the sensing system can be configured by the humanoid robot 20' alone.

[0084] FIG. 8(b) is a diagram showing an example of the working robot shown in FIG. 8(a). The humanoid robot 20' functioning as a working robot has the same configuration as the first humanoid robot 20a, except that it is equipped with a torso sensor 23'' (torso image capture device 24''). Specifically, the humanoid robot 20' includes a robot main body 21', a robot movement mechanism 22', a head sensor 23', a head image capture device 24' included in the head sensor 23', the torso sensor 23'', the torso image capture device 24'' included in the torso sensor 23'', an information processing device 25', and a robot arm 26'.

[0085] The robot main body 21' includes a robot torso 211' and a robot head 212'. The robot torso 211' and the robot head 212' configure a torso / head drive mechanism 21' (see FIG. 9), and are capable of changing a sensing area 230' (imaging area 240') of the head sensor 23' (head imaging device 24') and a sensing area 230'' (imaging area 240'') of the torso sensor 23'' (torso imaging device 24'').

[0086] The head sensor 23' (head imaging device 24') functions as a first sensor, and the torso sensor 23'' (torso imaging device 24'') functions as a second sensor. The torso sensor 23'' (torso imaging device 24'') senses, for example, the movement of the robot arm 26' as the robot movement. Since the head sensor 23' (head imaging device 24') and the torso sensor 23'' (torso imaging device 24'') are disposed at different height positions, the torso sensor 23'' (torso imaging device 24'') functioning as the second sensor senses a predetermined movement of the sensing target from a position different from that of the head sensor 23' (head imaging device 24'). Note that the roles of the head sensor 23' (head imaging device 24') and the torso sensor 23'' (torso imaging device 24'') may be reversed.

[0087] FIG. 9 is a block diagram showing an example of the functions of a working robot in the sensing system 100'. In the sensing system 100', the information processing device 25' includes an information processing unit 66', a communication interface 1222', and a storage device 1224', and the information processing unit 66' includes a determination unit 661', an adjustment unit 662', a learning unit 663', and an operation information generation unit 664'. That is, in the sensing system 100', the information processing unit 66' performs the same processing as the processing unit 66 of the management control device 60. The information processing device 25' is configured to be able to communicate with the head sensor 23' (head imaging device 24'), the body sensor 23'' (head imaging device 24''), the first moving mechanism 22', the head / body driving mechanism 21', and the arm driving mechanism 26'.

[0088] In the humanoid robot 20' of the sensing system 100', an information processing device 25' is provided with an information processing section 66', and therefore the humanoid robot 20' alone constitutes a sensing system.

[0089] 8, for example, an adjustment unit 662' of the humanoid robot 20' instructs the head sensor 23' (head image capture device 24') functioning as a first sensor to sense a predetermined movement of the worker 400, and the torso sensor 23'' (torso image capture device 24'') functioning as a second sensor to sense the arm 26' of the humanoid robot 20'. Then, from the sensor information (first information and second information) acquired by each sensor, movement control information is generated through learning by the learning unit 663', and the adjustment unit 662' compares the first information with the second information and adjusts the movement control information so that the robot movement of the humanoid robot 20' approximates the predetermined movement of the worker 400.

[0090] (Effects of Modification 1) According to this sensing system, the humanoid robot 20' can constitute a sensing system by itself, so that even in a place where communication with the management control device 60 is not possible, it is possible to check the robot operation of the work robot and adjust the robot operation to an appropriate operation.

[0091] In addition, since the self-acting robot 20' is equipped with multiple (two in this modified example) sensors (imaging devices), it is possible to check the robot operation of the work robot and adjust the robot operation to an appropriate operation, even in a place that is too narrow for sensing the worker 400, for example.

[0092] In this sensing system, the humanoid robot functioning as the work robot does not necessarily have to be one, but may be multiple. In this case, the more humanoid robots there are, the more humanoid robots will be performing tasks, and multiple tasks can be processed simultaneously in parallel at once.

[0093] (Modification 2 of the first embodiment) FIG. 10 is a diagram showing an example of a system configuration of a sensing system according to the second modification of this embodiment.

[0094] This sensing system is characterized in that the first humanoid robot 20a having the same function as the humanoid robot 20' described in the first modification senses the worker 400 and the third humanoid robot 20c functioning as a working robot. Specifically, an instruction is given so that the sensing area 230a1 (imaging area 240a1) of the head sensor 23a1 (head imaging device 24a1) of the first humanoid robot 20a targets the third humanoid robot 20c, and the sensing area 230a2 (imaging area 240a2) of the body sensor 23a2 (body imaging device 24a2) of the first humanoid robot 20a targets the worker 400. Note that, in this sensing system, the management control device 60 is not necessarily required as long as the first humanoid robot 20a and the third humanoid robot 20c are configured to be able to communicate with each other. Furthermore, the sensing areas of head sensor 23a1 (head imaging device 24a1) and body sensor 23a2 (body imaging device 24a2) may be configured inversely to that described above.

[0095] With this sensing system, the entire third humanoid robot 20c can be sensed, and therefore compared to variant example 1, it has the advantage that the robot movements of the third humanoid robot 20c can be more easily confirmed and appropriately controlled.

[0096] Note that, when there is no space to place the first humanoid robot 20a, the first modification also has the advantage that it is easier to configure a sensing system than the second modification. In addition, the first modification also has the advantage over the second modification in that it is not necessary to provide a communication configuration between the first humanoid robot 20a and the third humanoid robot 20c, and the humanoid robot can be omitted as a whole.

[0097] (Modification 3 of embodiment 1) FIG. 11 is a diagram illustrating an example of a sensing system according to the third modification of the present embodiment.

[0098] 11(a) is a diagram showing an example of a system configuration of a sensing system according to Modification 3 of Embodiment 1. This sensing system is characterized in that a first humanoid robot 20a functioning as a mobile robot holds a sensor mounting member 30.

[0099] 11(b) is a diagram showing an example of the sensor mounting member shown in (a). The sensor mounting member 30 includes a mounting member main body 31, a mounting member moving mechanism 32, a mounting member sensor 33, and a mounting member imaging device 34. The sensor mounting member 30 can be moved by the mounting member moving mechanism 32 provided below the mounting member main body 31. However, the mounting member moving mechanism 32 does not necessarily have to be provided.

[0100] The mounting member body 31 is, for example, a rod-like or cane-like member, and its material is not particularly limited. The length of the mounting member body 31 is longer than the height (back height) of the humanoid robot 20, for example, 2.1 meters. The mounting member body 31 is provided with a mounting member moving mechanism 32 below, preferably at the lower end, and a mounting member sensor 33 above, preferably at the upper end, of the mounting member body 31.

[0101] The mounting member moving mechanism 32 is configured with rotating bodies such as casters, and assists the movement of the sensor mounting member 30 in accordance with the movement of the humanoid robot 20. Note that in this embodiment, it is not assumed that the sensor mounting member 30 moves autonomously, but a mounting member control unit (not shown) that issues instructions to the mounting member moving mechanism 32 may be provided, and the mounting member moving mechanism 32 may be moved based on a signal from the mounting member control unit.

[0102] 11(a), the mounting member sensor 33 (mounting member imaging device 34) functioning as a first sensor is provided above the mounting member main body 31 and senses the worker 400. An example of the mounting member sensor 33 is similar to the robot sensor 23, and an example of the mounting member imaging device 34 is similar to the robot imaging device 24. In addition, an example of acquired sensor information is also similar to the robot sensor 23, and an example of the sensing timing of the sensor information is also similar to the robot sensor 23.

[0103] The mounting member imaging device 34 is included in the mounting member sensor 33. The mounting member sensor 33 including the mounting member imaging device 34 is disposed at a position higher than the height (back height) of the humanoid robot 20. This allows the mounting member sensor 33 to sense the movement of the worker 400 from a position higher than the robot sensor 23.

[0104] FIG. 12 is a diagram showing an example of the configuration and functions of a sensing system according to Modification 3 of Embodiment 1. In the sensing system 100'', the sensor mounting member 30 is configured to be able to communicate with the first information processing device of the first humanoid robot 20a via wireless or wired communication. However, the sensor mounting member 30 may be configured to be able to communicate with the communication unit 64 of the management control device 60 instead of or in addition to the first information processing device. The configurations of the first humanoid robot 20a, the third humanoid robot 20c functioning as a work robot, and the management control device 60 of the sensing system 100'' are the same as those of the sensing system 100.

[0105] 11, the first humanoid robot 20a grips a sensor mounting member 30 with a right gripping part (or a left gripping part) which is a part of the robot arm constituting the arm drive mechanism. The mounting member sensor 33 (mounting member imaging device 34) of the sensor mounting member 30 can change its sensing area 330 (imaging area 340) by the arm drive mechanism.

[0106] In the sensing system 100'', for example, the processing unit of the management control device 60 instructs the attachment member sensor 33 (attachment member image capture device 34) functioning as the first sensor to sense the worker 400, and the first robot sensor 23a (first robot image capture device 24a) functioning as the second sensor to sense the third humanoid robot 20c. Then, from the sensor information (first information and second information) acquired by each sensor, the motion control information is generated through learning by the learning unit of the management control device 60, and the adjustment unit of the management control device 60 compares the first information with the second information and adjusts the motion control information so that the robot motion of the third humanoid robot 20c is similar to the predetermined motion of the worker 400. Note that the roles (functions as the first sensor and the second sensor) of the attachment member sensor 33 (attachment member image capture device 34) and the first robot sensor 23a (first robot image capture device 24a) may be reversed. In other words, the first robot sensor 23a (first robot imaging device 24a) may be configured to sense the worker 400, and the mounting member sensor 33 (mounting member imaging device 34) may be configured to sense the third humanoid robot 20c.

[0107] (Effects of Modification 3) According to this sensing system, since the mounting member sensor 33 (mounting member imaging device 34) is configured as the second sensor, for example, when sensing a worker 400, it is possible to check the operation of the work robot and appropriately control it even in a narrow space where it is difficult to install multiple humanoid robots that function as mobile robots.

[0108] Furthermore, according to this sensing system, the sensor mounting member 30 has the mounting member sensor 30 (mounting member image capture device 40) disposed at a position higher than the height (back height) of the humanoid robot 20. This allows the movement of the worker 400 (or the working robot) to be sensed from a more bird's-eye view position, making it easier to avoid situations where sensing is difficult due to the back of the worker 400 or the working robot, for example, and allows efficient acquisition of data necessary for learning the work of the worker 400 or the working robot.

[0109] In addition, in this sensing system, the humanoid robot functioning as a mobile robot does not have to be one, and the sensor mounting member does not have to be one. For example, a plurality of humanoid robots 20 each having two sensor mounting members held by both holding parts 265, 266 may be provided. In this case, too, the number of sensors can be increased, making it possible to obtain a large amount of sensor information at one time. Also, the humanoid robot functioning as a work robot does not have to be one.

[0110] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment of the present invention, and various modifications and applications are possible without departing from the gist of the present invention.

[0111] In the sensing system 100 of this embodiment, two mobile robots (humanoid robots) equipped with sensors and moving mechanisms and one working robot (humanoid robot) are used, and one mobile robot is arranged for each worker 400 and one working robot. However, the relationship between the mobile robots, the worker 400, and the working robot is not limited to this. For example, if there are a large number of mobile robots equipped with sensors and moving mechanisms, it is possible to arrange multiple sensors so as to sense the predetermined movement of the worker 400 and the robotic movement of the working robot from different positions, heights, and / or directions. This makes it easier to obtain various data required for learning the predetermined movement of the worker 400 and the robotic movement of the working robot, and makes it possible to sense the movements of the worker 400 and the working robot so as to cover the entirety of each movement.

[0112] In the present embodiment, the management control device 60 has been described as acquiring the first information and the second information simultaneously. However, the management control device 60 may acquire the first information and the second information separately. In other words, the first sensing period during which the worker 400 is sensed to acquire the first information and the second sensing period during which the robot operation of the third humanoid robot 20c is sensed to acquire the second information do not have to overlap. This eliminates the need to operate the working robot in parallel with the worker 400 at the same time, making it possible to flexibly adjust the robot operation according to the predetermined operation of the worker 400.

[0113] In the present embodiment, the learning of the predetermined motion of the worker is described as being performed by automatic learning. However, the learning does not necessarily have to be automatic learning, and may be other known machine learning such as deep learning, unsupervised / supervised learning, reinforcement learning, etc.

[0114] In the present embodiment, the mobile robot and the working robot are described as being the same humanoid robot. In this case, it is possible to use the mobile robot as a working robot, and it is possible to save on the costs involved in manufacturing the robot. However, the mobile robot and the working robot may be different robots.

[0115] In addition, in the present embodiment, a worker (person) is used as the sensing target, but the present invention is not limited to this, and the sensing target may be, for example, a robot that can imitate a predetermined operation of a worker.

[0116] In this embodiment, the working robot has been described as being operated in the same work area 200 as the worker 400 and in the vicinity of the worker 400. However, the working robot does not have to be located in the vicinity of the worker, and does not have to be located in the same work area as the worker. [Explanation of symbols]

[0117] 100, 100', 100'' Sensing System 20, 20a, 20b, 20c, 20' Humanoid robots (mobile robots, working robots) 23, 23a, 23b, 23', 23'', 23a1, 23a2 Robot sensors 230, 230a, 230b, 230', 230'', 230a1, 230a2 sensing area 24, 24a, 24b, 24', 24'', 24a1, 24a2 Imaging device for robot 240, 240a, 240b, 240', 240'', 240a1, 240a2 Imaging area 25, 25a, 25' Information processing device 60 Management control device 62 Storage medium (storage unit) 1224, 1224' Storage device (storage unit)

Claims

1. A first sensor for sensing a predetermined motion of a sensing target; A work robot that operates according to operation instructions; a second sensor for sensing a robotic motion of the working robot; a management control device capable of communicating with the first sensor, the second sensor, and the work robot; The management control device includes: a learning unit that learns the predetermined motion by referring to first information acquired by the first sensor; a motion information generation unit that generates motion control information for providing the work robot with the motion instructions, by referring to a learning result of the predetermined motion by the learning unit; an adjustment unit that compares the first information with the second information acquired by the second sensor and adjusts the motion control information so that the robot motion of the working robot approximates the predetermined motion. A sensing system comprising:

2. The sensing system according to claim 1 , wherein the management control device simultaneously acquires the first information by the first sensor and acquires the second information by the second sensor.

3. The sensing system according to claim 1 , wherein the management control device separately acquires the first information by the first sensor and acquires the second information by the second sensor.

4. The management control device further includes a storage unit that stores the first information, the second information, and work manual information related to the predetermined operation, The sensing system according to any one of claims 1 to 3, wherein the motion information generation unit generates the motion control information by referring to the learning results of the specified motion by the learning unit and the work manual information.

5. The sensing system according to claim 4 , wherein the motion information generating section does not employ a learning result of the predetermined motion by the learning section that contradicts the work manual information when generating the motion control information.

6. the first sensor or the second sensor is disposed on a mobile robot having a movement mechanism, The mobile robot is capable of communicating with the management control device. The sensing system of claim 1 .

7. A first sensor for sensing a predetermined motion of a sensing target; A work robot that operates according to operation instructions; a second sensor for sensing a robotic motion of the working robot; a management control device capable of communicating with the first sensor, the second sensor, and the work robot; The management control device includes: learning the predetermined motion by referring to first information acquired by the first sensor; generating motion control information for instructing the working robot to perform the predetermined motion by referring to a learning result of the predetermined motion; comparing the first information with second information acquired by the second sensor, and adjusting the motion control information so that the robot motion of the working robot approximates the predetermined motion; A sensing method comprising:

8. A working robot that operates according to an operation instruction, A first sensor for sensing a predetermined motion of a sensing target; a second sensor for sensing a robotic motion of the working robot; an information processing unit capable of communicating with the first sensor and the second sensor, The information processing unit includes: a learning unit that learns the predetermined motion by referring to first information acquired by the first sensor; a motion information generation unit that generates motion control information for providing the work robot with the motion instructions, by referring to a learning result of the predetermined motion by the learning unit; an adjustment unit that compares the first information with the second information acquired by the second sensor and adjusts the motion control information so that the robot motion of the working robot approximates the predetermined motion. A working robot characterized by:

Images