Work robot adjustment method, sensing system, sensing method, mobile robot, operation changing system, operation changing method, work robot, work reproduction system, work reproduction method, work familiarity system, work familiarity method, and work reproduction robot

Abstract

A work robot adjustment method includes: moving a mobile robot having a sensor to an environment in which a worker operates; recording the action of the operator by using the sensor; a learning unit that learns the operation of the operator on the basis of the record; causing the work robot to perform the same operation as the operation of the worker on the basis of the learning; and performing adjustment so that the operation of the operator coincides with the operation of the work robot.

Description

Technical Field

[0001] The present invention relates to an adjustment method for a working robot performing work in a working environment, a sensing system, a sensing method, a mobile robot, a motion changing system, a motion changing method, a working robot, a work reproduction system, a work reproduction method, a work familiarization system, a work familiarization method and a work reproduction robot. Background Art

[0002] In recent years, a technology has been studied for making a working robot perform human work or assisting human work in, for example, assembly line work in a factory. For example, Patent Document 1 discloses a support system that determines the work content of a worker based on various information including the shape of a work object stored in a database, and displays information on a display device based on information obtained from various sensors provided on the working robot, thereby reducing the work burden of the worker.

[0003] In addition, in order to make the working robot perform the work of a person, it is necessary to detect (sense) the work of a person with a sensor and grasp his / her movements as a prerequisite. For example, Patent Document 2 discloses a work prediction device, which generates a work site image Ic based on a unit camera 30, generates an entire image Ia based on a ceiling camera 80, and performs image analysis.

[0004] In addition, for example, patent document 3 discloses a robot control method, which uses a sensor to read a two-dimensional code representing work information, and uses a sensor to photograph the work object, and generates action information for enabling the robot to perform the work based on the captured image and the work information, and enables the robot to perform the work based on the action information.

[0005] In addition, in a workplace where a prescribed operation is performed, an abnormal situation such as an accident may occur due to a worker's mistake or other reasons. In the event of an abnormal situation, finding out the cause is effective in suppressing the occurrence of the next abnormal situation. For example, in Patent Document 4, in order to provide a safety education system that effectively communicates the situation of an accident, the following technology is disclosed: based on three-dimensional data of an accident scene where an accident occurred and input viewpoint information, an image data of the accident scene obtained by observing the accident scene from a viewpoint specified by the viewpoint information is generated, and the image data is displayed on a display unit.

[0006] In addition, there has always been a job familiarization system to assist workers in becoming familiar with their jobs. For example, a job familiarization system is disclosed in Patent Document 5. To identify jobs with low familiarity, this job familiarization system uses out-of-specification job model information including the conditions of out-of-specification jobs (jobs indicating low familiarity), job sequence information, and internal images of the workplace to determine the job content of out-of-specification jobs.

[0007] Prior Art Documents

[0008] Patent Documents

[0009] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2021-130156

[0010] Patent Document 2: Japanese Unexamined Patent Application Publication No. 2022-113042

[0011] Patent Document 3: Japanese Unexamined Patent Application Publication No. 2022-042867

[0012] Patent Document 4: Japanese Unexamined Patent Application Publication No. 2007-226515

[0013] Patent Document 5: Japanese Unexamined Patent Application Publication No. 2020-086697 Summary of the Invention

[0014] Problems to be Solved by the Invention

[0015] When a work robot reproduces the work of a worker, it is necessary to analyze the movements of the worker and program the robot for each movement. In this case, the movement analysis and programming consume time, labor costs, and other costs.

[0016] Therefore, an object of the present disclosure is to provide a method for adjusting a work robot that reduces the time and costs consumed by the movement analysis and programming of a worker.

[0017] In addition, a worker's work sometimes involves movement or large movements, for example. In the work prediction device of Patent Document 2, since each sensor (camera) is fixedly arranged, it is sometimes impossible to sufficiently sense the work (prescribed movements) of the worker due to the relationship between the position of the worker and the sensor.

[0018] Therefore, an object of the present disclosure is to provide a sensing system, a sensing method, and a mobile robot that can sufficiently sense the prescribed movements of a worker.

[0019] In addition, in the robot control method of Patent Document 3, it is difficult to confirm whether the robot moves according to the content of the work included in the work information. In addition, when the content of the work is not properly performed, it is difficult to adjust the movement of the robot to an appropriate movement.

[0020] Accordingly, an object of the present disclosure is to provide a sensing system, a sensing method, and a working robot that can confirm the actions of a working robot and adjust them to appropriate actions.

[0021] In addition, regarding the work of a worker, since there are limitations in physical capabilities, there are also certain limitations in increasing their work speed. Therefore, if a working robot directly performs work using a learning model that has learned the specified actions of a worker, the working robot may sometimes not perform work efficiently.

[0022] Accordingly, an object of the present disclosure is to provide an action change system, an action change method, and a working robot such that in a working robot that performs work using a learning model that has learned the work of a worker, the working robot can perform work efficiently.

[0023] In addition, if an abnormal situation occurs, in most cases, it is caused by multiple factors involved. In order to identify the cause of the abnormal situation, the situation at the scene needs to be reproduced. In addition, relevant personnel (workers) etc. at the time of the abnormal situation are arranged at the reproduced scene to actually perform the actions at the time of the abnormal situation. In this case, the situation at the time of the abnormal situation can be reproduced more accurately, and it is easy to identify the cause of the abnormal situation. However, depending on the actual situation, it is sometimes difficult for relevant personnel to directly reproduce the situation of the accident.

[0024] Accordingly, an object of the present disclosure is to provide a work reproduction system, a work reproduction method, and a work reproduction robot that make it easy to identify the cause of an abnormal situation.

[0025] In addition, if only work with a low level of familiarity is determined, it is only possible to stay at researching improvement measures for the determined work, and it is not enough to achieve worker familiarity with the work.

[0026] Accordingly, an object of the present disclosure is to provide a work familiarity system, a work familiarity method, and a work reproduction robot that can achieve worker familiarity with the work.

[0027] Means for Solving the Problem

[0028] The working robot adjustment method according to the present disclosure has the following steps: moving a mobile robot having a sensor to the environment where a worker performs actions; using the sensor to record the actions of the worker; learning the actions of the worker based on the record; based on the learning, causing the working robot to perform the same actions as the worker; and making adjustments so that the actions of the worker are consistent with the actions of the working robot.

[0029] The sensing system according to the present disclosure is characterized in that it includes: a first sensor for sensing a specified action of a sensing object; a second sensor for sensing the specified action of the sensing object from a position different from that of the first sensor; a first mobile robot having the first sensor and a first moving mechanism; and a management control device capable of communicating with the first sensor, the second sensor, and the first moving mechanism, the management control device having: a determination unit for determining whether a specified part movable when the sensing object performs a specified action is sensed based on first information obtained by the first sensor and second information obtained by the second sensor; and a control unit for, when it is determined by the determination unit that the specified part is not sensed, causing the first moving mechanism to operate so that the specified part is sensed.

[0030] In addition, the sensing method according to the present disclosure is characterized in that it includes: a first sensor for sensing a specified action of a sensing object; a second sensor for sensing the specified action of the sensing object from a position different from that of the first sensor; a first mobile robot having the first sensor and a first moving mechanism; and a management control device capable of communicating with the first sensor, the second sensor, and the first moving mechanism. In the sensing method, the management control device determines whether a specified part movable when the sensing object performs a specified action is sensed based on first information obtained by the first sensor and second information obtained by the second sensor; and when it is determined by the determination unit that the specified part is not sensed, causes the first moving mechanism to operate so that the specified part is sensed.

[0031] In addition, the sensing system according to the present disclosure is characterized by including: a first sensor configured to sense a specified action of a sensing object; a second sensor configured to sense the specified action of the sensing object from a position different from that of the first sensor; a first mobile robot having the first sensor and a first moving mechanism; and a management control device capable of communicating with the first sensor, the second sensor, and the first moving mechanism, the management control device including: a determination unit configured to determine whether a specified part that is movable when the sensing object performs the specified action is sensed based on first information acquired by the first sensor and second information acquired by the second sensor, and to determine whether the specified part sensed by the first sensor is the same as the specified part sensed by the second sensor; and a control unit configured to, when it is determined that the specified part sensed by the first sensor is the same as the specified part sensed by the second sensor, cause the first moving mechanism to operate so that the specified part sensed by the first sensor is different from the specified part sensed by the second sensor.

[0032] In addition, the mobile robot according to the present disclosure is characterized by including: a moving mechanism; a first sensor configured to sense a sensing object; a second sensor configured to sense the sensing object from a position different from that of the first sensor; a driving mechanism capable of moving the position of the second sensor; and an information processing unit configured to control the first sensor, the second sensor, the moving mechanism, and the driving mechanism, the information processing unit including: a determination unit configured to determine whether a specified part that is movable when the sensing object performs a specified action is sensed based on first information acquired by the first sensor and second information acquired by the second sensor; and a control unit configured to, when it is determined by the determination unit that the specified part is not sensed, cause the moving mechanism or the driving mechanism to operate so that the specified part is sensed.

[0033] In addition, the mobile robot according to the present disclosure is characterized by including: a moving mechanism; a first sensor that senses a sensing object; a second sensor that senses the sensing object from a position different from that of the first sensor; a driving mechanism that can move the position of the second sensor; and an information processing unit that controls the first sensor, the second sensor, the moving mechanism, and the driving mechanism. The information processing unit includes: a determination unit that determines whether a specified part that can move when the sensing object performs a specified action is sensed based on first information obtained by the first sensor and second information obtained by the second sensor, and determines whether the specified part sensed by the first sensor is the same as the specified part sensed by the second sensor; and a control unit that, when it is determined that the specified part sensed by the first sensor is the same as the specified part sensed by the second sensor, causes the moving mechanism or the driving mechanism to operate so that the specified part sensed by the first sensor is different from the specified part sensed by the second sensor.

[0034] The sensing system according to the present disclosure is characterized by including: a first sensor that senses a specified action of a sensing object; a work robot that operates according to an action instruction; a second sensor that senses a robot action of the work robot; and a management control device that can communicate with the first sensor, the second sensor, and the work robot. The management control device includes: a learning unit that learns the specified action with reference to first information obtained by the first sensor; an action information generation unit that generates action control information for giving the action instruction to the work robot with reference to a learning result of the specified action by the learning unit; and an adjustment unit that compares the first information with second information obtained by the second sensor and adjusts the action control information so that the robot action of the work robot approximates the specified action.

[0035] In addition, the sensing method according to the present disclosure is characterized by including: a first sensor configured to sense a specified action of a sensing object; a work robot that operates according to an action instruction; a second sensor configured to sense the robotic action of the work robot; and a management control device capable of communicating with the first sensor, the second sensor, and the work robot. In the sensing method, the management control device learns the specified action by referring to first information acquired by the first sensor; generates action control information for giving the action instruction to the work robot by referring to the learning result of the specified action by the learning unit; and compares the first information with second information acquired by the second sensor, and adjusts the action control information so that the robotic action of the work robot approximates the specified action.

[0036] In addition, the work robot according to the present disclosure is characterized in that the work robot operates according to an action instruction, and the work robot includes: a first sensor configured to sense a specified action of a sensing object; a second sensor configured to sense the robotic action of the work robot; and an information processing unit capable of communicating with the first sensor and the second sensor. The information processing unit includes: a learning unit that learns the specified action by referring to first information acquired by the first sensor; an action information generation unit that generates action control information for giving the action instruction to the work robot by referring to the learning result of the specified action by the learning unit; and an adjustment unit that compares the first information with second information acquired by the second sensor, and adjusts the action control information so that the robotic action of the work robot approximates the specified action.

[0037] The action change system according to the present disclosure is characterized by including a work robot, a sensor, and a management control device capable of communicating with the work robot and the sensor. The management control device includes: a learning unit that learns a standard action model corresponding to the specified action of the sensing object based on sensing information corresponding to the specified action of the sensing object acquired by using the sensor; a model generation unit that generates a changed action model by referring to the standard action model, and the changed action model sets the execution time of each action in the standard action model to be shorter than the required time of each action when generating the standard action model; and a control unit that makes the work robot work by referring to the changed action model.

[0038] In addition, the action change system according to the present disclosure is characterized by including: a work robot, a plurality of sensors for respectively sensing a plurality of different sensing objects, and a management control device capable of communicating with the work robot and the plurality of sensors. The management control device includes: a learning unit that learns each specified action of the plurality of sensing objects and a plurality of standard action models corresponding to each specified action of the plurality of sensing objects based on a plurality of sensing information corresponding to the specified actions of the plurality of sensing objects obtained by using the plurality of sensors; a model generation unit that generates a changed action model by referring to the plurality of standard action models, the changed action model integrating at least a part of the specified actions of the plurality of sensing objects; and a control unit that causes the work robot to perform work by referring to the changed action model.

[0039] In addition, the action change method according to the present disclosure is characterized by learning a standard action model corresponding to a specified action of a sensing object based on sensing information corresponding to the specified action of the sensing object obtained by using a sensor; generating a changed action model by referring to the standard action model, the changed action model setting the execution time of each action in the standard action model to be shorter than the required time of each action when generating the standard action model; and causing the work robot to perform work by referring to the changed action model.

[0040] In addition, the work robot according to the present disclosure is characterized by including: a driving mechanism for causing the work robot to perform an action; a learning unit that learns a standard action model corresponding to a specified action of a sensing object based on sensing information corresponding to the specified action of the sensing object obtained by using a sensor; a model generation unit that generates a changed action model by referring to the standard action model, the changed action model setting the execution time of each action in the standard action model to be shorter than the required time of each action when generating the standard action model; and a control unit that controls the driving mechanism by referring to the changed action model so that the work robot performs work.

[0041] The operation reproduction system according to the present disclosure is characterized by including an operation reproduction robot, a sensor capable of sensing the operation of the operation reproduction robot, and a management control device capable of communicating with the operation reproduction robot and the sensor. The management control device includes: a learning unit that learns a standard operation model corresponding to a specified operation of a sensing object based on first sensing information corresponding to the specified operation of the sensing object; a control unit that refers to the standard operation model and causes the operation reproduction robot to perform one or more reproduction operations; an input unit that inputs accident or malfunction information; and a detection unit that detects the occurrence of the accident or malfunction based on second sensing information corresponding to the reproduction operation of the operation reproduction robot obtained by using the sensor.

[0042] The operation reproduction system according to the present disclosure is characterized by including an operation reproduction robot, a sensor capable of sensing the operation of the operation reproduction robot, and a management control device capable of communicating with the operation reproduction robot and the sensor. The management control device includes: a learning unit that learns a standard operation model corresponding to a specified operation of a sensing object based on first sensing information corresponding to the specified operation of the sensing object; a control unit that refers to the standard operation model and causes the operation reproduction robot to perform one or more reproduction operations; a storage unit that stores operation manual information or process sheet information of the sensing object; and a detection unit that detects the occurrence of an operation different from the operation manual information or the process sheet information based on second sensing information corresponding to the reproduction operation of the operation reproduction robot obtained by using the sensor.

[0043] In addition, the operation reproduction method according to the present disclosure is characterized by including an operation reproduction robot, a sensor capable of sensing the operation of the operation reproduction robot, and a management control device capable of communicating with the operation reproduction robot and the sensor. In the operation reproduction method, the management control device learns a standard operation model corresponding to a specified operation of a sensing object based on first sensing information corresponding to the specified operation of the sensing object; refers to the standard operation model and causes the operation reproduction robot to perform one or more reproduction operations; inputs accident or malfunction information; and detects the occurrence of the accident or malfunction based on second sensing information corresponding to the reproduction operation of the operation reproduction robot obtained by using the sensor.

[0044] In addition, the operation reproduction method according to the present disclosure is characterized by including an operation reproduction robot, a sensor capable of sensing the actions of the operation reproduction robot, and a management control device capable of communicating with the operation reproduction robot and the sensor. In the operation reproduction method, the management control device learns a standard action model corresponding to a specified action of the sensing object based on first sensing information corresponding to the specified action of the sensing object; refers to the standard action model to cause the operation reproduction robot to perform one or more reproduction actions; stores operation manual information or process table information of the sensing object; and detects the occurrence of an action different from the operation manual information or the process table information based on second sensing information corresponding to the reproduction actions of the operation reproduction robot obtained by using the sensor.

[0045] In addition, the operation reproduction robot according to the present disclosure is characterized by including a sensor capable of sensing the actions of the operation reproduction robot and an information processing device capable of communicating. The information processing device includes: a learning unit that learns a standard action model corresponding to a specified action of the sensing object based on first sensing information corresponding to the specified action of the sensing object; a control unit that refers to the standard action model to cause the operation reproduction robot to perform one or more reproduction actions; an input unit that inputs accident or malfunction information; and a detection unit that detects the occurrence of the accident or malfunction based on second sensing information corresponding to the reproduction actions of the operation reproduction robot obtained by using the external sensor.

[0046] In addition, the operation reproduction robot according to the present disclosure is characterized by including a sensor capable of sensing the actions of the operation reproduction robot and an information processing device capable of communicating. The information processing device includes: a learning unit that learns a standard action model corresponding to a specified action of the sensing object based on first sensing information corresponding to the specified action of the sensing object; a control unit that refers to the standard action model to cause the operation reproduction robot to perform one or more reproduction actions; a storage unit that stores operation manual information or process table information of the sensing object; and a detection unit that detects the occurrence of an action different from the operation manual information or the process table information based on second sensing information corresponding to the reproduction actions of the operation reproduction robot obtained by using the sensor.

[0047] The operation familiarization system related to the present disclosure is characterized by including an operation reproduction robot, a sensor capable of sensing the actions of a new operator, and a management control device capable of communicating with the operation reproduction robot and the sensor. The management control device includes: a storage unit that stores a standard action model learned based on first sensing information corresponding to the specified actions of a skilled operator; a control unit that refers to the standard action model and causes the operation reproduction robot to perform a reproduction action; and a detection unit that detects the differences between the actions of the new operator and the standard action model based on second sensing information corresponding to the actions of the new operator obtained using the sensor.

[0048] In addition, the operation familiarization method related to the present disclosure is characterized by including an operation reproduction robot, a sensor capable of sensing the actions of a new operator, and a management control device capable of communicating with the operation reproduction robot and the sensor. In the operation familiarization method, the management control device stores a standard action model learned based on first sensing information corresponding to the specified actions of a skilled operator; refers to the standard action model and causes the operation reproduction robot to perform a reproduction action; and detects the differences between the actions of the new operator and the standard action model based on second sensing information corresponding to the actions of the new operator obtained using the sensor.

[0049] In addition, the operation reproduction robot related to the present disclosure is characterized by including a sensor capable of sensing the actions of a new operator and an information processing device capable of communicating. The information processing device includes: a storage unit that stores a standard action model learned based on first sensing information corresponding to the specified actions of a skilled operator; a control unit that refers to the standard action model and causes the operation reproduction robot to perform a reproduction action; and a detection unit that detects the differences between the actions of the new operator and the standard action model based on second sensing information corresponding to the actions of the new operator obtained using the sensor.

[0050] Advantages of the Invention

[0051] According to the present disclosure, there is provided an operation robot adjustment method that reduces the time and cost consumed for action analysis and programming of an operator.

[0052] In addition, according to the present disclosure, there is provided a sensing system, a sensing method, and a mobile robot that can sufficiently sense the specified actions of an operator.

[0053] In addition, according to the present disclosure, there is provided a sensing system, a sensing method, and a work robot that can confirm the actions of the work robot and adjust them to appropriate actions.

[0054] In addition, according to the present disclosure, there is provided a method for changing actions and a work robot, such that in a work robot that performs work using a learning model that has learned the work of a worker, the work robot can perform work efficiently.

[0055] In addition, according to the present disclosure, there is provided a work reproduction system, a work reproduction method, and a work reproduction robot that make it easy to identify the cause of an abnormal situation.

[0056] In addition, according to the present disclosure, there is provided a work familiarity system, a work familiarity method, and a work reproduction robot that can achieve worker familiarity with the work. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 FIG. is an example showing the system configuration in the work robot adjustment method according to Embodiment 1 of the present disclosure.

[0058] Figure 2A FIG. shows Figure 1 an example of the work robot shown.

[0059] Figure 2B FIG. shows Figure 1 an example of the sensor mounting member shown.

[0060] Figure 3 FIG. is a block diagram showing an example of the functional configuration in the work robot adjustment method according to Embodiment 1 of the present disclosure.

[0061] Figure 4 FIG. is an example of a flowchart showing the processing of the work robot adjustment method according to Embodiment 1 of the present disclosure.

[0062] Figure 5 FIG. is an example of a modified example of the sensor mounting member.

[0063] Figure 6A FIG. is an example showing the system configuration in the sensing system according to Embodiment 2 of the present disclosure.

[0064] Figure 6B FIG. shows Figure 6A an example of the mobile robot shown.

[0065] Figure 7 FIG. is an example showing the state when the mobile robot in the sensing system according to Embodiment 2 of the present disclosure moves.

[0066] Figure 8 It is a block diagram showing an example of the structure and function in the sensing system of Embodiment 2 related to the present disclosure.

[0067] Figure 9 It is a block diagram showing an example of the function of the management control device in the sensing system of Embodiment 2 related to the present disclosure.

[0068] Figure 10 It is a flowchart showing an example of the processing of the sensing system of Embodiment 2 related to the present disclosure.

[0069] Figure 11 It shows Figure 10 An example of a flowchart showing a more detailed process of the specified part sensing determination process shown in step S2103.

[0070] Figure 12 It shows Figure 10 An example of a flowchart showing a more detailed process of the action information generation process shown in step S2104.

[0071] Figure 13A It is a diagram showing an example of the system structure in the sensing system related to Modification 1 of Embodiment 2 related to the present disclosure.

[0072] Figure 13B It shows Figure 13A A diagram showing an example of the mobile robot shown.

[0073] Figure 14 It is a block diagram showing an example of the function of the mobile robot in the sensing system related to Modification 1 of Embodiment 2 related to the present disclosure.

[0074] Figure 15A It is a diagram showing an example of the system structure in the sensing system related to Modification 2 of Embodiment 2 related to the present disclosure.

[0075] Figure 15B It shows Figure 15A A diagram showing an example of the sensor mounting member shown.

[0076] Figure 16 It is a diagram showing an example of the structure and function in the sensing system related to Modification 2 of Embodiment 2 related to the present disclosure.

[0077] Figure 17A It is a diagram showing an example of the system structure in the sensing system of Embodiment 3 related to the present disclosure.

[0078] Figure 17B It shows Figure 17AA diagram of an example of the humanoid robot shown.

[0079] Figure 18 It is a diagram showing an example of the structure and function in the sensing system of Embodiment 3 related to the present disclosure.

[0080] Figure 19 It is a diagram showing an example of the function of the management control device in the sensing system of Embodiment 3 related to the present disclosure.

[0081] Figure 20 It is a diagram showing an example of each sensing period in the sensing system of Embodiment 3 related to the present disclosure.

[0082] Figure 21 It is a diagram showing an example of the flowchart of the processing of the sensing system of Embodiment 3 related to the present disclosure.

[0083] Figure 22 It shows Figure 21 An example of a flowchart of a more detailed process of the action control information generation process shown in step S3103.

[0084] Figure 23 It shows Figure 21 An example of a flowchart of a more detailed process of the action control information adjustment process shown in step S3106.

[0085] Figure 24A It is a diagram showing an example of the system structure in the sensing system related to Modification 1 of Embodiment 3 related to the present disclosure.

[0086] Figure 24B It shows Figure 24A A diagram of an example of the humanoid robot shown.

[0087] Figure 25 It is a block diagram showing an example of the function of the humanoid robot in the sensing system related to Modification 1 of Embodiment 3 related to the present disclosure.

[0088] Figure 26 It is a diagram showing an example of the system structure in the sensing system related to Modification 2 of Embodiment 3 related to the present disclosure.

[0089] Figure 27A It is a diagram showing an example of the system structure in the sensing system related to Modification 3 of Embodiment 3 related to the present disclosure.

[0090] Figure 27B It shows Figure 27A A diagram of an example of the sensor mounting member shown.

[0091] Figure 28 It is a diagram showing an example of the structure and function in the sensing system related to Modification 3 of Embodiment 3 according to the present disclosure.

[0092] Figure 29A It is a diagram showing an example of the system structure in the action change system of Embodiment 4 according to the present disclosure.

[0093] Figure 29B It shows Figure 29A a diagram showing an example of the humanoid robot shown.

[0094] Figure 30 It is a diagram showing an example of the relationship between the standard action model and the action change model in the action change system of Embodiment 4 according to the present disclosure.

[0095] Figure 31 It is a block diagram showing an example of the structure and function in the action change system of Embodiment 4 according to the present disclosure.

[0096] Figure 32 It is a block diagram showing an example of the function of the management control device in the action change system of Embodiment 4 according to the present disclosure.

[0097] Figure 33 It is a diagram showing an example of the flowchart of the processing in the action change system of Embodiment 4 according to the present disclosure.

[0098] Figure 34 It shows Figure 33 a diagram showing an example of a more detailed flowchart of the changed action model generation process shown in step S4103.

[0099] Figure 35A It is a diagram showing an example of the system structure in the action change system related to Modification 1 of Embodiment 4 according to the present disclosure.

[0100] Figure 35B It shows Figure 35B a diagram showing an example of the humanoid robot shown.

[0101] Figure 36 It is a diagram showing an example of the function of the humanoid robot in the action change system related to Modification 1 of Embodiment 4 according to the present disclosure.

[0102] Figure 37 It is a diagram showing an example of a more detailed flowchart of the changed action model generation process in the action change system related to Modification 1 of Embodiment 4 according to the present disclosure.

[0103] Figure 38It is a diagram showing an example of the relationship between a standard action model and an action change model in an action change system according to Modification 1 of Embodiment 4 related to the present disclosure.

[0104] Figure 39A It is a diagram showing an example of sensing when a worker reproduces an action in a work reproduction system according to Embodiment 5 related to the present disclosure.

[0105] Figure 39B It is a diagram showing an example of sensing when a work reproduction robot reproduces an action.

[0106] Figure 40 It is an example of a humanoid robot in a work reproduction system according to Embodiment 5 related to the present disclosure.

[0107] Figure 41 It is a block diagram showing an example of the structure and functions in a work reproduction system according to Embodiment 5 related to the present disclosure.

[0108] Figure 42 It is a block diagram showing an example of the functions of a management control device in a work reproduction system according to Embodiment 5 related to the present disclosure.

[0109] Figure 43 It is an example of a flowchart of the processing in a work reproduction system according to Embodiment 5 related to the present disclosure.

[0110] Figure 44 It is shown Figure 43 An example of a flowchart showing a more detailed process of the worker action learning / standard action model generation process shown in step S5102.

[0111] Figure 45 It is an example of a flowchart of the processing in a work reproduction system according to Modification 1 of Embodiment 5 related to the present disclosure.

[0112] Figure 46A It is a diagram showing an example of the system structure in a work reproduction system according to Modification 2 of Embodiment 5 related to the present disclosure.

[0113] Figure 46B It is shown Figure 46A A diagram showing an example of the humanoid robot shown.

[0114] Figure 47 It is a block diagram showing an example of the functions of a humanoid robot in a work reproduction system according to Modification 2 of Embodiment 5 related to the present disclosure.

[0115] Figure 48AThis is a diagram showing an example of the system configuration in the job familiarization system according to Embodiment 6 of the present disclosure.

[0116] Figure 48B This is a diagram showing Figure 48A an example of the humanoid robot shown.

[0117] Figure 49 This is a block diagram showing an example of the configuration and functions in the job familiarization system according to Embodiment 6 of the present disclosure.

[0118] Figure 50 This is a block diagram showing an example of the functions of the management control device in the job familiarization system according to Embodiment 6 of the present disclosure.

[0119] Figure 51 This is a flowchart showing an example of the processing in the job familiarization system according to Embodiment 6 of the present disclosure.

[0120] Figure 52 This is a diagram showing Figure 51 an example of a more detailed flowchart of the operator motion learning and model generation process shown in step S6102.

[0121] Figure 53 This is a diagram showing Figure 51 an example of a more detailed flowchart of the motion detection process shown in step S6105.

[0122] Figure 54A This is a diagram showing an example of the system configuration in the job familiarization system according to Modification 1 of Embodiment 6 of the present disclosure.

[0123] Figure 54B This is a diagram showing Figure 54A an example of the humanoid robot shown.

[0124] Figure 55 This is a block diagram showing an example of the functions of the humanoid robot in the job familiarization system according to Modification 1 of Embodiment 6 of the present disclosure. Detailed Description of the Invention

[0125] Hereinafter, a job robot adjustment method, a sensing system, a sensing method, a mobile robot, an action change system, an action change method, a job robot, a job reproduction system, a job reproduction method, a job familiarization system, a job familiarization method, and a job reproduction robot will be described with reference to the accompanying drawings. However, it should be noted that the scope of the technology of the present disclosure is not limited to these embodiments, but encompasses the disclosed content described in the claims of the patent and its equivalents.

[0126] (Embodiment 1)

[0127] Figure 1 This is a diagram for explaining the operation robot adjustment method. It should be noted that, in order to avoid Figure 1 becoming complicated, for the reference numerals of the sensing areas of the respective sensors, only the sensing area 330a of the sensor 33 for the mounting member described later is labeled. In addition, regarding the details of the operation robot adjustment system 100 for performing the operation robot adjustment method, Figure 3 it will be described later.

[0128] In the operation robot adjustment method, there are provided a plurality of humanoid robots 20a to 20d that function as mobile robots, and sensor mounting members 30a to 30d respectively connected to the humanoid robots 20a to 20d.

[0129] Each of the humanoid robots 20a to 20d receives instructions from a management control device 60 (refer to Figure 3 ), which will be described later, or moves near an operator 400 working on an operation line 201 in an operation site 200 according to an instruction from an information processing device 25 (refer to Figure 3 ), which is provided inside the humanoid robots 20a to 20d. Since the sensor mounting members 30a to 30d are connected to the humanoid robots 20a to 20d, they move as the humanoid robots 20a to 20d move.

[0130] Moreover, the actions of the operator 400 are recorded by sensors 23a to 23d and 33a to 33d provided in the humanoid robots 20a to 20d and the sensor mounting members 30a to 30d. After the actions are recorded, learning of the actions of the operator 400 is performed based on the record. This learning is performed by automatic learning. Here, automatic learning refers to learning for automatically creating a learned completion model or learning for performing determination / analysis using a learned completion model.

[0131] After learning the actions of the operator 400, the humanoid robot 20 that functions as an operation robot receives instructions from the management control device 60 or moves according to an instruction from the information processing device 25 provided inside the humanoid robots 20a to 20d to perform the same actions as those of the operator 400. The process of making the humanoid robot 20 perform the same actions as those of the operator 400 is also performed by automatic learning.

[0132] In the process of making the humanoid robot 20 perform the same actions as those of the professional operator 400, the humanoid robot 20 is made to repeat the automatically learned operation about 300 times, for example, until the humanoid robot 20 can execute actions with the same actions, routes, speeds, etc. as those of the automatically learned operation. In addition, the process of increasing the speed of the automatically learned operation, for example, at 20 times the speed, is repeated until it can be executed with the same actions, routes, speeds, etc. as those of the automatically learned operation.

[0133] Reflect the learning results to multiple work robots (humanoid robots 20), and repeat the above processing until the actions of the multiple work robots are consistent from start to stop. Thereby, the time and cost consumed for the motion analysis and programming of the operator can be reduced.

[0134] FIG. 2 shows Figure 1 an example of the work robot and the sensor mounting member shown. Referring to FIG. 2, the structures of the humanoid robots 20a to 20d and the sensor mounting members 30a to 30d will be described.

[0135] As shown in FIG. 2(a), the humanoid robot 20 includes a robot main body 21, a robot moving mechanism 22, a robot sensor 23, a robot photographing device 24 included in the robot sensor 23, an information processing device 25 (refer to Figure 3 ), and an arm portion 26.

[0136] The humanoid robot 20 can move by the robot moving mechanism 22 provided below the robot main body 21. For example, it receives instructions from the outside of the humanoid robot 20 such as a management control device, or refers to a program recorded in the information processing device 25 and moves near the work line 201 of the work place 200.

[0137] In the robot main body 21, the robot moving mechanism 22 is provided below it, the arm portion 26 is provided above it, and the robot sensor 23 is provided further above the arm portion 26. In addition, the information processing device 25 is provided inside the robot main body 21.

[0138] The robot moving mechanism 22 can be of any structure. For example, it can be a structure provided with a rotating body driven by a motor, or a structure that imitates the shape of a human foot as a foot portion.

[0139] The robot sensor 23 is disposed above the humanoid robot 20, preferably at the top of the robot main body 21, in other words, near the head of the humanoid robot, to detect the operator 400. In addition, the robot sensor 23 sequentially obtains information indicating at least the distance and angle between an object to be operated by the humanoid robot 20 located in the periphery of the humanoid robot 20 and the arm 26. As an example of the robot sensor 23, the highest-performance camera, thermal imaging camera, high-pixel / long-focal-length / ultra-wide-angle / 360-degree / high-performance camera, radar, solid-state lidar (LiDAR), LiDAR, multi-color laser coaxial displacement meter, visual recognition, or other various sensor groups can be adopted. These are also an example of the robot imaging device 24. In addition, in addition to these, as another example of the robot sensor 23, a vibration meter, hardness meter, fine sound, ultrasonic wave, vibration, infrared ray, ultraviolet ray, electromagnetic wave, temperature, humidity, spot artificial intelligence (AI) weather forecast, high-precision multi-channel global positioning system (GPS), low-altitude satellite information, or long-tail event AI data, etc. are listed.

[0140] As an example of the sensor information obtained from the robot sensor 23, images, distances, vibrations, heat, odors, colors, sounds, ultrasonic waves, ultraviolet rays, infrared rays, etc. are listed. Preferably, the robot imaging device 24 is used to obtain the information of images and distances. As an example, the robot sensor 23 (robot imaging device 24) performs these detections every nanosecond. The sensor information is used, for example, for motion capture of the actions of the operator 400, 3D mapping of the work site 200, and analysis of the movement or actions of the operator 400 in the work site 200, such as navigation, steering, and speed.

[0141] The arm 26 is rotatably mounted above the robot main body 21. In addition, a gripping portion (not shown) for gripping an object is mounted at the front end of the arm 26. Through the gripping portion, the sensor mounting member 30 is connected to the humanoid robot 20.

[0142] It should be noted that the humanoid robot 20 may also be provided with a sensor at the central portion of the robot main body 21. As an example, a sensor is also provided in the trunk of the humanoid robot. In this case, the height position of this sensor is different from that of the robot sensor 23 provided near the top of the robot main body. Due to the different height positions, the sensors can detect the actions of the operator 400 from different angles.

[0143] As shown in Fig. 2(b), the sensor mounting member 30 includes a mounting member main body 31, a mounting member moving mechanism 32, a sensor for the mounting member 33, and a photographing device for the mounting member 34. The sensor mounting member 30 can be moved by the mounting member moving mechanism 32 provided below the mounting member main body 31.

[0144] The mounting member main body 31 is, for example, a rod-shaped or stick-shaped member, and its material is not particularly limited. The length of the mounting member main body 31 is longer than the height (body height) of the humanoid robot 20. Below the mounting member main body 31, it is preferable to provide the mounting member moving mechanism 32 at the lower end, and above the mounting member main body 31, it is preferable to provide the sensor for the mounting member 33 at the upper end.

[0145] The mounting member moving mechanism 32 has a structure in which, for example, a rotating body such as a caster is provided, and it assists the sensor mounting member 30 to move as the humanoid robot 20 moves. It should be noted that in the present embodiment, although it is not assumed that the sensor mounting member 30 will move autonomously, it can also be configured to provide a mounting member control unit (not shown) that gives an instruction to the mounting member moving mechanism 32, and based on a signal from the mounting member control unit, the mounting member moving mechanism 32 is activated.

[0146] The sensor for the mounting member 33 is provided above the mounting member main body 31 to detect the operator 400. In addition, the sensor for the mounting member 33 sequentially acquires information indicating at least the distance and angle between an object on which the humanoid robot 20 performs an operation and the arm 26 around the humanoid robot 20. An example of the sensor for the mounting member 33 is the same as that of the robot sensor 23, and an example of the photographing device for the mounting member 34 is the same as an example of the robot photographing device 24. Also, an example of the acquired sensor information is the same as that of the robot sensor 23, and an example of the detection time of the sensor information is the same as that of the robot sensor 23.

[0147] The photographing device for the mounting member 34 is included in the sensor for the mounting member 33. In addition, the sensor for the mounting member 33 including the photographing device for the mounting member 34 is arranged at a position higher than the height (body height) of the humanoid robot 20. Thus, the sensor for the mounting member 33 can detect the actions of the operator from a position higher than the robot sensor 23.

[0148] The sensor for the mounting member 33 is provided on the mounting member main body 31 in such a way that its sensing area 330 is in the direction of detecting the actions of the operator 400. For the robot sensor 23, it is also provided on the robot main body 21 in such a way that the sensing area (not shown) is in the direction of detecting the actions of the operator 400.

[0149] AsFigure 1 As shown, four humanoid robots 20a to 20d and four sensor mounting members 30a to 30d are respectively arranged to detect the actions of the operator 400 from different positions, heights, or / and orientations. Thus, in the present embodiment, since multiple sensors are arranged to detect the actions of the operator 400 from different positions, heights, or / and orientations, various data can be obtained in the learning of the actions of the operator 400. It should be noted that the number of robots for recording the actions of the operator 400 is not limited to four, and can also be one to three or five or more.

[0150] In addition, in Figure 1 , for the sensor mounting member 30d, in order to sense the vicinity of the hand of the operator 400, an extension member 35d and an additional sensor 33d2 for the mounting member are provided. The extension member 35d is a rod-shaped member arranged to extend horizontally from near the sensor 33d1 for the mounting member. In addition, the sensor 33d2 for the mounting member is provided at the front end of the extension member 35d, and the sensor 33d2 for the mounting member senses the operator 400 from above the operator 400. By means of the sensor 33d2 for the mounting member, it is possible to easily detect the actions of the operator 400. And when a sensor (imaging device) different from the robot sensor 23 is also provided in the robot main body 21 of each humanoid robot, the total number of sensors possessed by multiple humanoid robots is eight, and the total number of sensors possessed by multiple sensor mounting members is five. In total, through thirteen sensors, the actions of the operator 400 can be detected.

[0151] Figure 3 It is a block diagram showing an example of the functional structure in the work robot adjustment system 100.

[0152] The work robot adjustment system 100 is configured to include a humanoid robot 20, a sensor mounting member 30, and a management control device 60. The humanoid robots 20 are respectively connected to the relative robot communication unit 68 of the management control device 60 and the sensor mounting member 30 via wireless communication or wired communication, receive instructions from the management control device 60, and obtain detection results from the sensor mounting member 30. It should be noted that the sensor mounting member 30 can also be configured to be able to communicate with the management control device 60. In addition, the humanoid robots 20 can also be set such that not one but multiple are connected to the management control device 60.

[0153] The humanoid robot 20 includes a robot sensor 23, a robot imaging device 24 included in the robot sensor 23, and an information processing device 25.

[0154] The information processing apparatus 25 according to the present embodiment includes a CPU (Central Processing Unit) 1212, a RAM (Random Access Memory) 1214, and a graphics controller 1216 that are interconnected via a host controller 1210. In addition, the information processing apparatus 25 includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive can be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 can be a hard disk drive, a solid state drive, or the like. In addition, the information processing apparatus 25 includes a ROM (Read Only Memory) 1230 and input / output units such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0155] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 from a frame buffer or the like provided in the RAM 1214 or itself, and causes the image data to be displayed on the display device 1218.

[0156] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the information processing apparatus 25. The DVD drive reads a program or data from a DVD-ROM or the like and provides it to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to the IC card.

[0157] The ROM 1230 stores a boot program or the like executed by the information processing apparatus 25 at startup and / or a program dependent on the hardware of the information processing apparatus 25. In addition, the input / output chip 1240 can also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, or the like.

[0158] The program is provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The program is read from the computer-readable storage medium, installed in a storage device 1224, a RAM 1214, or a ROM 1230, which are also examples of computer-readable storage media, and executed by a CPU 1212. The information processing described in these programs is read by an information processing device 25, enabling cooperation between the programs and the above-mentioned various types of hardware resources. The device or method may be configured to implement operations or processing of information according to the use of the information processing device 25.

[0159] For example, when communication is performed between the information processing device 25 and an external device, the CPU 1212 may execute a communication program loaded into the RAM 1214 and, based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in a transmission buffer area provided in a recording medium such as the RAM 1214, the storage device 1224, the DVD-ROM, or the IC card, and transmits the read transmission data to the network, or writes the received data received from the network to a reception buffer area provided on the recording medium.

[0160] In addition, the CPU 1212 may cause all or a necessary part of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), or an IC card to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. Then, the CPU 1212 may write the processed data back to the external recording medium.

[0161] Various types of information such as various types of programs, data, tables, and databases may be stored in a recording medium to undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214. The various types of processing include various types of operations, information processing, conditional judgments, conditional branches, unconditional branches, information retrieval / replacement, etc. specified by an instruction sequence of a program described throughout this disclosure, and write the result back to the RAM 1214. In addition, the CPU 1212 may retrieve information in files, databases, etc. within the recording medium.

[0162] The programs or software modules described above may be stored on a computer-readable storage medium on or near the information processing device 25. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet may be used as a computer-readable storage medium, thereby providing a program to the information processing device 25 via the network.

[0163] The flowcharts and the boxes in the figures in this embodiment may represent stages of a process of performing operations or "parts" of a device having the function of performing operations. Specific stages and "parts" may be implemented by dedicated circuits, programmable circuits supplied together with computer-readable instructions stored on a computer-readable storage medium, and / or processors supplied together with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuits may include digital and / or analog hardware circuits, and may also include integrated circuits (ICs) and / or discrete circuits. The programmable circuits may include reconfigurable hardware circuits such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), which include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, and storage elements.

[0164] The computer-readable storage medium may include any tangible device capable of storing instructions executable by an appropriate device. As a result, a computer-readable storage medium having instructions stored thereon in a tangible device has a product including instructions that can be executed to create a unit for performing the operations specified in the flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy (registered trademark) disks, magnetic disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disc read-only memories (CD-ROMs), digital versatile discs (DVDs), Blu-Ray (registered trademark) disks, memory sticks, integrated circuit cards, etc.

[0165] Computer-readable instructions may include any one of assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source code or object code described in any combination of one or more programming languages, the one or more programming languages including object-oriented programming languages such as Smalltalk (registered trademark), JAVA (registered trademark), C++, etc. and traditional procedural programming languages such as the "C" programming language or the like.

[0166] The computer-readable instructions may be provided locally or via a local area network (LAN), a wide area network (WAN) such as the Internet, etc. to the processor or programmable circuit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, causing the processor or programmable circuit of the general-purpose computer, special-purpose computer, or other programmable data processing device to execute the computer-readable instructions to generate units for performing the operations specified in the flowchart or block diagram. Examples of the processor include a computer processor, a processing unit, a microprocessor, a digital signal processor, a controller, a microcontroller, etc.

[0167] The sensor mounting member 30 is connected to the arm 26 of the humanoid robot 20 and moves with the humanoid robot 20. The mounting member-mounted sensor 33 (the mounting member-mounted imaging device 34) detects information of an object and transmits the information to the information processing device 25 via a mounting member-mounted communication unit (not shown).

[0168] The management control device 60 is a control device that gives instructions to the humanoid robot 20 to implement the operation adjustment robot adjustment method. In addition, the management control device 60 may also acquire the sensor information stored in the storage device 1224.

[0169] The management control device 60 is composed of a CPU 60A, a RAM 60B, a ROM 60C, an input / output unit (I / O) 60D, a bus 60E such as a data bus or a control bus connecting them, and a transceiver unit 68 for the robot. A recording medium 62 is connected to the I / O 60D.

[0170] In addition, a transceiver unit 68 for the robot is connected to the I / O 60D, and the transceiver unit 68 for the robot transmits and receives motion control information including operation information between the control systems of the humanoid robot 20.

[0171] (Processing of the operation robot adjustment method according to Embodiment 1 related to the present disclosure)

[0172] Figure 4This is an example of a flowchart showing the processing of the work robot adjustment method according to the present embodiment.

[0173] First, according to the instruction of the management control device 60 or according to the read instruction of the program stored in the storage device 1224 of the information processing device 25, the information processing device 25 gives an instruction to cause the humanoid robot 20 functioning as a mobile robot to move to the work place (work environment) 200 (step S101). The movement is performed based on the operation of the robot movement mechanism 22 of the humanoid robot 20. At this time, since the sensor mounting member 30 is connected to the humanoid robot 20, it moves along with the movement of the humanoid robot 20.

[0174] During the movement, an instruction is given so that the sensing areas (imaging areas) of the robot sensors 23 (robot imaging devices 24) and the sensor mounting member sensors 33 (mounting member imaging devices 34) detect the worker 400 from different orientations. The configuration of the humanoid robot 20 and the sensor mounting member 30 is performed, for example, by recording the floor plan of the work place 200 in advance in the storage device 1224 and / or the recording medium 62 and making the positions of the humanoid robots 20, etc. correspond to the recorded floor plan. Alternatively, the configuration of the humanoid robot 20, etc. is based on the position optimized by machine learning.

[0175] Next, the actions on the work line 201 of the worker 400 are detected by the plurality of sensors 23a to 23d, 33a to 33d (the plurality of imaging devices 24a to 24d, 34a to 34d) (step S102). The information processing devices 25 of the respective humanoid robots 20a to 20d acquire the sensor information detected by the various sensors. The acquired sensor information is stored in the storage device 1224.

[0176] The information processing device 25 learns the actions of the worker 400 based on the sensor information stored in the storage device 1224, in other words, based on the recorded sensor information (step S103). During the learning, action capture of the actions of the worker 400, analysis of the 3D map of the work place 200, navigation, turning, speed, etc. of the movement or actions of the worker 400 in the work place 200 are performed, and the optimal actions of the humanoid robot 20 are learned through automatic learning.

[0177] As a next step, the information processing device 25 or the management control device 60, based on the automatic learning in step S103, instructs the humanoid robot 20 that functions as a work robot to perform the same actions as the worker 40 (step S104). Specifically, the arm 26 of the robot and / or the robot moving mechanism 22 cause the humanoid robot 20 to repeatedly perform the actions based on the automatic learning in step S103 in the same manner as the work (actions) of the worker 400 obtained through the automatic learning in step S103. This action is repeated, for example, several hundred times, about 300 times as an example, and this process is repeated until the humanoid robot is the same as the work obtained through automatic learning in terms of actions, routes (movements), speeds, etc.

[0178] In addition, in S104, for example, the speed of the work (actions) learned automatically is increased by 20 times, etc., and the humanoid robot 20 is made to perform the actions based on the automatic learning in step S103. This action is also repeated, for example, several hundred times, about 300 times as an example, and this process is repeated until the humanoid robot 20 performs actions with the same actions, routes (movements), and speeds as the work obtained through automatic learning.

[0179] Furthermore, the information processing device 25 or the management control device 60 performs the following processing: adjustment is performed so that the actions of the humanoid robot (work robot) coincide with the actions of the worker 400 (step S105). Specifically, the information processing device 25 or the management control device 60, based on the results obtained in step S104, instructs a plurality of work robots, in this embodiment, a plurality of humanoid robots 20, to perform the same actions as the worker 400. This process is repeated until the actions of the plurality of humanoid robots 20 are synchronized. If the actions of the plurality of humanoid robots 20 are synchronized, then the series of work robot adjustment methods ends.

[0180] (Function and effect of the work robot adjustment method according to Embodiment 1)

[0181] According to the work robot adjustment method according to this embodiment, the humanoid robot 20, which is a mobile robot, is equipped with sensors. On the other hand, the humanoid robot 20 automatically moves to an appropriate position according to a pre-stored program or the result of machine learning. Therefore, compared with the case where fixed sensors are arranged in the work place 200, there is no need to re-arrange sensors or increase the number of sensors according to the work place, and the sensing environment of the worker 400 can be adjusted in a short period at low cost.

[0182] In addition, the humanoid robot 20 is connected to the sensor mounting member 30 equipped with sensors (imaging devices). Therefore, when the humanoid robot 20 moves, a plurality of sensors can be moved simultaneously.

[0183] In addition, in the method for adjusting a work robot, the following structure is adopted: a mobile robot (humanoid robot 20) equipped with sensors senses the work (actions) of the worker 400. Therefore, it is also possible to move the mobile robot as the worker 400 moves, and to sense at a position suitable for the actions of the worker 400.

[0184] In addition, the sensor mounting member 30 is provided with a sensor 30 for the mounting member (imaging device 40 for the mounting member) at a position higher than the height (height) of the humanoid robot. Therefore, it is possible to sense the actions of the worker 400 from a more top-down position.

[0185] In addition, in the method for adjusting a work robot, the following structure is adopted: a plurality of humanoid robots 20a to 20d and a plurality of sensor mounting members 30a to 30d are respectively arranged at different positions, and the actions of the worker 400 are sensed by these sensors. Therefore, it is possible to sense one work of the worker 400 from different positions, and it is possible to obtain a large amount of required data at one time on the basis of performing automatic learning. As a result, it is possible to reduce the time and cost consumed for analyzing and programming the actions of the worker 400.

[0186] In addition, in the method for adjusting a work robot, the actions of the worker 400 are automatically learned based on sensor information. During automatic learning, action capture of the actions of the worker 400, 3D mapping of the work site 200, and analysis of navigation, steering, speed, etc. of the movement or actions of the worker 400 in the work site 200 are performed. Therefore, it is possible to analyze the actions of the worker 400 from multiple angles at one time, and it is possible to reduce the time and cost consumed for analyzing and programming the actions of the worker 400.

[0187] In addition, in the method for adjusting a work robot, the following structure is adopted: the work robot (humanoid robot 20) performs actions based on learning, and the process is repeated until the work learned automatically and the actions of the work robot become the same in terms of actions, routes (movement), speed, etc. Thereby, the time for confirming whether the actions of the work robot are accurate with respect to the work of the worker 400 is shortened.

[0188] In addition, in the present embodiment, the mobile robot and the work robot use the same robot (humanoid robot 20). Therefore, there is no need to manufacture robots in different processes, and the cost and time for adjusting the work robot are shortened.

[0189] In addition, in the method for adjusting a work robot, the following structure is adopted: the work robot (humanoid robot 20) is adjusted so that its actions are consistent with the actions of the worker. Thereby, the accuracy of the actions of the work robot with respect to the work of the worker 400 is ensured.

[0190] In addition, in the method for adjusting a work robot, the following structure is adopted: the results of automatic learning are applied to multiple work robots (humanoid robot 20), and the process is repeated until the actions of the multiple work robots are the same. Therefore, the accuracy of the actions of the work robot and the work of the worker 400 is further ensured.

[0191] (Deformation example of sensor mounting member)

[0192] Figure 5 is an example of a deformation example of the sensor mounting member.

[0193] The difference between the sensor mounting member 30' and the sensor mounting member 30 is that three or more (eight in Figure 5 ) sensor mounting member sensors 33a' to 33g' are provided in the sensor mounting member 30'.

[0194] Specifically, the sensor mounting member 30' includes a mounting member main body 30', a mounting member moving mechanism 22', a plurality of sensor mounting member sensors 33a' to 33g' (a plurality of photographing devices), and a plurality of extension members 35a', 35b'. It should be noted that in Figure 5 , in order to avoid complicating the drawings, only the reference numerals of the extension members are marked as 35a', 35b'.

[0195] Through the plurality of extension members, the appearance of the sensor mounting member 30' has a structure like that of a spider's leg. In addition, for the sensor mounting member 30', the plurality of sensor mounting member sensors 33a' to 33g' are respectively arranged in different orientations and / or heights.

[0196] According to the sensor mounting member 30', a plurality of sensors (photographing devices) are arranged, and a large number of sensors (photographing devices) can be moved at one time by the mounting member moving mechanism 32'. Thus, for example, in a work place where the space is narrow for arranging multiple sensor mounting members, it is also possible to sense the work of the worker from various heights, positions, and / or orientations.

[0197] The above describes the embodiments of the present disclosure, but the present disclosure is not limited to the above-described embodiments of the present disclosure, and various deformations or applications can be made without departing from the gist of the present disclosure.

[0198] In the present embodiment, an example in which the mobile robot (humanoid robot) and the sensor mounting members are plural has been described. However, it is not limited thereto. For example, it is also possible to move a single mobile robot equipped with sensors and record the work of the worker.

[0199] In this embodiment, an example in which the learning of S103 is performed by automatic learning has been described. However, the learning does not necessarily have to be automatic learning, and it can also be other known machine learning, such as deep learning, unsupervised learning / supervised learning, reinforcement learning, etc.

[0200] In this embodiment, a structure in which one mobile robot (humanoid robot) is connected to one sensor mounting member is shown. However, it is not limited to this. For example, one mobile robot may be connected to a plurality of sensor mounting members.

[0201] In this embodiment, an example in which the mobile robot and the work robot are the same humanoid robot has been described. However, the mobile robot and the work robot may also be different robots.

[0202] (Embodiment 2)

[0203] Figure 6A 、 Figure 6B A diagram for explaining the sensing system.

[0204] Figure 6A It is a diagram showing an example of the system structure in the sensing system according to Embodiment 2 of the present disclosure. The sensing system includes a first humanoid robot 2020a and a second humanoid robot 2020b that function as mobile robots. It should be noted that the number of humanoid robots is not limited to two.

[0205] Each humanoid robot 2020a, 2020b receives an instruction from a management control device 2060 (refer to Figure 8 ) described later, or moves near a worker 400 working on a work line 201 in a work place 200 according to an instruction from each information processing device 2025a, 2025b (refer to Figure 8 ) provided in each humanoid robot 2020a, 2020b. Moreover, the sensing system senses a specified action of the worker 400 through a first sensor 2023a (first photographing device 2024a) included in the first humanoid robot 2020a and a second sensor 2023b (second photographing device 2024b) included in the second humanoid robot 2020b. The second sensor 2023b (second photographing device 2024b) senses the specified action of the worker 400 from a position different from that of the first sensor 2023a (first photographing device 2024a).

[0206] Figure 6B It is shown Figure 6AA diagram of an example of the mobile robot shown. The humanoid robot 2020 that functions as a mobile robot includes a robot main body 2021, a robot moving mechanism 2022, a robot sensor 2023, a robot imaging device 2024 included in the robot sensor 2023, an information processing device 2025, and a robot arm 2026.

[0207] The humanoid robot 2020 can move by means of the robot moving mechanism 2022 provided below the robot main body 2021. For example, it receives an instruction from outside the humanoid robot 2020 such as a management control device 2060, or refers to a program stored in the information processing device 2025 and moves near the operation line 201 of the work place 200.

[0208] The robot main body 2021 includes a robot torso 2211 and a robot head 2212. The robot torso 2211 and the robot head 2212 constitute a first drive mechanism that can change the sensing area 2230 (imaging area 2240) of the robot sensor 2023 (robot imaging device 2024). The structure of the drive mechanism is not particularly limited. For example, it may be configured such that, using a servo motor (not shown), the robot head 2212 rotates by a predetermined angle relative to the robot torso 2211, or the robot torso 2211 rotates by a predetermined angle relative to the robot moving mechanism 22.

[0209] The robot moving mechanism 2022 is provided below the robot torso 2211, the robot arm 2026 is provided on the side of the robot torso 2211, and the robot sensor 2023 is provided on the robot head 2212. In addition, the information processing device 2025 is provided inside the robot main body 2021.

[0210] The robot moving mechanism 2022 can be of any structure. For example, it can be a structure in which a rotating body driven by a motor is provided, or it can be a structure that mimics the shape of a human foot as the feet. As an example, when the robot moving mechanism 2022 is configured to mimic the shape of a human foot, servo motors are provided at positions corresponding to human joints, and the moving mechanism is constituted by rotating them by a predetermined angle.

[0211] The robot sensor 2023 is preferably arranged on the robot head 2212 to sense the operator 400. In addition, the robot sensor 2023 sequentially obtains information indicating at least the distance and angle between an object on which the humanoid robot 2020 operates and the robot arm 2026 located around the humanoid robot 2020. As an example of the robot sensor 2023, the highest-performance camera, thermal imaging camera, high-pixel / long-focus / ultra-wide-angle / 360-degree / high-performance camera, radar, solid-state LiDAR, LiDAR, multi-color laser coaxial displacement gauge, visual recognition, or other various sensor groups can be adopted. These are also an example of the robot imaging device 2024. In addition, besides this, as another example of the robot sensor 2023, a vibrometer, hardness tester, micro-vibrometer, ultrasonic measuring instrument, vibration measuring instrument, infrared measuring instrument, ultraviolet measuring instrument, electromagnetic wave measuring instrument, thermometer, hygrometer, fixed-point AI weather forecast, high-precision multi-channel GPS, low-altitude satellite information, or long-tail event AI data, etc. are listed.

[0212] As an example of the sensor information obtained from the robot sensor 2023, images, distances, vibrations, heat, odors, colors, sounds, ultrasounds, electric waves, ultraviolet rays, infrared rays, humidity, etc. are listed. It is preferable to obtain image and distance information through the robot imaging device 2024. As an example, the robot sensor 2023 (robot imaging device 2024) performs these detections every nanosecond. The sensor information is used, for example, for motion capture of the actions of the operator 400, 3D mapping of the work area 200, and analysis of the movement or actions of the operator 400 in the work area 200, such as navigation, steering, and speed.

[0213] The robot arm 2026 includes a right arm 2261 and a left arm 2262. In addition, the right arm 2261 includes a right gripping support 2263 and a right gripper 2265, and the left arm 2262 includes a left gripping support 2264 and a left gripper 2266. The right gripping support 2263 is a mechanism for supporting the right gripper 2265, and the left gripping support 2264 is a mechanism for supporting the left gripper 2266. As an example, it can be a mechanism that mimics the shape of a human wrist. The grippers 2265 and 2266 are mechanisms for gripping, for example, work components, etc. As an example, it can be a mechanism that mimics the shape of a human hand.

[0214] The robot arm 2026 constitutes the second drive mechanism. The structure of the drive mechanism is not particularly limited. For example, when the robot arm 2026 mimics the shape of a human, a structure is adopted in which servo motors are provided at each joint part such as the part corresponding to the human shoulder, the part corresponding to the elbow, the part corresponding to the wrist, and the part corresponding to the knuckle, and each is rotated by a specified angle.

[0215] It should be noted that, for example, the humanoid robot 2020 may also be provided with sensors in the robot trunk 2211 (refer to Figure 13B ). In this case, the height positions of the sensors are different from those of the robot sensors 2023 provided on the robot head 2212. Due to the different height positions, the sensors can sense the actions of the operator 400 from different angles.

[0216] Return Figure 6A , when the first sensor 2023a (the first imaging device 2024a) and the second sensor 2023b (the second imaging device 2024b), which are robot sensors (robot imaging devices), sense the actions of the operator 400, the sensing system causes the moving mechanism or the drive mechanism of each humanoid robot 2020a, 2020b to operate, so that the second sensor 2023b (the second imaging device 2024b) senses the specified actions of the operator 400 from a position different from that of the first sensor 2023a (the first imaging device 2024a), and the sensing areas 2230a, 2230b (the imaging areas 2240a, 2240b) of the respective sensors sense different specified parts of the operator 400. Among them, the sensing areas 2230a, 2230b (the imaging areas 2240a, 2240b) of the respective sensors do not need to be completely different, as long as different specified parts are set in a part of the area. As the specified parts, for example, the head, wrist, and hand of the operator are listed. In addition, for the recognition of the specified parts by each sensor, either known image recognition technology can be used, or the specified parts can be recognized through the learning of the learning unit 2663 (refer to Figure 9 ).

[0217] In the present embodiment, the moving mechanism 2022 or the drive mechanism of each humanoid robot operates so that the sensing area 2230a (the imaging area 2240a) of the first sensor 2023a (the first imaging device 2024a) provided on the first humanoid robot 2020a senses the left arm of the operator 400, and the sensing area 2230b (the imaging area 2240b) of the second sensor 2023b (the second imaging device 2024b) provided on the second humanoid robot 2020b senses the right arm of the operator 400.

[0218] During the sensing by each sensor, the sensing system determines whether a specified part that can move when the operator 400 performs a specified action is sensed based on the first information acquired by the first sensor 2023a (the first imaging device 2024a) and the second information acquired by the second sensor 2023b (the second imaging device 2024b). There are various cases of specified actions, such as assembly of components or movement of components, painting of products, movement of the operator himself, etc.

[0219] In the case where it is determined that the specified part that can move when the operator 400 performs a specified action is not sensed, the sensing system causes the first moving mechanism 2022a of the first humanoid robot 2020a (refer to Figure 8 ) and / or the second moving mechanism 2022b of the second humanoid robot 2020b (refer to Figure 8 ) to operate so that the specified part is sensed.

[0220] Figure 7 FIG. is an example showing the movement of the mobile robot in the sensing system of the present embodiment.

[0221] As Figure 7 shown, when the operator 400 performs a specified action, for example, when viewed from the back side of the operator 400, there may be a situation where the left arm of the operator 400 is hidden behind the back. Therefore, it is determined whether the specified part that can move when the operator 400 performs a specified action is sensed. In the case where it is determined that the specified part is not sensed, the sensing system causes the first moving mechanism 2022a of the first humanoid robot 2020a to operate so that the specified part is sensed. Thus, the first humanoid robot 2020a can move to a position where it is easy for the first sensor 2023a (the first imaging device 2024a) to sense, and the first sensor 22023a (the first imaging device 2024a) can sufficiently sense the specified part (the left arm in the present embodiment). It should be noted that in Figure 7 , as the specified action of the operator 400 changes, the position of the right arm also changes. Therefore, the second moving mechanism 2022b of the second humanoid robot 2020b (the second sensor 2023b (the second imaging device 2024b)) that senses the right arm is also caused to operate.

[0222] When the first information and the second information are stored, the sensing system learns the specified actions of the operator based on the stored first information and second information. This learning is, for example, performed by automatic learning, which is learning to automatically create a learned model or learning to automatically perform determination / analysis using a learned model.

[0223] The sensing system refers to the learning results, the operation manual information and / or the process sheet information of the operator 400, and generates action information for giving action instructions to the operation robot that operates the operator 400. Thereby, the operation robot can perform human operations. It should be noted that the operation manual information includes, for example, the names and contents of each operation item, the order of the operation items, the information on the standard operation time required for each operation item, etc. In addition, the process sheet information includes, for example, the information indicating the operation time or start time / end time of the whole operation, the information indicating the operation time or start time / end time of each operation item, the information indicating the operator of each operation item, etc.

[0224] Figure 8 It is a block diagram showing an example of the structure and function in the sensing system 2100 of the present embodiment.

[0225] The sensing system 2100 is configured to include a first humanoid robot 2020a, a second humanoid robot 2020b, and a management control device 2060. The first humanoid robot 2020a and the second humanoid robot 2020b are respectively connected to the communication unit 2064 of the management control device 2060 through wireless communication or wired communication, receive instructions from the management control device 2060, and send the information acquired by each sensor. In addition, it is also possible that the first humanoid robot 2020a and the second humanoid robot 2020b are also connected through wireless communication or wired communication to receive and send the information and instructions acquired by each sensor.

[0226] The first humanoid robot 2020a includes a first moving mechanism 2022a, a first sensor 2023a as a robot sensor, a first photographing device 2024a as a robot photographing device included in the first sensor 2023a, a first information processing device 2025a, a first driving mechanism, and a second driving mechanism. In addition, the second humanoid robot 2020b also includes a second moving mechanism 2022b, a second sensor 2023b as a robot sensor, a second photographing device 2024b as a robot photographing device included in the second sensor 2023b, a second information processing device 2025b, and two driving mechanisms. In the present embodiment, the structures of the first humanoid robot 2020a and the second humanoid robot 2020b are the same.

[0227] The first information processing apparatus 2025a according to the present embodiment includes a CPU (Central Processing Unit) 1212, a RAM (Random Access Memory) 1214, and a graphics controller 1216 that are interconnected via a host controller 1210. In addition, the first information processing apparatus 2025a includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. In addition, the first information processing apparatus 2025a includes a ROM (Read Only Memory) 1230 and input / output units such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0228] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 from a frame buffer or the like provided in the RAM 1214 or itself, and causes the image data to be displayed on the display device 1218.

[0229] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the first information processing apparatus 2025a. In addition, the storage device 1224 may also store first information and second information. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to the IC card.

[0230] The ROM 1230 stores therein a boot program or the like executed by the first information processing apparatus 2025a at startup and / or a program dependent on the hardware of the first information processing apparatus 2025a. In addition, the input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, or the like.

[0231] The program is provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The program is read from the computer-readable storage medium, installed in a storage device 1224, a RAM 1214, or a ROM 1230 which are also examples of computer-readable storage media, and executed by a CPU 1212. The information processing described in these programs is read by a first information processing device 2025a, and enables cooperation between the programs and the above-mentioned various types of hardware resources. The device or method may be configured by implementing operations or processing of information according to the use of the first information processing device 2025a.

[0232] For example, when communication is performed between the first information processing device 2025a and an external device, the CPU 1212 may execute a communication program loaded into the RAM 1214, and based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in a transmission buffer area provided in a recording medium such as the RAM 1214, the storage device 1224, the DVD-ROM, or the IC card, and transmits the read transmission data to the network, or writes the received data received from the network to a reception buffer area provided on the recording medium.

[0233] In addition, the CPU 1212 may cause all or a necessary part of a file or a database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. Then, the CPU 1212 may write the processed data back to the external recording medium.

[0234] Various types of information such as various types of programs, data, tables, and databases may be stored in the recording medium to undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214. The various types of processing include various types of operations, information processing, conditional judgments, conditional branches, unconditional branches, information retrieval / replacement, etc. specified by an instruction sequence of a program described throughout this disclosure, and write the result back to the RAM 1214. In addition, the CPU 1212 may retrieve information in files, databases, etc. in the recording medium.

[0235] The programs or software modules described above can be stored on a computer-readable storage medium on or near the first information processing device 2025a. Additionally, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, and thereby a program can be provided to the first information processing device 2025a via the network.

[0236] The same applies to the second information processing device 2025b for what has been described so far.

[0237] The management control device 2060 is a control device that gives instructions to the humanoid robots 2020a and 2020b in order to implement the sensing system 2100. Additionally, the management control device 2060 acquires the sensor information (first information and second information) stored in the storage device 1224.

[0238] The management control device 2060 is composed of a CPU 2060A, a RAM 2060B, a ROM 2060C, an input / output section (I / O) 2060D, a bus 2060E such as a data bus or a control bus that connects them, and a communication section 2068. A storage medium 2062 is connected to the I / O 2060D.

[0239] Additionally, a communication section 2064 is connected to the I / O 2060D, and the communication section 2064 transceives sensor information, operation manual information, process sheet information, etc. between the control system of the humanoid robot 2020.

[0240] Figure 9 It is a block diagram showing an example of the functions of the management control device 2060 in the sensing system of the present embodiment.

[0241] The management control device 2060 includes a storage medium 2062, a communication section 2064, and a processing section 2066.

[0242] The storage medium 2062 includes, for example, at least one of a semiconductor storage device, a magnetic tape device, a magnetic disk device, or an optical disk device. The storage medium 2062 stores a driver program, an operating system program, an application program, data, etc. for processing in the processing section 2066. For example, the storage medium 2062 stores the first information and the second information. Additionally, the storage medium 2062 stores the operation manual information and / or the process sheet information of the operator 400.

[0243] The communication section 2064 has a wireless communication interface circuit such as Wi-Fi (registered trademark) and / or a wired communication interface circuit such as Ethernet (registered trademark). The communication section 2064 transceives various information through the humanoid robots 2020a and 2020b and the interface circuit.

[0244] The processing unit 2066 has one or more processors and their peripheral circuits. The processing unit 2066 is a component that centrally controls the overall operation of the sensing system 2100, such as a CPU. The processing unit 2066 executes processing with reference to programs (driver programs, operating system programs, application programs, etc.) stored in the storage medium 2062. In addition, the processing unit 2066 can execute multiple programs (application programs, etc.) in parallel.

[0245] The processing unit 2066 includes a determination unit 2661, a control unit 2662, a learning unit 2663, and a motion information generation unit 2664. These units are functional modules implemented by programs executed by the processors included in the processing unit 2060. Alternatively, these units can also be implemented as firmware in the processing unit 2066.

[0246] The determination unit 2661 determines whether a specified part that can move when the operator 400 performs a specified action is sensed based on the first information and the second information. In addition, the determination unit 2661 determines whether the specified part sensed by the first sensor 2023a (first imaging device 2024a) is the same as the specified part sensed by the second sensor 2023b (second imaging device 2024b).

[0247] When it is determined by the determination unit 2661 that the specified part of the operator 400 is not sensed, the control unit 2662 operates the first moving mechanism 2022a of the humanoid robot 2020a and / or the second moving mechanism 2022b of the humanoid robot 2020b so that the specified part is sensed. In addition, when it is determined that the specified part sensed by the first sensor 2023a (first imaging device 2024a) is the same as the specified part sensed by the second sensor 2023b (second imaging device 2024b), the control unit 2662 operates the first moving mechanism 2022a of the humanoid robot 2020a and / or the second moving mechanism 2022b of the humanoid robot 2020b so that the specified part sensed by the first sensor 2023a (first imaging device 2024a) is different from the specified part sensed by the second sensor 2023b (second imaging device 2024b).

[0248] The learning unit 2663 learns the specified actions of the operator 400 with reference to the first information and the second information stored in the storage medium 2062 and / or the storage device 1224.

[0249] The motion information generation unit 2664 generates motion information for giving motion instructions to the humanoid robot 2020 that functions as a work robot, referring to the learning result of the learning unit 2663. It should be noted that when generating the motion information, the motion information generation unit 2664 may also refer to the work manual information and / or the process sheet information.

[0250] (Processing of the sensing system according to Embodiment 2 of the present disclosure)

[0251] Figure 10 This is an example of a flowchart showing the processing of the sensing system of the present embodiment.

[0252] First, according to the instruction of the management control device 2060 or according to the reading instruction of the program stored in the storage medium 2062 or the storage device 1224, the information processing device 2025 gives an instruction to cause a plurality of (two in the present embodiment) humanoid robots 2020 that function as mobile robots to move to the work place 200 (step S2101). The movement is performed based on the operation of the robot movement mechanism 2022 of each humanoid robot 2020.

[0253] During the movement, an instruction is given so that each sensing area 2230 (shooting area 2240) of the plurality of robot sensors 2023 (robot shooting devices 2024) senses the operator 400 from different orientations. Such an arrangement of the plurality of humanoid robots 2020 is performed, for example, by previously storing the floor plan of the work place 200 in the storage device 1224 and / or the storage medium 2062 and corresponding the positions of the respective humanoid robots 2020 to the stored floor plan. Alternatively, the arrangement of the humanoid robots 2020 may also be based on the positions obtained by machine learning and optimization.

[0254] Next, a plurality of sensors 2023a, 2023b (a plurality of shooting devices 2024a, 2024b) are used to sense the specified actions on the operation line 201 of the operator 400 (step S2102). In the present embodiment, the control unit 2662 gives an instruction so that the sensing area 2230a (shooting area 2240a) of the first sensor 2023a (first shooting device 2024a) senses the left arm of the operator 400, the sensing area 2230b (shooting area 2240b) of the second sensor 2023b (second shooting device 2024b) senses the right arm of the operator 400, and the movement mechanism 2022 or the drive mechanism of each humanoid robot operates.

[0255] The first information acquired by the first sensor 2023a (first imaging device 2024a) and the second information acquired by the second sensor 2023b (second imaging device 2024b) are stored in the storage medium 2062 via the storage device 1224 and / or the communication unit 2064. The storage device 1224 and the storage medium 2062 function as a storage unit.

[0256] When acquiring the first information and the second information, it is determined whether a specified part of the operator 400 is sensed (step S2103). When it is determined by the determination unit 2661 that the specified part is not sensed, the control unit 2662 operates the first moving mechanism 2022a and / or the second moving mechanism 2022b so that the specified part is sensed.

[0257] The management control device 2060 learns a specified action by referring to the first information and the second information stored in the storage device 1224 and / or the storage medium 2062, in other words, the stored first information and second information, and generates action information for giving an action instruction to the humanoid robot 2020 that functions as a work robot, referring to the learning result (step S2104).

[0258] Figure 11 is a Figure 10 An example of a flowchart of a more detailed process of the specified part sensing determination process shown in step S2103.

[0259] When the first information and the second information are acquired (step S2201), the determination unit 2661 determines whether a specified part that can move when the operator 400 performs a specified action is sensed based on the first information and the second information (steps S2202, S2203). As an example, when receiving an instruction to cause the first sensor 2023a (first imaging device 2024a) to sense the left arm of the operator 400 and the second sensor 2023b (second imaging device 2024b) to sense the right arm of the operator 400, there may be a situation where the left arm or the right arm of the operator 400 is hidden behind the back when performing a specified action. Therefore, the determination unit 2661 refers to the first information and the second information and determines whether the first sensor 2023a (first imaging device 2024a) senses the left arm as the specified part.

[0260] In the case where it is determined that it has not been sensed (S2203 - "No"), the control unit 2662 operates the first moving mechanism 2022a and / or the second moving mechanism 2022b so that a specified part is sensed (step S2206). As an example, the control unit 2662 operates the first moving mechanism 2022a and moves the first humanoid robot 2020a so that the first sensor 2023a (first imaging device 2024a) can sense the left arm, which is a part of the specified part of the operator 400. In addition, the control unit 2662 operates the second moving mechanism 2022b and moves the second humanoid robot 2020b so that the second sensor 2023b (second imaging device 2024b) can sense the other part of the specified part of the operator 400. Thus, it is possible to avoid a situation where the specified part is not sensed due to the specified actions or movements of the operator 400, and sufficient sensing can be performed. In addition, since each sensor senses a part and the other part of the specified part, it is possible to efficiently acquire data (information) required for learning the operation of the operator 400.

[0261] On the other hand, in the case where it is determined that it has been sensed (S2203 - "Yes"), the determination unit 2661 determines whether the specified part sensed by the first sensor 2023a (first imaging device 2024a) is the same as the specified part sensed by the second sensor 2023b (second imaging device 2024b) (steps S2204, S2205). As an example, at the start of sensing, the first sensor 2023a (first imaging device 2024a) senses the left arm of the operator 400, and the second sensor 2023b (second imaging device 2024b) senses the right arm of the operator 400, but there may be a situation where the same specified part (e.g., the back) is sensed due to the specified actions of the operator 400. Therefore, the determination unit 2661 determines whether the specified parts sensed by the respective sensors are the same.

[0262] When it is determined that the specified parts sensed by the respective sensors are the same (S2205 - "Yes"), the control unit 2662 operates the first moving mechanism 2022a and / or the second moving mechanism 2022b so that the specified part sensed by the first sensor 2023a (first imaging device 2024a) is different from the specified part sensed by the second sensor 2023b (second imaging device 2024b) (S2206). As an example, when the first sensor 2023a (first imaging device 2024a) and the second sensor 2023b (second imaging device 2024b) jointly sense the back of the operator 400, the control unit 2662 operates the first moving mechanism 2022a and the second moving mechanism 2022b so that the first sensor 2023a (first imaging device 2024a) senses the left arm of the operator and the second sensor 2023b (second imaging device 2024b) senses the right arm of the operator. Thereby, it is possible to avoid a situation where multiple sensors sense the same specified part, and it is possible to efficiently acquire data required for learning of the operation.

[0263] Figure 12 Shows Figure 10 An example of a flowchart of a more detailed process of the action information generation process shown in step S2104.

[0264] When the first information and the second information are stored in the storage device 1224 and / or the storage medium 2062 (step S2301), the learning unit 2663 refers to the first information and the second information stored in the storage device 1224 and / or the storage medium 2062, and learns the actions of the operator 400 (step S2302). During the learning process, action capture of the actions of the operator 400, analysis of the 3D map of the work area 200, navigation, turning, speed, etc. of the movement or actions of the operator 400 in the work area 200 are performed. Through automatic learning, the learning can also learn the optimal actions of the humanoid robot 2020 that can function as a work robot. Thereby, it is possible to analyze the specified actions of the operator 400 from multiple angles at one time, and it is possible to reduce the time and cost required for action analysis and programming of the operator 400.

[0265] After that, the action information generation unit 2664 refers to the learning result of the learning unit 2663 (step S2303), and generates action information for giving an action instruction to the humanoid robot 2020 that functions as a work robot (step S2304). When generating the action information, the action information generation unit 2664 may also refer to the operation manual information and / or the process sheet information of the sensing object (S2303). By referring to the generated action information, the humanoid robot 2020 that functions as a work robot can perform the operation (specified action) of the operator 400.

[0266] (Effect of the sensing system according to Embodiment 2)

[0267] According to the sensing system 2100 according to the present embodiment, when a specified part that can move when the operator 400 as the sensing object performs a specified action is not sensed, the first moving mechanism 2022a operates to cause the specified part to be sensed. Therefore, it is possible to prevent the situation where the specified part is not sensed due to the specified action of the operator 400, and sufficient sensing can be performed.

[0268] In addition, according to the sensing system 2100 according to the present embodiment, the humanoid robot 2020 as a mobile robot is equipped with a sensor 2023 (imaging device 2024). On the other hand, the humanoid robot 2020 moves to an appropriate position not only according to a pre-stored program or machine learning result but also according to the sensing result. Therefore, compared with the case where a fixed sensor is arranged in the work place 200, the situation of reconfiguring the sensor or increasing the number of sensors does not occur, and a sufficient sensing environment can be achieved.

[0269] In addition, according to the sensing system 2100 according to the present embodiment, multiple (two in the present embodiment) mobile robots each have a moving mechanism. When a specified part that can move when the operator 400 as the sensing object performs a specified action is not sensed, each moving mechanism operates to cause the specified part to be sensed. Therefore, the specified action of the operator 400 can be sensed from multiple different angles.

[0270] In addition, according to the sensing system 2100 according to the present embodiment, for multiple (two in the present embodiment) sensors 2023 (imaging devices 2024), the moving mechanism is operated so that one of the sensors 2023 senses a part of the specified part of the operator 400, and the moving mechanism is operated so that the other sensor (imaging device) senses the other part of the specified part. Therefore, the specified action of the operator 400 is sensed from multiple different angles and for multiple specified parts. Thus, multiple mobile robots can perform sensing while moving collaboratively, and sufficient sensing can be performed.

[0271] In addition, according to the sensing system 2100 according to the present embodiment, sensor information (in the present embodiment, the first information and the second information) acquired by each sensor is stored, the specified actions of the operator 400 are learned with reference to the stored sensor information, and with reference to the learning result, action information for giving an action instruction to the humanoid robot functioning as a work robot is generated. Therefore, action information reflecting sufficient sensing results is generated. Thereby, it is possible to efficiently produce a work robot (humanoid robot 2020) capable of performing the work of the operator 400.

[0272] In addition, according to the sensing system 2100 according to the present embodiment, when generating action information, the action information is generated on the basis of referring to work manual information and / or process table information. The operator 400 does not always perform actions faithful to the work, and sometimes performs unnecessary actions according to the situation, or sometimes omits required actions. Therefore, by referring to the work manual information and the process table information, it is possible to suppress unnecessary or inappropriate specified actions performed by the operator 400 from being reflected in the action information.

[0273] (Modification 1 of Embodiment 2)

[0274] Figure 13A 、 Figure 13B is a diagram showing an example of the sensing system according to Modification 1 of the present embodiment.

[0275] Figure 13A is a diagram showing an example of the system configuration in the sensing system according to Modification 1 of Embodiment 2 according to the present disclosure. In this sensing system, it is characterized in that a torso sensor 2023d (torso imaging device 2024d) equivalent to the second sensor is provided in the humanoid robot 2020c functioning as a mobile robot. In addition, in this sensing system, the management control device 2060 is not essential, and the humanoid robot 2020c can constitute the sensing system alone.

[0276] Figure 13B is a diagram showing Figure 13A an example of the mobile robot shown. The humanoid robot 2020c functioning as a mobile robot includes a robot main body 2021c, a robot moving mechanism 2022c, a head sensor 2023c, a head imaging device 2024c included in the head sensor 2023c, a torso sensor 2023d, a torso imaging device 2024d included in the torso sensor 2023d, an information processing device 2025c, and a robot arm.

[0277] The robot main body 2021c includes a robot torso 2211 and a robot head 2212. The robot torso 2211 and the robot head 2212 constitute the first drive mechanism 2021c (refer to Figure 14 ), and can change the sensing area 2230c (shooting area 2240c) of the head sensor 2023c (head shooting device 2024c) and the sensing area 2230d (shooting area 2240d) of the torso sensor 2023d (torso shooting device 2024d). The robot moving mechanism 2022c functions as the first moving mechanism.

[0278] The head sensor 2023c (head shooting device 2024c) functions as the first sensor, and the torso sensor 2023d (torso shooting device 2024d) functions as the second sensor. Since the head sensor 2023c (head shooting device 2024c) and the torso sensor 2023d (torso shooting device 2024d) are arranged at different height positions, the torso sensor 2023d (torso shooting device 2024d) functioning as the second sensor senses the specified actions of the sensing object from a position different from that of the head sensor 2023c (head shooting device 2024c).

[0279] The structure of the information processing device 2025c is the same as that of the first information processing device 2025a of the first humanoid robot 2200a. The same applies to the robot arm as that of the first humanoid robot 2020a.

[0280] Figure 14 It is a block diagram showing an example of the functions of the mobile robot in this sensing system. In the sensing system 2100', the information processing device 2025c includes an information processing unit 2066c, a communication interface 1222c, and a storage device 1224c. The information processing unit 2066c includes a determination unit 2661c, a control unit 2662c, a learning unit 2663c, and a motion information generation unit 2664c. That is, in the sensing system 2100', the information processing unit 2066c performs the same processing as the processing unit 2066 of the management control device 2060. It should be noted that the information processing device 2025c is configured to be able to communicate with the head sensor 2023c (head shooting device 2024c), the torso sensor 2023d (head shooting device 2024d), the first moving mechanism 2022c, and the first drive mechanism 2021c.

[0281] Since the humanoid robot 2020c of the sensing system 2100' has the information processing unit 2066c in the information processing device 2025c, the humanoid robot 2020c alone constitutes the sensing system.

[0282] Referring to FIG. 13, for example, the control unit 2662c of the humanoid robot 2020c gives an instruction such that the head sensor 2023c (head imaging device 2024c) functioning as a first sensor senses the left arm of the operator 400, and the torso sensor 2023d (torso imaging device 2024d) functioning as a second sensor senses the right arm of the operator 400. Further, based on the sensor information (first information and second information) acquired by each sensor, the determination unit 2661c determines whether a specified part that can move when the operator 400 performs a specified action is sensed. When it is determined that the specified part is not sensed, the control unit 2662c operates the first moving mechanism 2022c and / or the first driving mechanism 2021c so that the specified part is sensed.

[0283] In addition, the determination unit 2661c determines whether the specified part sensed by the head sensor 2023c (head imaging device 2024c) is the same as the specified part sensed by the torso sensor 2023d (torso imaging device 2024d). When it is determined that the specified parts sensed by each sensor (imaging device) are the same, the control unit 2662c operates the first moving mechanism 2022c and / or the first driving mechanism 2021c so that the specified parts sensed by each sensor (imaging device) are different.

[0284] (Function and effect of Modification 1)

[0285] According to this sensing system, since the humanoid robot 2020c can independently form a sensing system, for example, even in a place where communication with the management control device 2060 is impossible, sensing can be sufficiently performed.

[0286] In addition, since this humanoid robot 2020c is equipped with multiple (two in this modification) sensors (imaging devices), for example, even in a narrow place for sensing the operator 400, sensing can be sufficiently performed.

[0287] It should be noted that in this sensing system, the humanoid robot functioning as a mobile robot does not necessarily have to be one, but can also be multiple. In this case, as long as the number of humanoid robots increases, the number of sensors increases in multiples of the number of humanoid robots, and a large amount of sensor information can be acquired at one time.

[0288] (Modification 2 of Embodiment 2)

[0289] Figure 15A 、 Figure 15B FIG. is a diagram showing an example of a sensing system according to Modification 2 of the present embodiment.

[0290] Figure 15AFIG. is a diagram showing an example of the system configuration in the sensing system according to the second modification of Embodiment 2. In this sensing system, it is characterized in that, in addition to the humanoid robot 2020 that functions as a mobile robot, a sensor mounting member 2030 is further provided.

[0291] Figure 15B is a diagram showing Figure 15A an example of the sensor mounting member shown. The sensor mounting member 2030 includes a mounting member main body 2031, a mounting member moving mechanism 2032, a sensor 2033 for the mounting member, and a photographing device 2034 for the mounting member. The sensor mounting member 2030 can be moved by the mounting member moving mechanism 2032 provided below the mounting member main body 2031. However, the mounting member moving mechanism 2032 may not be provided.

[0292] The mounting member main body 2031 is, for example, a rod-shaped or stick-shaped member, and its material is not particularly limited. The length of the mounting member main body 2031 is longer than the height (height) of the humanoid robot 2020, for example, 2.1 meters. Below the mounting member main body 2031, the mounting member moving mechanism 2032 is preferably provided at the lower end, and above the mounting member main body 2031, the sensor 2033 for the mounting member is preferably provided at the upper end.

[0293] The mounting member moving mechanism 2032 has a structure in which, for example, a rotating body such as a caster is provided, and assists the sensor mounting member 2030 to move as the humanoid robot 2020 moves. It should be noted that, in this embodiment, although it is not assumed that the sensor mounting member 2030 will move autonomously, it may also be configured to provide a mounting member control unit (not shown) that gives an instruction to the mounting member moving mechanism 2032, and based on a signal from the mounting member control unit, the mounting member moving mechanism 2032 is activated.

[0294] Also referring to Figure 15A , the sensor 2033 for the mounting member (the photographing device 2034 for the mounting member) that functions as a second sensor is provided above the mounting member main body 2031 to sense the operator 400. An example of the sensor 2033 for the mounting member is the same as that of the sensor 2023 for the robot. In addition, an example of the photographing device 2034 for the mounting member is also the same as an example of the photographing device 2024 for the robot. In addition, an example of the acquired sensor information is also the same as that of the sensor 2023 for the robot, and an example of the sensing time of the sensor information is also the same as that of the sensor 2023 for the robot.

[0295] The imaging device 2034 for the mounting member is included in the sensor 2033 for the mounting member. In addition, the sensor 2033 for the mounting member including the imaging device 2034 for the mounting member is disposed at a position higher than the height (body height) of the humanoid robot 2020. Thus, the sensor 2033 for the mounting member can sense the actions of the operator 400 from a position higher than that of the robot sensor 2023.

[0296] Figure 16 FIG. is an example showing the structure and function in the sensing system according to Modification 2 of Embodiment 2. In the sensing system 2100”, the sensor mounting member 2030 is configured to be able to communicate with the information processing device 2025 of the humanoid robot 2020 wirelessly or wiredly. However, the sensor mounting member 2030 may be configured to be able to communicate with the communication unit 2064 of the management control device 2060 instead of or together with the information processing device 2025. Note that the structures of the humanoid robot 2020 and the management control device 2060 in the sensing system 2100” are the same as those of the humanoid robot and the management control device in the sensing system 2100.

[0297] Also referring to FIG. 15, the humanoid robot 2020 holds the sensor mounting member 2030 by the right holding part 2265 (or the left holding part 2266) which is a part of the robot arm part 2026 constituting the second drive mechanism. The sensor 2033 (imaging device 2034) for the mounting member of the sensor mounting member 2030 can change the sensing area 2330 (imaging area 2340) through the second drive mechanism.

[0298] In the sensing system 2100”, for example, the control unit 2662 gives an instruction such that the robot sensor 2023 (robot imaging device 2224) functioning as the first sensor senses the left arm of the operator 400, and the sensor 2033 (imaging device 2034) for the mounting member functioning as the second sensor senses the right arm of the operator 400. Then, based on the sensor information (first information and second information) acquired by each sensor, the determination unit 2661 determines whether the specified part that can move when the operator 400 performs a specified action is sensed. If it is determined that the specified part is not sensed, the control unit 2662 operates the first moving mechanism 2022 and / or the second drive mechanism 2026 so that the specified part is sensed. In addition, when the determination unit 2661 determines that the specified parts sensed by each sensor (imaging device) are the same, the control unit 2662 operates the first moving mechanism 2022 and / or the second drive mechanism 2026 so that the specified parts sensed by each sensor (imaging device) are different.

[0299] Effect of Modification 2

[0300] According to this sensing system, since the sensor 2033 for the mounting member (the photographing device 2034 for the mounting member) is configured as the second sensor, for example, even in a narrow place for the sensing operator 400, sensing can be sufficiently performed.

[0301] In addition, according to this sensing system, the sensor 2030 for the mounting member (the photographing device 2040 for the mounting member) is arranged at a position higher than the height (body height) of the humanoid robot 2020 on the sensor mounting member 2030. Therefore, the actions of the operator 400 can be sensed from a more top-down position. For example, it is easy to avoid the situation where sensing is difficult due to the back of the operator 400, and data required for learning the operations of the operator 400 can be efficiently obtained.

[0302] It should be noted that in this sensing system, the number of humanoid robots that function as mobile robots may not be one, and the number of sensor mounting members may not be one either. For example, the humanoid robot 2020 that holds two sensor mounting members by two holding parts 2265 and 2266 may have multiple units. In this case, the number of sensors can also be increased, and a large amount of sensor information can be obtained at one time.

[0303] The embodiments of the present disclosure have been described above, but the present disclosure is not limited to the above-described embodiments of the present disclosure, and various modifications or applications can be made without departing from the gist of the present disclosure.

[0304] In the sensing system 2100 of this embodiment, an example of using two mobile robots (humanoid robots) equipped with sensors and mobile mechanisms has been described. However, the number of mobile robots can also be two or more. For example, if there are many mobile robots equipped with sensors and mobile mechanisms, multiple sensors can be arranged so as to sense the specified actions of the operator 400 from different positions, heights, and / or orientations respectively. Thereby, it is easy to obtain various data required for learning the specified actions of the operator 400, and sensing can be performed in a manner that can overall cover each specified part of the operator 400.

[0305] In addition, in the present embodiment, an example in which the first sensor (first imaging device) senses the left arm of the operator and the second sensor (second imaging device) senses the left arm of the operator has been described. However, the specified part to be sensed is not limited to this, and the specified part sensed by each sensor is not limited to this either. For example, it is also possible that the first sensor (first imaging device) senses the fingertips of the right hand of the operator and the second sensor (second imaging device) senses the movement of the neck of the operator.

[0306] In addition, in the present embodiment, an example of learning the specified actions of the operator through automatic learning has been described. However, the learning does not necessarily have to be automatic learning, and other known machine learning methods can also be used, such as deep learning, unsupervised / supervised learning, reinforcement learning, etc.

[0307] In addition, in the present embodiment, an example in which the mobile robot and the work robot are the same humanoid robot has been described. In this case, the mobile robot can be used as the work robot, and the costs and expenses involved in manufacturing the robot can be saved. However, the mobile robot and the work robot can also be different robots.

[0308] In addition, in the present embodiment, the operator (person) has been described as the sensing object. However, it is not limited to this. For example, a robot that can imitate the specified actions of the operator can also be used as the sensing object.

[0309] (Embodiment 3)

[0310] Figure 17A 、 Figure 17B is a diagram for explaining the sensing system.

[0311] Figure 17A is a diagram showing an example of the system configuration in the sensing system according to Embodiment 3 of the present disclosure. In the sensing system, a first humanoid robot 3020a that functions as a mobile robot, a second humanoid robot 3020b, and a third humanoid robot 3020c that functions as a work robot are provided. It should be noted that the number of humanoid robots that function as mobile robots and work robots is not limited to this.

[0312] The first humanoid robot 3020a receives instructions from the management control device 3060 described later (refer to Figure 18 ), or from the first information processing device 3025a provided in the first humanoid robot 3020a (refer to Figure 18) instruction, move near the worker 400 working on the production line 201 in the workplace 200. The sensing system senses the specified actions of the worker 400 through the first robot sensor 3023a (first robot imaging device 3024a) equipped on the first humanoid robot 3020a. There are various specified actions, for example, assembly of components or movement of components, painting of products, movement of the worker himself, etc. It should be noted that when sensing the worker 400, known image recognition technology can be used, or the worker 400 or his specified actions can be recognized through learning based on the learning unit 2663 (refer to Figure 19 ) learning. The same applies to the sensing of the work robot described later.

[0313] The sensing system learns the specified actions of the worker 400 with reference to the first information obtained by the first robot sensor 3023a (first robot imaging device 3024a) that functions as the first sensor. In addition, the sensing system generates motion control information for giving motion instructions to the third humanoid robot 3020c with reference to the learning results of the specified actions.

[0314] The second humanoid robot 3020b receives an instruction from the management control device 3060, or moves near the third humanoid robot 3020c working on the production line 201 in the workplace 200 according to an instruction from the second information processing device provided in the second humanoid robot 3020b. Similarly, the third humanoid robot 3020c receives an instruction from the management control device 3060, or moves near the worker 400 in the workplace 200 according to an instruction from the third information processing device provided in the third humanoid robot 3020c.

[0315] The sensing system operates the third humanoid robot 3020c with reference to the motion control information. In addition, the sensing system senses the robot motion of the third humanoid robot 3020c through the second robot sensor 3023b (second robot imaging device 3024b) equipped on the second humanoid robot 3020b. Thus, in this sensing system, it is possible to confirm the robot motion of the third humanoid robot 3020c that functions as a work robot.

[0316] The sensing system compares the first information with the second information obtained by the second robot sensor 3023b (second robot imaging device 3024b) that functions as the second sensor, and adjusts the motion control information so that the robot motion of the third humanoid robot 3020c approximates the specified actions. Thus, the robot motion of the third humanoid robot 3020c can be adjusted to appropriate motions.

[0317] Figure 17B is a diagram showing Figure 17A an example of the humanoid robot 3020 shown. The humanoid robot 3020 that functions as a mobile robot and a working robot includes a robot main body 3021, a robot moving mechanism 3022, a robot sensor 3023, a robot photographing device 3024 included in the robot sensor 3023, an information processing device 3025, and a robot arm part 3026.

[0318] The humanoid robot 3020 can move by means of the robot moving mechanism 3022 provided below the robot main body 3021. For example, it receives an instruction from the outside of the humanoid robot 3020 such as the management control device 3060, or refers to a program stored in the information processing device 3025, and moves near the working line 201 of the working place 200.

[0319] The robot main body 3021 includes a robot torso 3211 and a robot head 3212. The robot torso 3211 and the robot head 3212 constitute a torso / head drive mechanism, and can change the sensing area 3230 (photographing area 3240) of the robot sensor 3023 (robot photographing device 3024). The structure of the drive mechanism is not particularly limited. For example, it may be configured such that, using a servo motor (not shown), the robot head 3212 rotates a specified angle relative to the robot torso 3211, or the robot torso 3211 rotates a specified angle relative to the robot moving mechanism 3022.

[0320] The robot moving mechanism 3022 is provided below the robot torso 3211, the robot arm part 3026 is provided on the side of the robot torso 3211, and the robot sensor 3023 is provided on the robot head 3212. In addition, the information processing device 3025 is provided inside the robot main body 3021.

[0321] The robot moving mechanism 3022 can have any structure. For example, it can be a structure in which a rotating body driven by a motor is provided, or it can be a structure that mimics the shape of a human foot as the feet. As an example, when the robot moving mechanism 3022 is a structure that mimics the shape of a human foot, a servo motor is provided at a position corresponding to a human joint, and the moving mechanism is constituted by rotating it a specified angle.

[0322] The robot sensor 3023 that functions as the first sensor and the second sensor is preferably disposed on the robot head 3212 to sense the operator 400 or the working robot. In addition, the robot sensor 3023 sequentially obtains information indicating at least the distance and angle between the object being worked on by the humanoid robot 3020 located around the humanoid robot 3020 and the robot arm 3026. As an example of the robot sensor 3023, the highest performance camera, thermal imaging camera, high pixel / long focal length / ultra-wide angle / 360-degree / high performance camera, radar, solid-state LiDAR, LiDAR, multi-color laser coaxial displacement meter, visual recognition, or other various sensor groups can be adopted. These are also an example of the robot imaging device 3024. In addition, in addition to this, as another example of the robot sensor 3023, a vibrometer, hardness tester, micro-vibrometer, ultrasonic measuring instrument, vibration measuring instrument, infrared measuring instrument, ultraviolet measuring instrument, electromagnetic wave measuring instrument, thermometer, hygrometer, fixed-point AI weather forecast, high-precision multi-channel GPS, low-altitude satellite information, or long-tail event AIdata, etc. are listed.

[0323] As an example of the sensor information obtained from the robot sensor 3023, images, distances, vibrations, heat, odors, colors, sounds, ultrasounds, electric waves, ultraviolet rays, infrared rays, humidity, etc. are listed. It is preferable to obtain image and distance information through the robot imaging device 3024. As an example, the robot sensor 3023 (robot imaging device 3024) performs these detections every nanosecond. The sensor information is used, for example, for motion capture of the actions of the operator 400, 3D mapping of the work site 200, and analysis of the movement or actions of the operator 400 in the work site 200, such as navigation, steering, and speed.

[0324] The robot arm 3026 includes a right arm 3261 and a left arm 3262. In addition, the right arm 3261 includes a right grip support 3263 and a right grip 3265, and the left arm 3262 includes a left grip support 3264 and a left grip 3266. The right grip support 3263 is a mechanism for supporting the right grip 3265, and the left grip support 3264 is a mechanism for supporting the left grip 3266. As an example, it can be a mechanism that mimics the shape of a human wrist. The grips 3265 and 3266 are mechanisms for gripping, for example, work components, etc. As an example, it can be a mechanism that mimics the shape of a human hand.

[0325] The robot arm 3026 constitutes an arm driving mechanism. The structure of the driving mechanism is not particularly limited. For example, when the robot arm 3026 imitates the shape of a human, a structure is adopted in which servo motors are provided at each joint part corresponding to the shoulder, elbow, wrist, knuckle, etc. of a human and rotated by a specified angle.

[0326] It should be noted that, for example, the humanoid robot 3020 may also be provided with sensors in the robot torso 3211 (refer to Figure 24B ). In this case, the height positions of the sensors are different from those of the robot sensors 3023 provided in the robot head 3212. Due to the different height positions, the sensors can sense the actions of the operator 400 from different angles.

[0327] Figure 18 It is a block diagram showing an example of the structure and function in the sensing system 3100 of the present embodiment.

[0328] The sensing system 3100 is configured to include a first humanoid robot 3020a, a second humanoid robot 3020b, a third humanoid robot 3020c, and a management control device 3060. The first humanoid robot 3020a, the second humanoid robot 3020b, and the third humanoid robot 3020c are respectively connected to the communication unit 3064 of the management control device 3060 via wireless communication or wired communication, receive instructions from the management control device 3060, and send information acquired by each sensor. It should be noted that the humanoid robots 3020a to 3020c may also be connected to each other via wireless communication or wired communication to transmit and receive information or instructions acquired by each sensor.

[0329] The first humanoid robot 3020a that functions as a mobile robot is provided with a first moving mechanism 3022a, a first robot sensor 3023a that functions as a first sensor, a first robot imaging device 3024a included in the first robot sensor 3023a, a first information processing device 3025a, a first torso / head driving mechanism 3021a, and a first arm driving mechanism 3026a. In the present embodiment, the structures of the second humanoid robot 3020b that functions as a mobile robot and the third humanoid robot 3020c that functions as a working robot are also the same as those of the first humanoid robot 3020a.

[0330] The first information processing apparatus 3025a according to the present embodiment includes a CPU (Central Processing Unit) 1212, a RAM (Random Access Memory) 1214, and a graphics controller 1216 that are interconnected by a host controller 1210. In addition, the first information processing apparatus 3025a includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. In addition, the first information processing apparatus 3025a includes a ROM (Read Only Memory) 1230 and input / output units such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0331] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 from a frame buffer or the like provided in the RAM 1214 or itself, and causes the image data to be displayed on the display device 1218.

[0332] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the first information processing apparatus 3025a. In addition, the storage device 1224 may also store first information and second information. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to the IC card.

[0333] The ROM 1230 stores therein a boot program or the like executed by the first information processing apparatus 3025a at startup and / or a program dependent on the hardware of the first information processing apparatus 3025a. In addition, the input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, or the like.

[0334] The program is provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The program is read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. The information processing described in these programs is read by the first information processing device 3025a, and enables cooperation between the programs and the above-mentioned various types of hardware resources. The device or method may be configured by implementing operations or processing of information according to the use of the first information processing device 3025a.

[0335] For example, when communication is performed between the first information processing device 3025a and an external device, the CPU 1212 may execute a communication program loaded into the RAM 1214, and based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in the transmission buffer area provided in a recording medium such as the RAM 1214, the storage device 1224, the DVD-ROM, or the IC card, and transmits the read transmission data to the network, or writes the received data received from the network to the reception buffer area provided on the recording medium.

[0336] In addition, the CPU 1212 may cause all or a necessary part of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. Then, the CPU 1212 may write the processed data back to the external recording medium.

[0337] Various types of information such as various types of programs, data, tables, and databases may be stored in the recording medium to undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214. The various types of processing include various types of operations, information processing, conditional judgment, conditional branch, unconditional branch, information retrieval / replacement, etc. specified by the instruction sequences of the programs described throughout this disclosure, and write the results back to the RAM 1214. In addition, the CPU 1212 may retrieve information in files, databases, etc. in the recording medium.

[0338] The programs or software modules described above can be stored on the first information processing device 2305a or on a computer-readable storage medium near the first information processing device 3025a. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, and thus the program can be provided via the first information processing device 3025a.

[0339] The content described so far is the same for the information processing devices respectively provided in the second humanoid robot 3020b and the third humanoid robot 3020c.

[0340] The management control device 3060 is a control device that gives instructions to the humanoid robots 3020a to 3020c to implement the control of the sensing system 3100. In addition, the management control device 3060 acquires the sensor information (the first information and the second information) stored in the storage device 1224.

[0341] The management control device 3060 is composed of a CPU 3060A, a RAM 3060B, a ROM 3060C, an input / output unit (I / O) 3060D, a bus 3060E such as a data bus or a control bus that connects them, and a communication unit 3068. A storage medium 3062 is connected to the I / O 3060D.

[0342] In addition, a communication unit 3064 is connected to the I / O 3060D. The communication unit 3064 transmits and receives sensor information or operation manual information, process table information, etc. related to the specified actions of the operator 400 between the control systems of the humanoid robot 3020. The operation manual information includes, for example, the names and contents of each operation item, the order of the operation items, information on the standard operation time required for each operation item, etc. In addition, the process table information includes, for example, information indicating the operation time or start time / end time of the entire operation, information indicating the operation time or start time / end time of each operation item, information indicating the operator of each operation item, etc.

[0343] Figure 19 It is a block diagram showing an example of the functions of the management control device 3060 in the sensing system of the present embodiment.

[0344] The management control device 3060 includes a storage medium 3062, a communication unit 3064, and a processing unit 3066.

[0345] The storage medium 3062 includes, for example, at least one of a semiconductor storage device, a magnetic tape device, a magnetic disk device, or an optical disk device. The storage medium 3062 stores a driver for processing in the processing unit 3066, an operating system program, an application program, data, and the like. For example, the storage medium 3062 stores first information and second information. In addition, the storage medium 3062 stores the operation manual information of the operator 400. It should be noted that the storage medium 3062 may also store process sheet information.

[0346] The communication unit 3064 includes a wireless communication interface circuit such as Wi-Fi (registered trademark) and / or a wired communication interface circuit such as Ethernet (registered trademark). The communication unit 3064 transmits and receives various information through the humanoid robots 3020a to 3020c and the interface circuit.

[0347] The processing unit 3066 includes one or more processors and their peripheral circuits. The processing unit 3066 is a component that centrally controls the overall operation of the sensing system 3100, such as a CPU. The processing unit 3066 executes processing by referring to programs (driver, operating system program, application program, etc.) stored in the storage medium 3062. In addition, the processing unit 3066 can execute multiple programs (application programs, etc.) in parallel.

[0348] The processing unit 3066 includes a determination unit 3661, an adjustment unit 3662, a learning unit 3663, and an action information generation unit 3664. These units are functional modules implemented by programs executed by the processors included in the processing unit 3066. Alternatively, these units may also be implemented as firmware in the processing unit 3066.

[0349] The determination unit 3661 determines whether there is manual information related to a specified action, and if there is operation manual information, determines whether the learning result of the specified action is contrary to the operation manual information.

[0350] The adjustment unit 3662 compares the first information with the second information and adjusts the motion control information so that the robot motion of the third humanoid robot 3020c approximates the specified motion. Preferably, the adjustment unit 3662 adjusts the motion control information during the overlapping period so that the robot motion approximates the specified motion. The overlapping period is the period in which the first sensing period and the second sensing period overlap. Here, the first sensing period is the period in which the first robot sensor 3023a (the first robot imaging device 3024a) that functions as the first sensor senses the operator 400 to obtain the first information. In addition, the second sensing period is the period in which the second robot sensor 3023b (the second robot imaging device 3024b) that functions as the second sensor senses the robot motion of the third humanoid robot 3020c to obtain the second information. That is, the overlapping period is the period in which the acquisition of the first information and the acquisition of the second information are performed simultaneously.

[0351] The learning unit 3663 learns the specified motion of the operator 400 with reference to the first information stored in the storage medium 3062 and / or the storage device 1224. This learning is, for example, performed by automatic learning, which is learning that automatically creates a learned model or automatically performs determination / analysis using a learned model.

[0352] The motion information generation unit 3664 generates motion control information for giving a motion instruction to the third humanoid robot 3020c that functions as a work robot with reference to the learning result of the specified motion of the operator 400 by the learning unit 3663. The motion information generation unit 3664 may also refer to the work manual information when generating the motion control information. Thus, even if there are inappropriate motions in the specified motion of the operator 400, the specified motion is not reflected, and the third humanoid robot 3020c can perform appropriate motions (work).

[0353] Figure 20 This is a diagram showing an example of each sensing period in the present sensing system 3100. In the present sensing system 3100, the first sensor senses and the second sensor senses in such a way that an overlapping period is generated. This sensing is performed according to the instruction of the management control device 3060.

[0354] In the present sensing system 3100, the robot motion of the third humanoid robot 3020c starts from the sensing period of the specified motion of the operator 400 (i.e., the first sensing period), and during the sensing period of the robot motion of the third humanoid robot 3020c (i.e., the second sensing period), the adjustment unit 3662 adjusts the motion control information so that the robot motion approximates the specified motion of the operator 400. Thus, the robot motion of the third humanoid robot 3020c as a work robot can be adjusted on the spot.

[0355] (Processing of the sensing system according to Embodiment 3 of the present disclosure)

[0356] Figure 21 This is an example of a flowchart showing the processing of the sensing system of this embodiment.

[0357] First, according to the instruction of the management control device 3060 or according to the read instruction of the program stored in the storage medium 3062 or the storage device 1224, the information processing device of each humanoid robot 3020 gives an instruction to cause multiple (three in this embodiment) humanoid robots 3020 that function as mobile robots and work robots to move to the work site 200 (step S3101). The movement is performed based on the operation of the robot movement mechanism 3022 of each humanoid robot 3020.

[0358] During the movement, an instruction is given such that the sensing area 3230a (imaging area 3240a) of the first robot sensor 3023a (first robot imaging device 3024a) provided in the first humanoid robot 3020a targets the worker 400, while the sensing area 3230b (imaging area 3240b) of the second robot sensor 3023b (second robot imaging device 3024b) provided in the second humanoid robot 3020b targets the third humanoid robot 3020c. Such an arrangement of multiple humanoid robots 3020 is performed, for example, by storing a floor plan of the work site 200 in advance in the storage device 1224 and / or the storage medium 3062 and making the positions of the respective humanoid robots 3020 correspond to the stored floor plan. Alternatively, the arrangement of the humanoid robots 3020 can also be based on positions optimized by machine learning.

[0359] Next, a specified action on the work line 201 of the worker 400 is sensed by the first robot sensor 3023a (first robot imaging device 3024a) (step S3102). In this embodiment, the processing unit 3066 gives an instruction such that the sensing area 3230a (imaging area 3240a) of the first sensor 3023a (first imaging device 3024a) targets the specified action of the worker 400, and upon receiving this instruction, the first information processing device 3025a operates the first movement mechanism 3022a and the torso / head drive mechanism 3021a of the first humanoid robot 3020a.

[0360] The first information acquired by the first robot using the sensor 3023a (the first robot imaging device 3024a) is stored in the storage medium 3062 through the storage device 1224 and / or the communication unit 3064. The storage device 1224 and the storage medium 3062 function as a storage unit.

[0361] The management control device 3060 learns a specified action by referring to the first information stored in the storage unit, in other words, the stored first information, and generates action control information for giving an action instruction to the third humanoid robot 3020c that functions as a work robot, based on the learning result (step S3103). S3103 is preferably implemented during the first sensing period. Thus, the sensing system 3100 can cause the third humanoid robot 3020c to work from the stage where the operator 400 performs a specified action in order to perform a task.

[0362] The management control device 3060 causes the third humanoid robot 3020c to work by referring to the action control information (step S3104). The third humanoid robot 3020c performs an action according to the action instruction given by the action control information.

[0363] Before and after the robot action (S3104) of the third humanoid robot 3020c based on the action control information, the robot action of the third humanoid robot 3020c is sensed by the second robot sensor 3023b (the second robot imaging device 3024b) (step S3105). Thus, the robot action of the third humanoid robot 3020c can be confirmed.

[0364] In the present embodiment, the processing unit 3066 gives an instruction such that the sensing area 3230b (imaging area 3240b) of the second robot sensor 3023b (the second robot imaging device 3024b) targets the robot action of the third humanoid robot 3020c. Receiving this instruction, the second information processing device of the second humanoid robot 3020b causes the second moving mechanism and the second torso / head drive mechanism of the second humanoid robot 3020b to work. The second information acquired by the second sensor 3023b (the second imaging device 3024b) is stored in the storage unit.

[0365] The management control device 3060 adjusts the action control information so that the robot action approximates the specified action (step S3106). Preferably, step S3106 is implemented during the overlapping period. To achieve this, the management control device 3060 simultaneously acquires the first information by the first sensor and the second information by the second sensor. Thus, the sensing system 3100 can make adjustments from the stage where the operator 400 performs a specified action, so that the robot action of the third humanoid robot 3020c approximates the specified action of the operator 400.

[0366] Figure 22 is a flowchart showing Figure 21 a more detailed process of the action control information generation process shown in step S3103.

[0367] When the first information is stored in the storage unit (step S3201), the learning unit 3663 refers to the first information stored in the storage unit and learns the specified actions of the operator 400 (step S3202). During the learning process, action capture of the actions of the operator 400, analysis of the 3D map of the work site 200, navigation, turning, speed, etc. of the movement or actions of the operator 400 in the work site 200 are performed. Through automatic learning, the learning can also learn the optimal actions of the humanoid robot 3020 that can function as a work robot. Thus, the specified actions of the operator 400 can be analyzed from multiple angles at once, and the time and cost consumed for action analysis and programming of the operator 400 can be reduced.

[0368] Here, the determination unit 3661 determines whether there is manual action information related to the specified action (steps S3203, S3204). In the case where there is no manual action information related to the specified action (S3204 - "No (NO)"), the action information generation unit 3664 refers to the learning result of the specified action by the learning unit 3663 (step S3208) and generates action control information for giving an action instruction to the third humanoid robot 3020c (step S3209). Then, the processing unit 3066 refers to the action control information and makes the third humanoid robot 3020c work (step S3210). Thus, the third humanoid robot 3020c can perform robot actions corresponding to the work (specified actions) of the operator 400.

[0369] On the other hand, when there is manual operation information related to the specified operation (S3204 - "Yes"), the determination unit 3661 determines whether the learning result of the specified operation is contrary to the work manual information (Steps S3205 and S3206). When it is determined by the determination unit 3661 that the learning result of the specified operation is contrary to the work manual information (S3206 - "Yes"), it is possible that the operation of the operator 400 is not suitable for the operation content of the operation items included in the work manual information. Therefore, when it is determined that the learning result of the specified operation is contrary to the work manual information, the action information generation unit 3664 does not adopt the learning result of the specified operation when generating the action control information (Step S3208). In this case, the action information generation unit 3664 refers to the work manual information (Step S3207) and generates the action control information (Step S3209). Then, the processing unit 3066 refers to the action control information and operates the third-person type robot 3020c (Step S3210). Thereby, it is possible to suppress the unnecessary or inappropriate specified operations of the operator 400 from being reflected in the action control information and make the third-person type robot 3020c perform appropriate operations.

[0370] Figure 23 is a diagram showing Figure 21 an example of a flowchart of a more detailed process of the action control information adjustment process shown in Step S3106.

[0371] When the second information is stored in the storage unit (Step S3301), the adjustment unit 3662 compares the first information with the second information (Step S3302) and adjusts the action control information so that the robot operation is approximated to the specified operation (Step S3303). That is, an adjustment is made so that the robot operation of the third-person type robot 3020c is approximated to the specified operation of the operator 400. (Effect of the sensing system according to Embodiment 3)

[0372] According to the sensing system 3100 according to the present embodiment, the first information acquired by the first sensor that senses the specified operation of the operator 400 is compared with the second information acquired by the second sensor that senses the operation robot, and the action control information is adjusted so that the robot operation of the operation robot is approximated to the specified operation. Thereby, while confirming the robot operation of the operation robot, it is possible to adjust the operation of the operation robot to an appropriate operation.

[0373] As an example, in the case where the worker 400 repeats the same specified actions, first, the worker 400 performs the first specified action, and the sensing system 3100 acquires the first information. Then, during the period when the worker 400 performs the second specified action, the sensing system 3100 generates action control information to make the work robot operate. During this period, since the worker 400 also performs the second specified action, the sensing system 3100 generates action control information based on the first information generated according to the second specified action, and adjusts the action control information so that the robot action of the work robot approximates the specified action of the worker 400. By repeating this process, the sensing system 3100 can adjust the robot action so that the robot action of the work robot approximates the specified action of the worker 400.

[0374] In addition, according to the sensing system 3100 according to the present embodiment, since the management control device 3060 simultaneously acquires the first information and the second information, it is possible to provide a system that can adjust the action of the work robot on the spot while comparing the specified actions of the work robot and the worker 400.

[0375] In addition, according to the sensing system 3100 according to the present embodiment, when generating the action control information, the action control information is generated on the basis of referring to the work manual information. The worker 400 does not necessarily always perform actions faithful to the work, and sometimes may perform unnecessary actions according to the situation, or sometimes may omit required actions. Therefore, by referring to the work manual information, appropriate action information can be reflected in the action control information, and the work robot can be adjusted to operate more appropriately.

[0376] Furthermore, according to the sensing system 3100 according to the present embodiment, when generating the action control information, the learning result of the specified action by the learning unit 3663 that is contrary to the work manual information is not used. Thereby, it is possible to suppress unnecessary or inappropriate specified actions performed by the worker 400 from being reflected in the action control information.

[0377] (Modification Example 1 of Embodiment 3)

[0378] Figure 24A 、 Figure 24B is a diagram showing an example of the sensing system according to Modification Example 1 of the present embodiment.

[0379] FIG. 24 is a diagram showing an example of the system configuration in the sensing system according to Modification 1 of Embodiment 3 of the present disclosure. In this sensing system, it is characterized in that in the humanoid robot 3020' that functions as a work robot, the head sensor 3023' (head imaging device 3024”) functions as a first sensor, and the torso sensor 3023” (torso imaging device 3024”) that functions as a second sensor is provided in the humanoid robot 3020'. In addition, in this sensing system, the management control device 3060 is not essential, and the humanoid robot 3020' can independently constitute the sensing system.

[0380] Figure 24B is a diagram showing Figure 24A an example of the work robot shown. The humanoid robot 3020' that functions as a work robot has the same structure as the first humanoid robot 3020a except for having the torso sensor 3023” (torso imaging device 3024”). Specifically, the humanoid robot 3020' includes a robot main body 3021', a robot moving mechanism 3022', a head sensor 3023', a head imaging device 3024' included in the head sensor 3023', a torso sensor 3023”, a torso imaging device 3024” included in the torso sensor 3023”, an information processing device 3025', and a robot arm 3026'.

[0381] The robot main body 3021' includes a robot torso 3211' and a robot head 3212'. The robot torso 3211' and the robot head 3212' constitute a torso / head drive mechanism 3021' (refer to Figure 25 ), and can change the sensing area 3230' (imaging area 3240') of the head sensor 3023' (head imaging device 3024') and the sensing area 3230” (imaging area 3240”) of the torso sensor 3023” (torso imaging device 3024”).

[0382] The head sensor 3023' (the head imaging device 3024') functions as the first sensor, and the torso sensor 3023'' (the torso imaging device 3024'') functions as the second sensor. The torso sensor 3023'' (the torso imaging device 3024'') senses the movement of the robotic arm 3026' of the robot for example during a robotic operation. Since the head sensor 3023' (the head imaging device 3024') and the torso sensor 3023'' (the torso imaging device 3024'') are arranged at different height positions, the torso sensor 3023'' (the torso imaging device 3024'') functioning as the second sensor senses the specified movement of the sensed object from a position different from that of the head sensor 3023' (the head imaging device 3024'). It should be noted that the functions of the head sensor 3023' (the head imaging device 3024') and the torso sensor 3023'' (the torso imaging device 3024'') can be interchanged.

[0383] Figure 25 FIG. is a block diagram showing an example of the functions of the work robot in the sensing system 3100'. In the sensing system 3100', the information processing device 3025' includes an information processing unit 3066', a communication interface 1222', and a storage device 1224'. The information processing unit 3066' includes a determination unit 3661', an adjustment unit 3662', a learning unit 3663', and a motion information generation unit 3664'. That is, in the sensing system 3100', the information processing unit 3066' performs the same processing as the processing unit 3066 of the management control device 3060. It should be noted that the information processing device 3025' is configured to be able to communicate with the head sensor 3023' (the head imaging device 3024'), the torso sensor 3023'' (the head imaging device 3024''), the first moving mechanism 3022', the head / torso drive mechanism 3021', and the arm drive mechanism 3026'.

[0384] Since the humanoid robot 3020' of the sensing system 3100' includes the information processing unit 3066' in the information processing device 3025', the humanoid robot 3020' alone constitutes the sensing system.

[0385] Referring to FIG. 24, for example, the adjustment unit 3662' of the humanoid robot 3020' gives an instruction such that the head sensor 3023' (the head imaging device 3024'), which functions as the first sensor, senses the specified actions of the operator 400, and the torso sensor 3023'' (the torso imaging device 3024''), which functions as the second sensor, senses the arm 3026' of the humanoid robot 3020'. Further, based on the sensor information (the first information and the second information) obtained by each sensor, the learning unit 3663' generates action control information through learning. The adjustment unit 3662' compares the first information with the second information and adjusts the action control information so that the robot actions of the humanoid robot 3020' approximate the specified actions of the operator 400.

[0386] (Function and effect of Modification 1)

[0387] According to this sensing system, since the humanoid robot 3020' can independently form a sensing system, for example, even in a place where communication with the management control device 3060 is not possible, it is possible to adjust the robot actions to appropriate actions while confirming the robot actions of the working robot.

[0388] In addition, since this humanoid robot 3020' is equipped with a plurality of (two in this modification) sensors (imaging devices), for example, even in a narrow place for sensing the operator 400, it is possible to adjust the robot actions to appropriate actions while confirming the robot actions of the working robot.

[0389] It should be noted that in this sensing system, the humanoid robot functioning as the working robot does not necessarily have to be one, and multiple humanoid robots are also possible. In this case, as long as the number of humanoid robots increases, the number of humanoid robots performing the work will increase, and a large amount of work can be processed simultaneously in parallel at one time.

[0390] (Modification 2 of Embodiment 3)

[0391] Figure 26 FIG. is an example showing the system configuration in the sensing system according to Modification 2 of the present embodiment.

[0392] In this sensing system, a first humanoid robot 3020a having the same functions as the humanoid robot 3020' described in Modification 1 senses an operator 400 and a third humanoid robot 3020c that functions as a work robot. Specifically, an instruction is given such that the sensing area 3230a1 (imaging area 3240a1) of the head sensor 3023a1 (head imaging device 3024a1) of the first humanoid robot 3020a targets the third humanoid robot 3020c, and the sensing area 3230a2 (imaging area 3240a2) of the torso sensor 3023a2 (torso imaging device 3024a2) of the first humanoid robot 3020a targets the operator 400. Note that in this sensing system, if the first humanoid robot 3020a and the third humanoid robot 3020c are configured to be able to communicate, the management control device 3060 is not necessarily required. In addition, the sensing areas of the head sensor 3023a1 (head imaging device 3024a1) and the torso sensor 3023a2 (torso imaging device 3024a2) may have a structure opposite to the above description.

[0393] In this sensing system, since the entire third humanoid robot 3020c can be sensed, compared with Modification 1, there is an advantage that it is easier to confirm the robot motion of the third humanoid robot 3020c and the robot motion can be appropriately controlled.

[0394] Note that Modification 1 has an advantage of being easier to configure a sensing system than Modification 2 in a case where there is no space for arranging the first humanoid robot 3020a. In addition, it does not require a communication structure between the first humanoid robot 3020a and the third humanoid robot 3020c, and also has an advantage over Modification 2 in that the humanoid robot can be entirely omitted.

[0395] (Modification 3 of Embodiment 3)

[0396] Figure 27A 、 Figure 27B is a diagram showing an example of the sensing system according to Modification 3 of the present embodiment.

[0397] Figure 27A is a diagram showing an example of the system configuration in the sensing system according to Modification 3 of Embodiment 3. In this sensing system, a first humanoid robot 3020a that functions as a mobile robot holds a sensor mounting member 3030.

[0398] Figure 27B is a diagram showing Figure 27AA diagram of an example of the sensor mounting member shown. The sensor mounting member 3030 includes a mounting member main body 3031, a mounting member moving mechanism 3032, a sensor for the mounting member 3033, and a photographing device for the mounting member 3034. The sensor mounting member 3030 can be moved by the mounting member moving mechanism 3032 provided below the mounting member main body 3031. However, the mounting member moving mechanism 3032 may not be provided.

[0399] The mounting member main body 3031 is, for example, a rod-shaped or stick-shaped member, and its material is not particularly limited. The length of the mounting member main body 3031 is longer than the height (body height) of the humanoid robot 3020, for example, 2.1 meters. Below the mounting member main body 3031, it is preferable to provide the mounting member moving mechanism 3032 at the lower end, and above the mounting member main body 3031, it is preferable to provide the sensor for the mounting member 3033 at the upper end.

[0400] The mounting member moving mechanism 3032 has a structure in which, for example, rotating bodies such as casters are provided, and assists the sensor mounting member 3030 to move as the humanoid robot 3020 moves. It should be noted that in the present embodiment, although it is not assumed that the sensor mounting member 3030 will move autonomously, it may also be configured to provide a mounting member control unit (not shown) that gives an instruction to the mounting member moving mechanism 3032, and based on a signal from the mounting member control unit, the mounting member moving mechanism 3032 is activated.

[0401] Also refer to Figure 27A , the sensor for the mounting member 3033 (the photographing device for the mounting member 3034) that functions as the first sensor is provided above the mounting member main body 3031 to sense the operator 400. An example of the sensor for the mounting member 3033 is the same as that of the sensor for the robot 3023, and an example of the photographing device for the mounting member 3034 is also the same as an example of the photographing device for the robot 3024. In addition, an example of the acquired sensor information is the same as that of the sensor for the robot 3023, and an example of the sensing time of the sensor information is also the same as that of the sensor for the robot 3023.

[0402] The photographing device for the mounting member 3034 is included in the sensor for the mounting member 3033. In addition, the sensor for the mounting member 3033 including the photographing device for the mounting member 3034 is arranged at a position higher than the height (body height) of the humanoid robot 3020. Thus, the sensor for the mounting member 3033 can sense the actions of the operator 400 from a position higher than the sensor for the robot 3023.

[0403] Figure 28FIG. 0 is a diagram showing an example of the structure and functions in the sensing system according to Modification 3 of Embodiment 3. In the sensing system 3100”, the sensor mounting member 3030 is configured to be able to communicate with the first information processing device of the first humanoid robot 3020a wirelessly or wiredly. However, the sensor mounting member 3030 may be configured to be able to communicate with the communication unit 3064 of the management control device 3060 instead of or together with the first information processing device. It should be noted that the structures of the first humanoid robot 3020a, the third humanoid robot 3020c functioning as a work robot, and the management control device 3060 in the sensing system 3100” are the same as those of the humanoid robot and the management control device in the sensing system 3100.

[0404] Also referring to FIG. 27, the first humanoid robot 3020a holds the sensor mounting member 3030 by a right holding part (or left holding part) which is a part of the robot arm portion constituting the arm driving mechanism. The sensor 3033 (imaging device 3034) for the mounting member of the sensor mounting member 3030 can change the sensing area 3330 (imaging area 3340) through the arm driving mechanism.

[0405] In the sensing system 3100”, for example, the processing unit of the management control device 3060 gives an instruction such that the sensor 3033 (imaging device 3034) for the mounting member functioning as the first sensor senses the operator 400, and the first robot sensor 3023a (first robot imaging device 3024a) functioning as the second sensor senses the third humanoid robot 3020c. Then, based on the sensor information (first information and second information) obtained by each sensor, the learning unit of the management control device 3060 generates action control information through learning respectively. The adjustment unit of the management control device 3060 compares the first information with the second information and adjusts the action control information so that the robot actions of the third humanoid robot 3020c approximate the specified actions of the operator 400. It should be noted that the roles (functions as the first sensor and the second sensor) of the sensor 3033 (imaging device 3034) for the mounting member and the first robot sensor 3023a (first robot imaging device 3024a) may be interchangeable. That is, it may be configured to give an instruction such that the first robot sensor 3023a (first robot imaging device 3024a) senses the operator 400 and the sensor 3033 (imaging device 3034) for the mounting member senses the third humanoid robot 3020c.

[0406] (Operation and Effect of Modification 3)

[0407] According to this sensing system, the sensor 3033 for the mounting member (the photographing device 3034 for the mounting member) is configured as the second sensor. Therefore, for example, when sensing the operator 400, even in a narrow space for multiple humanoid robots that function as mobile robots, it is possible to appropriately control while confirming the actions of the working robots.

[0408] In addition, according to this sensing system, the sensor mounting member 3030 is provided with the sensor 3030 for the mounting member (the photographing device 3040 for the mounting member) at a position higher than the height (height) of the humanoid robot 3020. Therefore, it is possible to sense the actions of the operator 400 (or the working robot) from a more overhead position. For example, it is easy to avoid situations where it is difficult to sense through the back of the operator 400 or the working robot, and it is possible to efficiently obtain the data required for learning the operations of the operator 400 or the working robot.

[0409] It should be noted that in this sensing system, the number of humanoid robots that function as mobile robots may not be one, and the number of sensor mounting members may not be only one. For example, it is also possible to have multiple humanoid robots 3020 that hold two sensor mounting members by two holding parts 3265 and 3266. In this case, it is also possible to increase the number of sensors and obtain a large amount of sensor information at one time. In addition, the number of humanoid robots that function as working robots may not be one.

[0410] The embodiments of the present disclosure have been described above, but the present disclosure is not limited to the above-described embodiments of the present disclosure, and various modifications or applications can be made without departing from the gist of the present disclosure.

[0411] In the sensing system 3100 of this embodiment, the following structure has been described: using two mobile robots (humanoid robots) equipped with sensors and moving mechanisms, using one working robot (humanoid robot), and arranging one mobile robot for each operator 400 and each working robot. However, the relationship between the mobile robots and the operator 400 and the working robots is not limited to this. For example, if the number of mobile robots equipped with sensors and moving mechanisms is large, multiple sensors can be arranged to sense the specified actions of the operator 400 and the robot actions of the working robots from different positions, heights, and / or orientations respectively. Thus, it is easy to obtain various data required for learning the specified actions of the operator 400 and the robot actions of the working robots, and it is possible to sense the actions of the operator 400 and the working robots in a way that covers the overall actions.

[0412] In addition, in the present embodiment, an example in which the management control device 3060 simultaneously acquires the first information and the second information has been described. However, the management control device 3060 may also independently acquire the first information and the second information. In other words, for the first sensing period during which the sensing operator 400 is sensed to acquire the first information and the second sensing period during which the robot motion of the third-person type robot 3020c is sensed to acquire the second information, these periods may not overlap. Thus, the work robot and the operator 400 can also be made not to work in parallel at the same time, and flexible adjustment of the robot motion corresponding to the specified motion of the operator 400 can be performed.

[0413] In addition, in the present embodiment, an example in which the specified motion of the operator is learned by automatic learning has been described. However, the learning does not necessarily have to be automatic learning, and other known machine learning may also be used. For example, deep learning, unsupervised learning / supervised learning, reinforcement learning, etc. may also be used.

[0414] In addition, in the present embodiment, an example in which the mobile robot and the work robot are the same humanoid robot has been described. In this case, the mobile robot can be used as the work robot, and the costs involved in manufacturing the robot can be saved. However, the mobile robot and the work robot may also be different robots.

[0415] In addition, in the present embodiment, the operator (human) is used as the sensing object has been described. However, it is not limited to this. For example, a robot that can imitate the specified motion of the operator may also be used as the sensing object.

[0416] In the present embodiment, an example in which the work robot works in the same work area 200 as the operator 400 and near the operator 400 has been described. However, the work robot may not be arranged near the operator, nor may it be arranged in the same work area as the operator.

[0417] (Embodiment 4)

[0418] Figure 29A 、 Figure 29B A diagram for explaining the motion change system.

[0419] Figure 29AFIG. 0 is a diagram showing an example of the system configuration in the operation change system according to Embodiment 4 of the present disclosure. In this operation change system, there are provided a first humanoid robot 4020a and a second humanoid robot 4020b that function as work robots, and a third humanoid robot 4020c that functions as a mobile robot. Note that the number of humanoid robots is not limited to this.

[0420] Each of the humanoid robots 4020a to 4020c receives an instruction from a management control device 4060 (see Figure 31 ) described later, or moves near an operator 400 working on an operation line 201 in an operation site 200 according to an instruction from each information processing device provided in each of the humanoid robots 4020a to 4020c. Further, this operation change system senses a specified action of the operator 400 by a third robot sensor 4023c (third robot imaging device 4024c) provided in the third humanoid robot 4020c. There are various specified actions, and examples include assembly of components, movement of components, painting of products, and movement of the operator himself / herself. Note that when sensing the operator 400, known image recognition technology may be used, or the operator 400 or his / her specified action may be learned and recognized based on a learning unit 4663 (see Figure 32 ).

[0421] This operation change system learns a standard action model corresponding to the specified action of the operator 400 based on the sensing information corresponding to the specified action of the operator 400 obtained by using the third robot sensor 4023c (third robot imaging device 4024c). The standard action model is a model representing the content of the operation item of the operator 400, that is, a set of each action specified in the operation item. Further, this operation change system refers to the standard action model and generates a changed action model in which the execution time of each action in the standard action model is set shorter than the required time of each action when generating the standard action model.

[0422] This operation change system operates the first humanoid robot 4020a and the second humanoid robot 4020b that function as work robots according to an instruction from the management control device 4060 or an instruction from an information processing device provided in each humanoid robot. At this time, this operation change system refers to the changed action model.

[0423] This motion change system refers to a change motion model that is shorter than the execution time of each motion to operate the first humanoid robot 4020a and the second humanoid robot 4020b, thereby improving the operation efficiency of each humanoid robot. As an example, when there are 6 workers on one production line 201 in the workplace 200 and originally 100 products are completed per hour, after configuring 6 humanoid robots on this line 201, they are made to work at 10 times the speed of the standard motion model. In other words, this motion change system refers to a motion change model that operates at one-tenth of the execution time of the motion in the standard motion model to make each humanoid robot work. Thus, 1000 products can be completed per hour.

[0424] It should be noted that in this motion change system, the first humanoid robot 4020a and the second humanoid robot 4020b that function as work robots respectively use the first robot sensor 4023a (the first robot imaging device 4024a) and the second robot sensor 4023b (the second robot imaging device 4024b) they are equipped with to sense their respective robot motions.

[0425] As an example, the first humanoid robot 4020a receives an instruction from the management control device 4060 or, according to the instruction of the first information processing device 4025a, operates the first torso / head drive mechanism 4021a (refer to Figure 31 ), so that the sensing area 4230a (imaging area 4240a) of the first robot sensor 4023a (the first robot imaging device 4024a) targets the first gripping parts 4265a, 4266a of the first humanoid robot 4020a. Similarly, the second humanoid robot 4020b also operates the second torso / head drive mechanism so that the sensing area 4230b (imaging area 4240b) of the second robot sensor 4023b (the second robot imaging device 4024b) targets the second gripping parts 4265b, 4266b of the second humanoid robot 4020b. Thus, it is possible to confirm whether the robot motion of each humanoid robot is the robot motion of the reference change motion model.

[0426] Figure 29B It shows Figure 29A A diagram showing an example of the humanoid robot shown. The humanoid robot 4020 that functions as a work robot and a mobile robot includes a robot main body 4021, a robot moving mechanism 4022, a robot sensor 4023, a robot imaging device 4024 included in the robot sensor 4023, an information processing device 4025, and a robot arm part 4026.

[0427] The humanoid robot 4020 can move by means of a robot moving mechanism 4022 provided below the robot main body 4021. For example, it receives instructions from the outside of the humanoid robot 4020 such as the management control device 4060, or refers to a program stored in the information processing device 4025 and moves near the operation line 201 of the work place 200.

[0428] The robot main body 4021 includes a robot torso 4211 and a robot head 4212. The robot torso 4211 and the robot head 4212 constitute a torso / head drive mechanism that can change the sensing area 4230 (shooting area 4240) of the robot sensor 4023 (robot shooting device 4024). The structure of the drive mechanism is not particularly limited. For example, it may be configured such that, using a servo motor (not shown), the robot head 4212 rotates a predetermined angle relative to the robot torso 4211, or the robot torso 4211 rotates a predetermined angle relative to the robot moving mechanism 4022.

[0429] A robot moving mechanism 4022 is provided below the robot torso 4211, a robot arm 4026 is provided on the side of the robot torso 4211, and a robot sensor 4023 is provided on the robot head 4212. In addition, an information processing device 4025 is provided inside the robot main body 4021.

[0430] The robot moving mechanism 4022 can have any structure. For example, it can be a structure in which a rotating body driven by a motor is provided, or it can be a structure that mimics the shape of a human foot as the feet. As an example, in the case where the robot moving mechanism 4022 is configured to mimic the shape of a human foot, a servo motor is provided at a position corresponding to a human joint, and the moving mechanism is constituted by rotating it by a predetermined angle.

[0431] The robot sensor 4023 is preferably disposed on the robot head 4212 to sense the actions of the operator 400, other working robots, or the humanoid robot 4020. Preferably, it senses the actions of the robot arm 4026, and more preferably, it senses the actions of the gripping parts 4255 and 4256. In addition, the robot sensor 4023 sequentially obtains information indicating at least the distance and angle between the object being operated by the humanoid robot 4020 located around the humanoid robot 4020 and the robot arm 4026. As an example of the robot sensor 4023, the highest-performance camera, thermal imaging camera, high-pixel / long-focal-length / ultra-wide-angle / 360-degree / high-performance camera, radar, solid-state LiDAR, LiDAR, multi-color laser coaxial displacement gauge, visual recognition, or other various sensor groups can be used. These are also an example of the robot imaging device 4024. In addition, in addition to this, as another example of the robot sensor 4023, a vibrometer, hardness tester, micro-vibrometer, ultrasonic measuring instrument, vibration measuring instrument, infrared measuring instrument, ultraviolet measuring instrument, electromagnetic wave measuring instrument, thermometer, hygrometer, fixed-point AI weather forecast, high-precision multi-channel GPS, low-altitude satellite information, or long-tail event AI data are listed.

[0432] As an example of the sensing information obtained from the robot sensor 4023, images, distances, vibrations, heat, odors, colors, sounds, ultrasonic waves, electric waves, ultraviolet rays, infrared rays, humidity, etc. are listed. Preferably, the robot imaging device 4024 is used to obtain image and distance information. As an example, the robot sensor 4023 (robot imaging device 4024) performs these detections every nanosecond. The sensing information is used, for example, for motion capture of the actions of the operator 400, 3D mapping of the work site 200, and analysis of the movement or actions of the operator 400 in the work site 200, such as navigation, steering, and speed.

[0433] The robot arm 4026 includes a right arm 4261 and a left arm 4262. In addition, the right arm 4261 includes a right gripping support 4263 and a right gripping part 4265, and the left arm 4262 includes a left gripping support 4264 and a left gripping part 4266. The right gripping support 4263 is a mechanism for supporting the right gripping part 4265, and the left gripping support 4264 is a mechanism for supporting the left gripping part 4266. As an example, it can be a mechanism that mimics the shape of a human wrist. The gripping parts 4265 and 4266 are mechanisms for gripping, for example, working parts, etc. As an example, it can be a mechanism that mimics the shape of a human hand.

[0434] The robot arm 4026 constitutes an arm drive mechanism. The structure of the drive mechanism is not particularly limited. For example, when the robot arm 4026 imitates the shape of a human, a structure is adopted in which servo motors are provided at each joint part such as the part corresponding to the human shoulder, the part corresponding to the elbow, the part corresponding to the wrist, and the part corresponding to the knuckle, and each is rotated by a specified angle.

[0435] It should be noted that, for example, the humanoid robot 4020 may also be provided with sensors in the robot torso 4211 (refer to Figure 35B ). In this case, the height positions of these sensors are different from those of the robot sensors 4023 provided in the robot head 4212. Due to the different height positions, the sensors can sense the actions of the operator 400 from different angles.

[0436] Figure 30 It is a diagram showing an example of the relationship between the standard action model and the action change model in this action change system.

[0437] As described above, the standard action model is a model representing the set of each action specified in the operation item and has multiple actions. As an example, there are a total of 26 actions specified in the operation item, and each action is set as action A, action B, action C to action Z.

[0438] In contrast, the action change model is a model in which the execution time of each action in the standard action model is set to be shorter than the required time of each action when generating the standard action model. For example, as the required time of each action in the standard action model, it is set that it takes T A seconds for action A and T B seconds for action B. In this case, in the action change model, for the same action A, a time t A seconds shorter than T A seconds is set as the execution time, and for action B, a time t B seconds shorter than T B seconds is set as the execution time. The same applies to actions C to Z. Thus, the execution time when using the action change model to execute the work robot can be shorter than the case of using the standard action model to execute the work robot.

[0439] Note that in the motion change model, for at least one motion, it is only necessary to set the execution time of this motion to be less than the required time of the motion in the standard motion model. For all the motions included in the standard motion model, it is also possible to set their execution times to be less than the required time of the motions in the standard motion model. In other words, for at least one motion, if the execution time of this motion is set to be less than the required time of the motion in the standard motion model, then the execution times of the motions in the standard motion model are set to be shorter than the required times of the motions when the standard motion model is generated.

[0440] Figure 31 It is a block diagram showing an example of the structure and function in the motion change system 4100 of the present embodiment.

[0441] The motion change system 4100 is configured to include a first humanoid robot 4020a, a second humanoid robot 4020b, a third humanoid robot 4020c, and a management control device 4060. The first humanoid robot 4020a, the second humanoid robot 4020b, and the third humanoid robot 4020c are respectively connected to the communication unit 4064 of the management control device 4060 via wireless communication or wired communication, receive instructions from the management control device 4060, and send information acquired by each sensor. Note that it is also possible that the humanoid robots 4020a to 4020c are also connected to each other via wireless communication or wired communication to transmit and receive information and instructions acquired by each sensor.

[0442] The first humanoid robot 4020a that functions as a work robot includes a first torso / head drive mechanism 4021a, a first robot movement mechanism 4022a, a first robot sensor 4023a, a first robot imaging device 4024a included in the first robot sensor 4023a, a first information processing device 4025a, and a first arm drive mechanism 4026a. In the present embodiment, the structures of the second humanoid robot 4020b that functions as a work robot and the third humanoid robot 4020c that functions as a mobile robot are the same as those of the first humanoid robot 4020a.

[0443] The first information processing device 4025a according to the present embodiment includes a CPU (Central Processing Unit) 1212, a RAM (Random Access Memory) 1214, and a graphics controller 1216 that are interconnected by a host controller 1210. In addition, the first information processing device 25a includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive can be a DVD-ROM drive, a DVD-RAM drive, etc. The storage device 1224 can be a hard disk drive, a solid state drive, etc. In addition, the first information processing device 4025a includes a ROM (Read Only Memory) 1230 and input / output units such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0444] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 obtains image data generated by the CPU 1212 from a frame buffer or the like provided in the RAM 1214 or itself, and causes the image data to be displayed on the display device 1218.

[0445] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the first information processing device 4025a. In addition, the storage device 1224 may also store sensing information. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to the IC card.

[0446] The ROM 1230 stores therein a boot program executed by the first information processing device 4025a at startup, and / or a program dependent on the hardware of the first information processing device 4025a. In addition, the input / output chip 1240 can also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0447] The program is provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The program is read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. The information processing described in these programs is read by the first information processing device 4025a, and enables cooperation between the program and the above-mentioned various types of hardware resources. The device or method may be configured by implementing the operation or processing of information according to the use of the first information processing device 4025a.

[0448] For example, when communication is performed between the first information processing device 4025a and an external device, the CPU 1212 may execute a communication program loaded into the RAM 1214, and based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in the transmission buffer area provided in a recording medium such as the RAM 1214, the storage device 1224, the DVD-ROM, or the IC card, and transmits the read transmission data to the network, or writes the received data received from the network to the reception buffer area provided on the recording medium.

[0449] In addition, the CPU 1212 may cause all or a necessary part of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), or an IC card to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. Then, the CPU 1212 may write the processed data back to the external recording medium.

[0450] Various types of information such as various types of programs, data, tables, and databases may be stored in a recording medium to undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214. The various types of processing include various types of operations, information processing, conditional judgment, conditional branch, unconditional branch, information retrieval / replacement, etc. specified by the instruction sequence of the program described throughout this disclosure, and write the result back to the RAM 1214. In addition, the CPU 1212 may retrieve information in files, databases, etc. in the recording medium.

[0451] The programs or software modules described above can be stored on a computer-readable storage medium on or near the first information processing device 4025a. Additionally, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, and thus the program can be provided to the first information processing device 4025a via the network.

[0452] The content described so far also applies to each information processing device included in the second humanoid robot 4020b and the third humanoid robot 4020c, respectively.

[0453] The management control device 4060 is a control device that gives instructions to each humanoid robot 4020a to 4020c in order to implement the action change system 4100. Additionally, the management control device 4060 acquires the sensing information stored in the storage device of each information processing device.

[0454] The management control device 4060 is composed of a CPU 4060A, a RAM 4060B, a ROM 4060C, an input / output section (I / O) 4060D, a bus 4060E such as a data bus or a control bus that connects them, and a communication section 4068. A storage medium 4062 is connected to the I / O 4060D.

[0455] Additionally, a communication section 4064 is connected to the I / O 4060D. The communication section 4064 transmits and receives sensor information, work manual information, process table information, etc. between the control system of the humanoid robot 4020. The work manual information includes, for example, the names and contents of each work item, the order of work items, information on the standard work time required for each work item, etc. Additionally, the process table information includes, for example, information indicating the work time or start / end time of the entire work, information indicating the work time or start / end time of each work item, information indicating the workers for each work item, etc.

[0456] Figure 32 It is a block diagram showing an example of the functions of the management control device 4060 in the action change system of the present embodiment.

[0457] The management control device 4060 includes a storage medium 4062, a communication section 4064, and a processing section 4066.

[0458] The storage medium 4062 includes, for example, at least one of a semiconductor storage device, a magnetic tape device, a magnetic disk device, or an optical disk device. The storage medium 4062 stores a driver for processing in the processing unit 4066, an operating system program, an application program, data, and the like. For example, the storage medium 4062 stores sensing information. In addition, the storage medium 4062 stores the work manual information and / or process sheet information of the operator 400.

[0459] The communication unit 4064 includes a wireless communication interface circuit such as Wi-Fi (registered trademark) and / or a wired communication interface circuit such as Ethernet (registered trademark). The communication unit 4064 transmits and receives various information through the humanoid robots 4020a, 4020b, and the interface circuit.

[0460] The processing unit 4066 includes one or more processors and their peripheral circuits. The processing unit 4066 is a component that centrally controls the overall operation of the motion change system 4100, such as a CPU. The processing unit 4066 executes processing by referring to the programs (driver, operating system program, application program, etc.) stored in the storage medium 4062. In addition, the processing unit 4066 can execute multiple programs (application programs, etc.) in parallel.

[0461] The processing unit 4066 includes a determination unit 4661, a control unit 4662, a learning unit 4663, and a model generation unit 4664. These units are functional modules implemented by programs executed by the processors included in the processing unit 4066. Alternatively, these units can also be implemented as firmware in the processing unit 4066.

[0462] When there are multiple sensing objects and multiple sensors for sensing, the determination unit 4661 determines whether the multiple sensors respectively sense different sensing objects. When making this determination, known image recognition techniques can be used, and it is also a method of referring to the learning performed by the learning unit 4663.

[0463] The control unit 4662 causes the first humanoid robot 4020a and / or the second humanoid robot 4020b that functions as a work robot to operate by referring to the change action model generated by the model generation unit 4664. Preferably, the control unit 4662 causes the robot movement mechanism, the torso / head drive mechanism, and / or the arm drive mechanism of each humanoid robot to operate by referring to the change action model.

[0464] The learning unit 4663 learns a standard motion model corresponding to the specified motion of the worker 400 based on the sensing information corresponding to the specified motion of the worker 400 obtained by using the third robot sensor 4023c (third robot imaging device 4024c). This learning is performed, for example, by automatic learning, which is learning that automatically creates a learned model or automatically performs determination / analysis using a learned model. Note that the learning unit 4663 may also refer to work manual information and / or process sheet information when generating the standard motion model. Thus, even if there are inappropriate motions in the specified motion of the worker 400, such specified motions will not be reflected, and each humanoid robot 4020 can perform appropriate motions (operations).

[0465] The model generation unit 4664 generates a modified motion model in which the execution time of each motion in the standard motion model is set to be shorter than the required time of each motion when generating the standard motion model, with reference to the standard motion model.

[0466] (Processing of the motion modification system according to Embodiment 4 of the present disclosure)

[0467] Figure 33 This is an example of a flowchart showing the processing of the motion modification system of the present embodiment.

[0468] First, according to an instruction from the management control device 4060 or according to a read instruction of a program stored in the storage medium 4062 or the storage device of the third information processing device of the third humanoid robot 4020c, the third information processing device of the third humanoid robot 4020c gives an instruction to cause the third humanoid robot 4020c that functions as a mobile robot to move to the work site 200 (step S4101). The movement is performed based on the operation of the third robot movement mechanism of the humanoid robot 4020c. Note that movement instructions may also be given to the humanoid robots 4020a and 4020b that function as work robots at this time.

[0469] When moving, an instruction is given such that the sensing area 4230c (imaging area 4240c) of the third robot sensor 4023c (third robot imaging device 4024c) of the third humanoid robot 4020c targets the worker 400. Such a configuration of the third humanoid robot 4020c is performed, for example, by pre-storing a floor plan of the work site 200 in the storage device and / or storage medium 4062 of the third humanoid robot 4020c and correlating the position of the third humanoid robot 4020c with the stored floor plan. Alternatively, the configuration of the third humanoid robot 4020c may be based on a position obtained by machine learning through optimization. The same applies to the configurations of the first humanoid robot 4020a and the second humanoid robot 4020b.

[0470] Next, the third robot sensor 4023c (third robot imaging device 4024c) is used to sense a specified action on the work line 201 of the worker 400 (step S4102). In the present embodiment, the control unit 4662 gives an instruction such that the sensing area 4230c (imaging area 4240c) of the third robot sensor 4023c (third robot imaging device 4024c) targets the worker 400, and the third robot moving mechanism or each drive mechanism of the third humanoid robot 4020c operates.

[0471] The sensing information acquired by the third robot sensor 4023c (third robot imaging device 4024c) is stored in the storage medium 4062 through the storage device and / or communication unit 4064 of the third humanoid robot 4020c. The storage device of each humanoid robot and the storage medium 4062 function as a storage unit.

[0472] Based on the sensing information stored in the storage unit, in other words, based on the stored sensing information, the management control device 4060 learns a standard action model corresponding to the specified action of the worker 400, and with reference to the standard action model, generates a modified action model in which the execution time of each action in the standard action model is set to be shorter than the required time of each action when the standard action model was generated (step S4103). As an example, when the required time of an action in the standard action model is 10 seconds, in the modified action model, the execution time of this action is set to 5 seconds.

[0473] The control unit 4662 causes the first humanoid robot 4020a and the second humanoid robot 4020b, which function as work robots, to work with reference to the generated modified motion model (step S4104). By referring to the generated modified motion model, the first humanoid robot 4020a and the second humanoid robot 4020b can perform the work (prescribed motion) of the worker 400 faster than the prescribed motion. For example, according to one example shown above, in the modified motion model, it is set to operate at half the execution time of the motion in the standard motion model. Therefore, each humanoid robot 4020 can work at twice the speed of the prescribed motion of the worker 400.

[0474] Figure 34 It is an example of a flowchart showing Figure 33 a more detailed process of the modified motion model process shown in step S4103.

[0475] When the sensing information is stored in the storage unit (step S4201), the learning unit 4663 learns the standard motion model corresponding to the prescribed motion of the worker 400 based on the sensing information corresponding to the prescribed motion of the worker 400 obtained using the third robot sensor 4023c (third imaging device 4024c) (step S4202).

[0476] The learning unit 4663 generates a standard motion model based on the learning result (step S4203). It should be noted that the learning unit 4663 may also refer to the work manual information and / or process sheet information of the worker 400 when generating the standard motion model.

[0477] The model generation unit 4664 generates a modified motion model with reference to the standard motion model (step S4204). The modified motion model is a model in which the execution time of each motion in the standard motion model is set to be shorter than the required time of each motion when generating the standard motion model. Therefore, the robot motions of the respective humanoid robots 4020a and 4020b that work with reference to the modified motion model are faster than the robot motions of the same robots that work with reference to the standard motion model.

[0478] (Operation and effect of the motion change system according to Embodiment 4)

[0479] According to the motion change system 4100 according to the present embodiment, since the work robot can be caused to work with reference to the modified motion model in which the required time of each motion when generating the standard motion model is set to be shorter, the work robot can work efficiently.

[0480] In addition, according to the action change system 4100 according to the present embodiment, when generating a standard action model, a standard action model is generated on the basis of referring to operation manual information and / or process sheet information. The operator 400 does not necessarily always perform actions faithful to the operation. Sometimes, unnecessary actions may be performed according to the situation, or sometimes required actions may be omitted. Therefore, by referring to the operation manual information and the process sheet information, it is possible to prevent unnecessary or inappropriate specified actions performed by the operator 400 from being reflected in the action information.

[0481] (Modification Example 1 of Embodiment 4)

[0482] Figure 35A 、 Figure 35B FIG. is a diagram showing an example of an action change system according to Modification Example 1 of the present embodiment.

[0483] Figure 35A FIG. is a diagram showing an example of the system configuration in the action change system according to Modification Example 1 of Embodiment 4 according to the present disclosure. In this action change system, one of the features is that each specified action of a plurality of sensed objects (operators 400a, 400b) is sensed for learning. In addition, another feature is that a torso sensor 4023” (torso imaging device 4024”) is provided in the humanoid robot 4020' that functions as both a mobile robot and a work robot. In addition, in this action change system, the management control device 4060 is not essential, and the humanoid robot 4020' can independently constitute an action change system.

[0484] Figure 35B FIG. shows Figure 35A an example of the humanoid robot shown. The humanoid robot 4020' that functions as both a mobile robot and a work robot includes a robot main body 4021', a robot moving mechanism 4022', a head sensor 4023', a head imaging device 4024' included in the head sensor 4023', a torso sensor 4023”, a torso imaging device 4024” included in the torso sensor 4023”, an information processing device 4025', and a robot arm 4026'.

[0485] The robot main body 4021' includes a robot torso 4211' and a robot head 4212'. The robot torso 4211' and the robot head 4212' constitute a torso / head drive mechanism 4021' (refer to Figure 36 ), and can change the sensing area 4230' (imaging area 4240') of the head sensor 4023' (head imaging device 4024') and the sensing area 4230” (imaging area 4240”) of the torso sensor 4023” (torso imaging device 4024”).

[0486] The head sensor 4023' (head imaging device 4024') and the torso sensor 4023'' (torso imaging device 4024'') are arranged at positions with different height levels. Therefore, the torso sensor 4023'' (torso imaging device 4024'') senses the specified actions of each sensing object (operators 400a, 400b) from a position different from that of the head sensor 4023' (head imaging device 4024').

[0487] Figure 36 It is an example order block diagram showing the functions of the humanoid robot in this motion change system. In the motion change system 4100', the information processing device 4025' includes an information processing unit 4066', a communication interface 1222', and a storage device 1224'. The information processing unit 4066' includes a determination unit 4661', a control unit 4662', a learning unit 4663', and a model generation unit 4664'. That is, in the motion change system 4100', the information processing unit 4066' performs the same processing as the processing unit 4066 of the management control device 4060. It should be noted that the information processing device 4025' is configured to be able to communicate with the head sensor 4023' (head imaging device 4024'), the torso sensor 4023'' (head imaging device 4024''), the torso / head drive mechanism 4021', the robot movement mechanism 4022', and the arm drive mechanism 4026'.

[0488] Since the humanoid robot 4020' of the motion change system 4100' includes an information processing unit 4066' in the information processing device 4025', the motion change system is constituted solely by the humanoid robot 4020'.

[0489] Reference Figure 35A , for example, the control unit 4662' of the humanoid robot 4020' gives an instruction such that the torso sensor 4023'' (torso imaging device 4024'') senses the operator 400a, and the head sensor 4023' (head imaging device 4024') senses the operator 400b. In other words, in this motion change system 4100', there are multiple sensors for respectively sensing multiple different sensing objects, and multiple sensing information corresponding to the specified actions of multiple operators obtained by using the multiple sensors is acquired. It should be noted that the functions of the head sensor 4023' (head imaging device 4024') and the torso sensor 4023'' (torso imaging device 4024'') can also be interchanged. In addition, when each sensor performs sensing, the determination unit 4661' determines whether each sensor senses different objects respectively.

[0490] Figure 37This is an example of a flowchart showing a more detailed process of generating a change action model in the action change system according to Modification 1 of Embodiment 4 related to the present disclosure.

[0491] In this action change system, compared with Embodiment 4, the difference is that in S4102, the specified actions of multiple workers are sensed by multiple sensors. In addition, in this action change system, there is a step S4103' that replaces S4103.

[0492] In step S4103', first, the respective sensing information obtained by the head sensor 4023' (head photographing device 4024') and the torso sensor 4023'' (torso photographing device 4024'') is stored in the storage unit (storage device 1224') (step S4201'). The learning unit 4663' learns the respective specified actions of multiple workers (two in this modification) based on multiple (two in this modification) pieces of sensing information (step S4202').

[0493] In addition, the learning unit 4663' learns multiple (two in this modification) standard action models corresponding to the respective specified actions of multiple workers based on multiple pieces of sensing information (step S4203'). For example, if the operation of worker 400a consists of actions A to M, and on the other hand, the operation of worker 400b consists of actions N to Z, the standard action models are the first standard action model consisting of actions A to M and the second standard action model consisting of actions N to Z. It should be noted that when generating the standard action model, the learning unit 4663' may also refer to operation manual information and / or process table information.

[0494] The model generation unit 4664' generates a change action model by integrating at least a part of the specified actions of multiple workers (step S4204').

[0495] Figure 38 This is an example of a diagram showing the relationship between the standard action model and the action change model in the action change system according to Modification 1 of Embodiment 4 related to the present disclosure.

[0496] For example, in the first standard action model composed of actions A to M, it is assumed that action M is an action for transferring the movement of a certain component. Similarly, in the second standard action model composed of actions N to Z, it is assumed that action N is an action for receiving the movement of the component. In this case, actions M and N are not required when performed by an integrated work robot. Therefore, the model generation unit 4664' integrates a part of the actions in the first standard action model, namely actions A to L, with a part of the actions in the second standard action model, namely actions O to Z, to generate a modified action model composed of actions A to L and actions O to Z. Thus, in this action modification system 4100, when the humanoid robot 4020', which also functions as a work robot, works with reference to the generated modified action model, it is possible to perform the specified actions carried out by multiple workers with one humanoid robot 4020' while omitting unnecessary actions as needed. As a result, the work robot can work efficiently.

[0497] As an example of this action modification system, when initially 100 products were completed per hour by 6 workers on a line 201 in the work site 200, by arranging 3 humanoid robots 4020' on the line 201, 100 products can be completed per hour. In particular, as described above, when multiple workers perform different operations respectively, the humanoid robot 4020' can centralize these operations and perform them while omitting unnecessary actions as needed.

[0498] It should be noted that for this modified action model, the model generation unit 4664' can also generate a second modified action model in which the execution time of each action is set shorter than the required time of each action when generating this modified action model. In this case, for the example shown above, in the modified action model, by referring to the second modified action model in which the actions are set to be executed in half the execution time of the actions when generating this modified action model and making the humanoid robot 4020' work, 200 products can be completed per hour.

[0499] (Function and effect of Modification Example 1)

[0500] According to this action modification system, since the humanoid robot 4020' can independently form an action modification system, for example, even in a place where communication with the management and control device 4060 is not possible, in a work robot that uses a learning model that has learned the operations of workers to perform operations, the work robot can also work efficiently.

[0501] In addition, since the humanoid robot 4020' is equipped with multiple (two in this modification example) sensors (imaging devices), it can perform operation learning of an operator even in a place that is narrow for sensing multiple operators, for example.

[0502] In addition, according to this operation change system, multiple standard operation models corresponding to respective specified operations of multiple operators are learned, and with reference to the standard operation models, a changed operation model obtained by integrating at least a part of the specified operations of the multiple operators is generated. Therefore, the operations (specified operations) of multiple operators can be replaced by operation robots in a smaller number than the number of the operators, and the operation efficiency can be improved. In addition, in this changed operation model, a second changed model in which the execution time of each operation is set to be shorter than the required time of each operation when generating this changed operation model is generated, and with reference to the second changed model, the operation robot is made to work, whereby the operation robot can be made to work more efficiently.

[0503] It should be noted that in this operation change system, the humanoid robot that functions as both a mobile robot and an operation robot does not necessarily have to be one as shown in one example, and multiple units are also possible. In this case, as long as the number of humanoid robots increases, the number of sensors increases in multiples of the number of humanoid robots, a large amount of sensor information can be acquired at once, and since the number of operation robots increases, for example, even when each operation robot performs the same operation, the operation efficiency can also be improved.

[0504] As described above, the embodiments of the present disclosure have been described, but the present disclosure is not limited to the above-described embodiments of the present disclosure, and various modifications or applications can be made without departing from the gist of the present disclosure.

[0505] In the operation change system 4100 of this embodiment, an example in which the number of mobile robots (humanoid robots) that sense the operator 400 is one has been described. However, the number of mobile robots can also be one or more. For example, if there are a large number of mobile robots each having a sensor and a moving mechanism, multiple sensors can be arranged so as to sense the specified operation of the operator 400 from different positions, heights, and / or orientations respectively. Thereby, it becomes easy to acquire various data required for learning the specified operation of the operator 400, and sensing can be performed in such a manner as to cover the respective specified operations of the operator 400 as a whole.

[0506] In addition, in the present embodiment, an example in which a standard motion model is generated after sensing an operator 400 (S4102) has been described. However, from sensing to generating a standard motion model, it is not necessarily required to be executed continuously in time. For example, when storing sensing information (S4201) during sensing by S4102, learning (S4202) with reference to the sensing information may also be performed after a predetermined time (such as 24 hours later or one week later) has elapsed after sensing. The same applies to between each of the steps S4101 to S4105. Additionally, contrary to the above example, S4103 to S4105 may also be executed during the execution of S4102. In this case, during the period when the operator 400 performs a predetermined motion, since the work robot refers to and changes the motion model to perform work, the work efficiency can be further improved.

[0507] In addition, in the present embodiment, an example in which learning of a predetermined motion of an operator is performed by automatic learning has been described. However, the learning does not necessarily have to be automatic learning, and other known machine learning methods may also be used, such as deep learning, unsupervised learning / supervised learning, reinforcement learning, and the like.

[0508] In addition, in the present embodiment, an example in which the mobile robot and the work robot are the same humanoid robot has been described. In this case, the mobile robot can be used as the work robot, and the costs involved in manufacturing the robot can be saved. However, the mobile robot and the work robot may also be different robots.

[0509] In addition, in the present embodiment, an example in which a changed motion model is set to perform a motion with half or one-tenth of the execution time of the motion in the standard motion model has been described. However, this motion change system only needs to be a system that sets the execution time of each motion in the standard motion model to be shorter than the required time of each motion when generating the standard motion model, and there is no particular limitation.

[0510] (Embodiment 5)

[0511] Figure 39A 、 Figure 39B is a diagram for explaining the operation reproduction system.

[0512] In this operation reproduction system, if an abnormal situation such as an accident or a malfunction occurs, the operator 400 is made to perform the actions at the time of the abnormal situation, and the second robot sensor 5023b (second robot imaging device 5024b) senses the specified actions of the operator 400. In addition, this operation reproduction system learns a standard action model corresponding to the specified actions of the operator 400 based on the first sensing information corresponding to the specified actions of the operator 400. The standard action model is a model representing an action that belongs to both the actions corresponding to the specified actions of the operator 400 and the actions designated as actions to be performed in the specified operation item. Then, this operation reproduction system refers to the standard action model and makes the first humanoid robot 5020a, which is an operation reproduction robot, perform the reproduction action one or more times. This operation reproduction system inputs accident or malfunction information and detects the occurrence of an accident or a malfunction based on the second sensing information corresponding to the reproduction action of the first humanoid robot 5020a obtained by using the sensor. Thereby, it is possible to analyze the problem points of the actions of the operator 400 and the shortcomings of the standard actions for the operation of the operator 400.

[0513] Figure 39A It is a diagram showing an example of sensing when making the operator reproduce the action.

[0514] In this operation reproduction system, there are provided a first humanoid robot 5020a that functions as an operation reproduction robot and a second humanoid robot 5020b that functions as a mobile robot. It should be noted that the number of humanoid robots is not limited to two.

[0515] The operation reproduction system senses the actions of the operator 400 through the second robot sensor 5023b (second robot imaging device 5024b) provided in the second humanoid robot 5020b. As an example of the actions of the operator 400, there is cited the action in the case where the operator 400 configures the component 320 at a place different from the originally arrived configuration part 310 when assembling the printed circuit board 300 on the operation line 201. It should be noted that for the sensor to recognize the actions of the operator 400, either known image recognition technology can be used or the learning unit 5663 (refer to Figure 42 ) can be used to learn and recognize the actions of the operator 400. The same also applies to the reproduction actions of the operation reproduction robot described later.

[0516] The second humanoid robot 5020b senses the actions of the operator 400 using the second robot sensor 5023b (the second robot imaging device 5024b). The first sensing information obtained by the second robot sensor 5023b (the second robot imaging device 5024b) is stored in the storage medium 5062 of the management control device 5060 (refer to Figure 41 ) or in the storage device of the second humanoid robot 5020b. Based on the first sensing information corresponding to the specified actions of the operator 400, the management control device 5060 learns the standard action model corresponding to the specified actions of the operator 400. The standard action model is stored in the storage medium 5062 of the management control device 5060, and / or in the storage device 1224 of the first information processing device 5025a of the first humanoid robot 5020a (refer to Figure 41 ) and / or in the storage device of the second information processing device of the second humanoid robot 5020b. It should be noted that the specified actions include various actions before and after the occurrence of an abnormal situation. For example, they include actions such as grasping an object or assembling a component, and actions when handling a tool.

[0517] Figure 39B FIG. is a diagram showing an example of sensing when the operation reproduction robot reproduces an action. The management control device 5060 refers to the stored standard action model and generates an action instruction for operating the first humanoid robot 5020a. The action instruction belongs to both the instruction generated by referring to the standard action model and the instruction for operating the operation reproduction robot (in this embodiment, the first humanoid robot 5020a). Moreover, for example, when the first humanoid robot 5020a is arranged at the reproduction site for reproducing the scene at the time of the occurrence of an abnormal situation, the management control device 5060 refers to the action instruction and operates the first humanoid robot 5020a.

[0518] The management control device 5060 refers to the standard action model and causes the first humanoid robot 5020a to perform a reproduction action at least once or more, preferably multiple times. For example, in the process of general assembly line operations, the operation procedures are clearly defined, and accidents or malfunctions rarely occur. Therefore, when an abnormal situation occurs, if the first humanoid robot 5020a is made to perform only one reproduction action, it may not be possible to detect the occurrence of an accident or malfunction. Therefore, by causing the first humanoid robot 5020a to perform multiple reproduction actions, it is easier to detect the occurrence of an accident or malfunction.

[0519] The management control device 5060 inputs accident or malfunction information and detects the occurrence of an accident or malfunction based on second sensing information corresponding to the reproduction action of the first humanoid robot 5020a obtained using the second robot sensor 5023b (second robot imaging device 5024b). Note that the accident information is, for example, information related to the accident such as when, where, and what kind of behavior was performed with a person. In addition, the malfunction information is information indicating the malfunction in the case where there is an error in the action of the relevant person when an abnormal situation occurs.

[0520] Figure 40 This is a diagram showing an example of a humanoid robot in this operation reproduction system. The humanoid robot 5020 that functions as an operation reproduction robot and a mobile robot includes a robot main body 5021, a robot moving mechanism 5022, a robot sensor 5023, a robot imaging device 5024 that the robot sensor 5023 may include, an information processing device 5025, and a robot arm 5026.

[0521] The humanoid robot 5020 can move by means of the robot moving mechanism 5022 provided below the robot main body 5021, for example, receive instructions from the outside of the humanoid robot 5020 such as the management control device 5060, or refer to a program stored in the information processing device 5025, and move to the work site 200, for example.

[0522] The robot main body 5021 includes a robot torso 5211 and a robot head 5212. The robot torso 5211 and the robot head 5212 constitute a torso / head drive mechanism that can change the sensing area 5230 (imaging area 5240) of the robot sensor 5023 (robot imaging device 5024). The structure of the drive mechanism is not particularly limited. For example, it may be configured such that, using a servo motor (not shown), the robot head 5212 rotates a predetermined angle relative to the robot torso 5211, or the robot torso 5211 rotates a predetermined angle relative to the robot moving mechanism 5022.

[0523] The robot moving mechanism 5022 is provided below the robot torso 5211, the robot arm 5026 is provided on the side of the robot torso 5211, and the robot sensor 5023 is provided on the robot head 5212. In addition, the information processing device 502 is provided inside the robot main body 5021.

[0524] The robot moving mechanism 5022 can be of any structure. For example, it can be a structure with a rotating body driven by a motor, or a structure that mimics the shape of a human foot to serve as the foot part. As an example, when the robot moving mechanism 5022 is set to a structure that mimics the shape of a human foot, a servo motor is provided at a position corresponding to the human joint, and the moving mechanism is formed by rotating it by a specified angle.

[0525] The robot sensor 5023 is preferably provided on the robot head 5212 to sense the actions of the operator 400 or the operation reproduction robot. In addition, the robot sensor 5023 sequentially obtains information indicating at least the distance and angle between the object being operated by the humanoid robot 5020 located around the humanoid robot 5020 and the robot arm 5026. As an example of the robot sensor 5023, the highest-performance camera, thermal imaging camera, high-pixel / long-focus / ultra-wide-angle / 360-degree / high-performance camera, radar, solid-state LiDAR, LiDAR, multi-color laser coaxial displacement meter, visual recognition, or other various sensor groups can be adopted. These are also an example of the robot photographing device 5024. In addition, in addition to this, as another example of the robot sensor 5023, a vibrometer, hardness tester, micro-vibrometer, ultrasonic measuring instrument, vibration measuring instrument, infrared measuring instrument, ultraviolet measuring instrument, electromagnetic wave measuring instrument, thermometer, hygrometer, fixed-point AI weather forecast, high-precision multi-channel GPS, low-altitude satellite information, or long-tail event AI data, etc. are listed.

[0526] As an example of the sensor information obtained from the robot sensor 5023, images, distances, vibrations, heat, odors, colors, sounds, ultrasounds, electric waves, ultraviolet rays, infrared rays, humidity, etc. are listed. It is preferable to obtain image and distance information through the robot photographing device 5024. As an example, the robot sensor 5023 (robot photographing device 5024) performs these detections every nanosecond. The sensor information is used, for example, for action capture of the actions of the operator 400 or the first humanoid robot 5020a, 3D mapping of the work site 200, analysis of the movement or actions of the operator 400 or the first humanoid robot 5020a in the work site 200, such as navigation, steering, speed, etc.

[0527] The robot arm 5026 includes a right arm 5261 and a left arm 5262. Additionally, the right arm 5261 includes a right gripping support portion 5263 and a right gripping portion 5265, and the left arm 5262 includes a left gripping support portion 5264 and a left gripping portion 5266. The right gripping support portion 5263 is a mechanism for supporting the right gripping portion 5265, and the left gripping support portion 5264 is a mechanism for supporting the left gripping portion 5266. As an example, it can be a mechanism that mimics the shape of a human wrist. The gripping portions 5265 and 5266 are mechanisms for gripping, for example, working components, etc. As an example, it can be a mechanism that mimics the shape of a human hand.

[0528] The robot arm 5026 constitutes a second drive mechanism. The structure of the drive mechanism is not particularly limited. For example, when the robot arm 5026 mimics the shape of a human, a structure is adopted in which servo motors are arranged at each joint part such as the part corresponding to the human shoulder, the part corresponding to the elbow, the part corresponding to the wrist, and the part corresponding to the knuckle, and are rotated by a specified angle.

[0529] It should be noted that, for example, the humanoid robot 5020 may also be provided with sensors in the robot trunk 5211 (refer to Figure 46B ). In this case, the height positions of these sensors are different from those of the robot sensors 5023 provided in the robot head 5212. Due to the different height positions, the sensors sense the actions of the operator 400 or the first humanoid robot 5020a from different angles.

[0530] Figure 41 It is a block diagram showing an example of the structure and functions in the operation reproduction system 5100 of the present embodiment.

[0531] The operation reproduction system 5100 is configured to include a first humanoid robot 5020a, a second humanoid robot 5020b, and a management control device 5060. The first humanoid robot 5020a and the second humanoid robot 5020b are respectively connected to the communication unit 5064 of the management control device 5060 via wireless communication or wired communication, receive instructions from the management control device 5060, and send the information acquired by each sensor. It should be noted that the first humanoid robot 5020a and the second humanoid robot 5020b are also connected via wireless communication or wired communication to transmit and receive the information and instructions acquired by each sensor.

[0532] The first humanoid robot 5020a that functions as a task reproduction robot includes a first robot moving mechanism 5022a, a first robot sensor 5023a, a first robot imaging device 5024a included in the first robot sensor 5023a, a first information processing device 5025a, a first torso / head drive mechanism 5021a, and a first arm drive mechanism 5026a. In the present embodiment, the structure of the second humanoid robot 5020b that functions as a mobile robot is also the same as that of the first humanoid robot 5020a.

[0533] The first information processing device 5025a according to the present embodiment includes a CPU (Central Processing Unit) 1212, a RAM (Random Access Memory) 1214, and a graphics controller 1216 that are interconnected via a host controller 1210. In addition, the first information processing device 5025a includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. In addition, the first information processing device 5025a includes a ROM (Read Only Memory) 1230 and input / output units such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0534] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 obtains image data generated by the CPU 1212 from a frame buffer or the like provided in the RAM 1214 or itself, and causes the image data to be displayed on the display device 1218.

[0535] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the first information processing device 5025a. In addition, the storage device 1224 may also store sensing information. The DVD drive reads a DVD-ROM or the like from a program or data and provides it to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to the IC card.

[0536] The ROM 1230 stores therein a boot program and the like executed by the first information processing device 5025a at startup, and / or a program dependent on the hardware of the first information processing device 5025a. In addition, the input / output chip 1240 can also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0537] The program is provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The program is read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. The information processing described in these programs is read by the first information processing device 5025a, and enables cooperation between the programs and the above-mentioned various types of hardware resources. The device or method can be configured by implementing operations or processing of information according to the use of the first information processing device 5025a.

[0538] For example, when communication is performed between the first information processing device 5025a and an external device, the CPU 1212 can execute a communication program loaded into the RAM 1214, and based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in the transmission buffer area provided in a recording medium such as the RAM 1214, the storage device 1224, the DVD-ROM, or the IC card, and transmits the read transmission data to the network, or writes the received data received from the network to the reception buffer area provided on the recording medium.

[0539] In addition, the CPU 1212 can cause all or a necessary part of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. Then, the CPU 1212 can write the processed data back to the external recording medium.

[0540] Multiple types of information such as various types of programs, data, tables, and databases can be stored in a recording medium to undergo information processing. The CPU 1212 can perform various types of processing on the data read from the RAM 1214. The various types of processing include various types of operations, information processing, conditional judgment, conditional branch, unconditional branch, information retrieval / replacement, etc. specified by the instruction sequences of the programs described throughout this disclosure, and write the results back to the RAM 1214. In addition, the CPU 1212 can retrieve information in files, databases, etc. within the recording medium.

[0541] The programs or software modules described above can be stored on a computer-readable storage medium on or near the first information processing device 5025a. In addition, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, and thus the program can be provided to the first information processing device 5025a via the network.

[0542] The content described so far also applies to the second information processing device included in the second humanoid robot 5020b.

[0543] The management control device 5060 is a control device that gives instructions to the humanoid robots 5020a and 5020b in order to implement the operation reproduction system 5100. In addition, the management control device 5060 acquires the sensing information stored in the respective storage devices of the humanoid robots 5020a and 5020b.

[0544] The management control device 5060 is composed of a CPU 5060A, a RAM 5060B, a ROM 5060C, an input / output unit (I / O) 5060D, a bus 5060E such as a data bus or a control bus that connects them, and a communication unit 5064. A storage medium 5062 is connected to the I / O 5060D.

[0545] In addition, a communication unit 5064 is connected to the I / O 5060D. The communication unit 5064 transmits and receives sensor information, operation manual information, process table information, etc. between the control systems of the humanoid robots 5020. The operation manual information includes, for example, the names and contents of each operation item, the order of the operation items, information on the standard operation time required for each operation item, etc. In addition, the process table information includes, for example, information indicating the operation time or start time / end time of the entire operation, information indicating the operation time or start time / end time of each operation item, information indicating the operator of each operation item, etc.

[0546] Figure 42It is a block diagram showing an example of the functions of the management control device 5060 in the operation reproduction system of the present embodiment.

[0547] The management control device 5060 includes a storage medium 5062, a communication unit 5064, and a processing unit 5066.

[0548] The storage medium 5062 includes, for example, at least one of a semiconductor storage device, a magnetic tape device, a magnetic disk device, or an optical disk device. The storage medium 5062 stores a driver program, an operating system program, an application program, data, etc. for processing in the processing unit 5066. For example, the storage medium 5062 stores sensing information. In addition, the storage medium 5062 may store the operation manual information and / or process sheet information of the operator 400.

[0549] The communication unit 5064 has a wireless communication interface circuit such as Wi-Fi (registered trademark) and / or a wired communication interface circuit such as Ethernet (registered trademark). The communication unit 5064 transmits and receives various information through the humanoid robots 5020a, 5020b, and the interface circuit.

[0550] The processing unit 5066 has one or more processors and their peripheral circuits. The processing unit 5066 is a component that centrally controls the overall operation of the operation reproduction system 5100, for example, a CPU. The processing unit 5066 executes processing by referring to the programs (driver program, operating system program, application program, etc.) stored in the storage medium 5062. In addition, the processing unit 5066 can execute multiple programs (application programs, etc.) in parallel.

[0551] The processing unit 5066 includes a determination unit 5661, a control unit 5662, a learning unit 5663, a generation unit 5664, an input unit 5665, and a detection unit 5666. These units are functional modules implemented by programs executed by the processors included in the processing unit 5066. Alternatively, these units may also be implemented as firmware in the processing unit 5066.

[0552] The determination unit 5661 determines whether a sensing object (operator 400 or the first humanoid robot 5020a) is being sensed. As a determination method, known image recognition technology can be used, or it can be a determination method based on learning by the learning unit 5663 (refer to Figure 42 ).

[0553] The control unit 5662 refers to the standard motion model and makes the first humanoid robot 5020a perform a reproduction motion one or more times. In addition, when it is determined that the sensing object (operator 400 or the first humanoid robot 5020a) is not sensed, the control unit 5662 operates the second torso / head drive mechanism or the second robot movement mechanism of the second humanoid robot 5020b.

[0554] Based on the first sensing information corresponding to the specified actions of the operator 400, the learning unit 5663 learns the standard action model corresponding to the specified actions of the operator 400. This learning is performed, for example, through automatic learning, which is learning to automatically create a learned model or learning to automatically perform determination / analysis using a learned model.

[0555] The generation unit 5664 generates a standard action model with reference to the learning result of the learning unit 5663. In addition, the generation unit 5664 generates an action instruction.

[0556] The input unit 5665 inputs accident or malfunction information. This input can be an input from outside the system of the operation reproduction system 5100, or the accident or malfunction information can be stored in advance in the storage medium 5062, the storage medium 1224 of the first information processing device 5025a, and / or the storage medium of the second information processing device, and these information are read according to the instruction of the management control device 5060, thereby performing the input.

[0557] Based on the second sensing information corresponding to the reproduction actions of the first humanoid robot 5020a obtained by using the second robot sensor 5023b (second robot imaging device 5024b), the detection unit 5666 detects the occurrence of an accident or a malfunction. In addition, as will be described later, the detection unit 5666 detects the occurrence of actions different from the operation manual information or the process sheet information based on the second sensing information corresponding to the reproduction actions of the operation reproduction robot obtained by using the sensor. It should be noted that, as an example of the detection object, the difference in the change of each information (data) over time or the significant deviation between the data when comparing the data is cited.

[0558] (Processing of the operation reproduction system according to Embodiment 5 of the present disclosure)

[0559] Figure 43 It is an example of a flowchart showing the processing of the operation reproduction system of the present embodiment.

[0560] First, according to the instruction of the management control device 5060 or according to the reading instruction of the program stored in the storage medium 5062 or the storage device of the second information processing device, the operation reproduction system 5100 uses the second robot sensor 5023b (the second robot imaging device 5024b) of the second humanoid robot 5020b to sense the actions in the work area 200 of the operator 400 (step S5101). The actions of the operator 400 are actions obtained by reproducing the actions actually performed by the operator 400 when an abnormal situation occurs. Preferably, these actions are performed in the work area 200 that faithfully reproduces the situation when an abnormal situation occurs. The control unit 5662 or the second information processing device operates the second torso / head drive mechanism or the second robot movement mechanism of the second humanoid robot 5020b so that the sensing area 5230b (imaging area 5240b) of the second robot sensor 5023b (the second robot imaging device 5024b) of the second humanoid robot 5020b includes the actions of the operator 400.

[0561] The sensing information (first sensing information) acquired by the second robot sensor 5023b (the second robot imaging device 5024b) is stored in the storage medium 5062 through the storage device and / or the communication unit 5064 of the second information processing device. The storage devices of the respective information processing devices and the storage medium 5062 function as a storage unit.

[0562] Based on the first sensing information stored in the storage unit, in other words, based on the stored first sensing information, the management control device 5060 learns a standard action model corresponding to the specified actions of the operator 400, and generates a standard action model with reference to the learning result (step S5102).

[0563] Figure 44 is a diagram showing Figure 43 an example of a flowchart of a more detailed process of the operator action learning / standard action model generation process shown in step S5102.

[0564] When the first sensing information is acquired, the first sensing information is stored in the storage unit (step S5201). At this time, the learning unit 5663 learns a standard action model corresponding to the specified actions of the operator 400 based on the first sensing information (step S5202). Then, the generation unit 5664 generates a standard action model with reference to the learning result of the learning unit 5663 (step S5203).

[0565] It should be noted that during the learning process of the learning department 5663, it is possible to perform motion capture of the actions of the operator 400, analyze the 3D map of the work site 200, and analyze the navigation, turning, speed, etc. of the movement or actions of the operator 400 in the work site 200. It is also possible to learn through automatic learning the optimal actions of the humanoid robot 5020 that can also function as a work reproduction robot. Thus, it is possible to analyze the specified actions of the operator 400 from multiple perspectives at once, and it is possible to reduce the time and cost consumed by the action analysis and programming of the operator 400.

[0566] Return Figure 43 , before and after generating the standard action model, the first humanoid robot 5020a is arranged at a specified position (step S5103). For the arrangement of the first humanoid robot 5020a at the specified position, for example, the following methods are listed: The floor plan of the work site 200, which is an example of the specified position, is pre-stored in the storage unit. Based on corresponding the position of the first humanoid robot 5020a with the stored floor plan, the first moving mechanism 5022a of the first humanoid robot 5020a is operated to move it to this position. Alternatively, the arrangement of the first humanoid robot 5020a can also be based on the position optimized through machine learning. The specified position is preferably the work site 200 that faithfully reproduces the situation when an abnormal situation occurs.

[0567] The control unit 5662 refers to the standard action model and makes the first humanoid robot 5020a perform a reproduction action one or more times (step S5104). In other words, the first humanoid robot 5020a performs a reproduction action one or more times based on the action instruction.

[0568] Before and after the operation (S5104) of the first humanoid robot 5020a, the input unit 5665 inputs accident or malfunction information (step S5105).

[0569] The work reproduction system 5100 uses the second robot sensor 5023b (second robot imaging device 5024b) of the second humanoid robot 5020b to sense the reproduction action of the first humanoid robot 5020a in the work site 200 (step S5106). The sensing information (second sensing information) obtained by the second robot sensor 5023b (second robot imaging device 5024b) is stored in the storage unit.

[0570] The detection unit 5666 detects the occurrence of an accident or malfunction based on the second sensing information obtained by using the second robot sensor 5023b (second robot imaging device 5024b) (step S5107).

[0571] (Effect of the operation reproduction system according to Embodiment 5)

[0572] According to the operation reproduction system 5100 according to the present embodiment, referring to the standard operation model, the operation reproduction robot performs one or more reproduction operations, and based on the second sensing information corresponding to the reproduction operation of the operation reproduction robot, the occurrence of an accident or a malfunction is detected. Thus, on the basis of causing the operation reproduction robot to reproduce the actions of the operator 400 when an abnormal situation occurs, the occurrence of an accident or a malfunction can be detected through the reproduction operation of the operation reproduction robot, and it is easy to identify the cause of the abnormal situation.

[0573] (Modification 1 of Embodiment 5)

[0574] Figure 45 It is an example of a flowchart showing the processing of the operation reproduction system according to Modification 1 of Embodiment 5 related to the present disclosure.

[0575] In the processing of the operation reproduction system according to this modification, the processing up to S5101 to S5104 is the same. On the other hand, S5105 is omitted, and as the processing after S5106, step S5107' is implemented.

[0576] In this operation reproduction system, the operation manual information or process table information of the operator 400 is stored in the storage unit, and the detection unit 5666 detects the occurrence of an action different from the operation manual information or process table information based on the second sensing information corresponding to the reproduction operation of the first humanoid robot 5020a obtained by using the second robot sensor 5023b (second robot photographing device 5024b) (S5107').

[0577] The operation manual information or process table information is information indicating actions or sequences that are originally set as appropriate. Therefore, by comparing the reproduction operation of the first humanoid robot 5020a referring to the standard operation model with the operation manual information or process table information, it is possible to confirm whether there are any deficiencies in the standard operation.

[0578] (Modification 2 of Embodiment 5)

[0579] Figure 46A 、 Figure 46B It is a diagram showing an example of the operation reproduction system according to Modification 2 of the present embodiment.

[0580] Figure 46AThis is a diagram showing an example of the system configuration in the operation reproduction system according to Modification 2 of Embodiment 5 related to the present disclosure. In this operation reproduction system, it is characterized in that a torso sensor 5023” (torso imaging device 5024”) is provided on the second humanoid robot 5020' that functions as a mobile robot. Specifically, an instruction is given such that the sensing area 5230' (imaging area 5240') of the head sensor 5023' (head imaging device 5024') of the second humanoid robot 5020' targets the first humanoid robot 5020a, and the sensing area 5230” (imaging area 5240”) of the torso sensor 5023” (torso imaging device 5023”) of the second humanoid robot 5020' targets the operator 400. It should be noted that in this operation reproduction system, if the first humanoid robot 5020a and the second humanoid robot 5020' are configured to be able to communicate, the management control device 5060 is not necessarily required.

[0581] Figure 46B This is a diagram showing Figure 46A an example of the humanoid robot shown. The second humanoid robot 5020' that functions as a mobile robot includes a robot main body 5021', a robot moving mechanism 5022', a head sensor 5023', a head imaging device 5024' included in the head sensor 5023', a torso sensor 5023”, a torso imaging device 5024” included in the torso sensor 5023”, an information processing device 5025', and a robot arm 5026'.

[0582] The robot main body 5021' includes a robot torso 5211' and a robot head 5212'. The robot torso 5211' and the robot head 5212' constitute a torso / head drive mechanism 5021' (refer to Figure 47 ), and can change the sensing area 5230' (imaging area 5240') of the head sensor 5023' (head imaging device 5024') and the sensing area 5230” (imaging area 5240”) of the torso sensor 5023” (torso imaging device 5024”).

[0583] Since the head sensor 5023' (head imaging device 5024') and the torso sensor 5023” (torso imaging device 5024”) are arranged at different height positions, the torso sensor 5023” (torso imaging device 5024”) and the head sensor 5023' (head imaging device 5024') sense the actions of the operator 400 or the first robot 5020a that functions as an operation reproduction robot from different positions respectively.

[0584] The structure of the information processing device 5025' is the same as that of the first information processing device 5025a of the first humanoid robot 5020a. The same applies to the robot arm 5026'.

[0585] Figure 47 FIG. 4 is a block diagram showing an example of the functions of the humanoid robot 5020' in the present operation reproduction system. In the operation reproduction system 5100', the information processing device 5025' includes an information processing unit 5066', a communication interface 1222', and a storage device 1224'. The information processing unit 5066' includes a determination unit 5661', a control unit 5662c', a learning unit 5663c', a generation unit 5664', an input unit 5665', and a detection unit 5666'. That is, in the operation reproduction system 5100', the information processing unit 5066' performs the same processing as the processing unit 5066 of the management control device 5060. It should be noted that the information processing device 5025' is configured to be able to communicate with the head sensor 5023' (head imaging device 5024'), the torso sensor 5023'' (head imaging device 5024''), the torso / head drive mechanism 5021', the robot movement mechanism 5022', and the arm drive mechanism 5026'. In addition, operation manual information or process table information may also be stored in the storage unit 1224'.

[0586] Since the second humanoid robot 5020' of the operation reproduction system 5100' includes the information processing unit 5066' in the information processing device 5025', it is configured such that the first humanoid robot 5020a and the second humanoid robot 5020' can communicate with each other, and thus an operation reproduction system can be configured without the management control device 5060.

[0587] Referring to FIG. 46, for example, the control unit 5662' of the second humanoid robot 5020' gives an instruction such that the torso sensor 5023'' (torso imaging device 5024'') senses the operator 400, and the head sensor 5023' (head imaging device 5024'), which functions as a second sensor, senses the first humanoid robot 5020a that functions as an operation reproduction robot. It should be noted that the functions of the head sensor 5023' (head imaging device 5024') and the torso sensor 5023'' (torso imaging device 5024'') may be interchanged. That is, it may also be configured to give an instruction such that the head sensor 5023' (head imaging device 5024') senses the operator 400 and the torso sensor 5023'' (torso imaging device 5024'') senses the first humanoid robot 5020a.

[0588] The operation reproduction system 5100' enables an operator 400 to reproduce the actions at the time of an abnormal situation, and senses the specified actions of the operator 400 through a torso sensor 5023” (a torso imaging device 5024”). A learning unit 5663' learns a standard action model corresponding to the specified actions of the operator 400 based on first sensing information corresponding to the specified actions of the operator 400. A control unit 5662' refers to the standard action model and causes a first humanoid robot 5020a to perform a reproduction action one or more times. An input unit 5665' inputs accident or malfunction information. Further, a detection unit 5666' detects the occurrence of an accident or a malfunction based on second sensing information corresponding to the reproduction action of the first humanoid robot 5020a obtained by using a head sensor 5023' (a head imaging device 5024'). Thereby, the first humanoid robot 5020a functioning as an operation reproduction robot reproduces the actions of the operator 400 at the time of an abnormal situation, and the occurrence of an accident or a malfunction can be detected by the reproduction action of the operation reproduction robot, making it easy to identify the cause of the abnormal situation.

[0589] In addition, the operation reproduction system 5100' enables an operator 400 to reproduce the actions at the time of an abnormal situation, and senses the specified actions of the operator 400 through a torso sensor 5023” (a torso imaging device 5024”). A learning unit 5663' learns a standard action model corresponding to the specified actions of the operator 400 based on first sensing information corresponding to the specified actions of the operator 400. A control unit 5662' refers to the standard action model and causes a first humanoid robot 5020a to perform a reproduction action one or more times. A detection unit 5666' detects the occurrence of an action different from operation manual information or process sheet information based on second sensing information corresponding to the reproduction action of the first humanoid robot 5020a obtained by using a head sensor 5023' (a head imaging device 5024'). Thereby, the first humanoid robot 5020a functioning as an operation reproduction robot reproduces the actions of the operator 400 at the time of an abnormal situation, and the drawbacks of the standard actions can be grasped by the reproduction action of the operation reproduction robot.

[0590] (Function and effect of Modification 1)

[0591] According to this operation reproduction system, since the humanoid robot 5020' can independently constitute the operation reproduction system, for example, even in a place where communication with the management control device 5060 is impossible, an operation reproduction system that can easily identify the cause of an abnormal situation can be provided.

[0592] In addition, the humanoid robot 5020' is equipped with multiple (two in this modification example) sensors (imaging devices). Therefore, for example, even in a narrow space where the operator 400 and the operation reproduction robot are arranged side by side for operation reproduction, an operation reproduction system that can easily identify the cause of an abnormal situation can be provided.

[0593] It should be noted that in this operation reproduction system, the humanoid robot that functions as a mobile robot does not necessarily have to be one, but can also be multiple. In this case, as long as the number of humanoid robots increases, the number of sensors increases in multiples of the number of humanoid robots, and a large amount of sensor information can be obtained at one time.

[0594] The embodiments of the present disclosure have been described above, but the present disclosure is not limited to the above-described embodiments of the present disclosure, and various modifications or applications can be made without departing from the gist of the present disclosure.

[0595] In the operation reproduction system 5100 of this embodiment, a structure in which one mobile robot is respectively arranged for the operator 400 or the operation reproduction robot has been described. However, the number of mobile robots can also be one or more. For example, if the number of mobile robots equipped with sensors and a moving mechanism is large, multiple sensors can be arranged so as to sense the specified actions of the operator 400 from different positions, heights, and / or orientations respectively. Thereby, it is easy to obtain various data required for learning the actions of the operator 400 or the operation reproduction robot, and sensing can be performed in a manner that can overall cover each action of the operator 400 or the operation reproduction robot.

[0596] In addition, in the operation reproduction system 5100 of this embodiment, a structure in which one mobile robot (humanoid robot) equipped with a sensor and a moving mechanism senses the operator 400 and the operation reproduction robot respectively has been described. With this structure, the mobile robot can be used for each sensing, and the cost and time required for manufacturing the robot can be shortened. However, the sensing of the operator 400 and the sensing of the operation reproduction robot can also be performed by different mobile robots respectively. Further, each sensing can also be performed by a mobile robot.

[0597] In addition, in this embodiment, an example of learning the specified actions of an operator through automatic learning has been described. However, the learning does not necessarily have to be automatic learning, and other known machine learning methods can also be used, such as deep learning, unsupervised learning / supervised learning, reinforcement learning, etc.

[0598] (Embodiment 6)

[0599] Figure 48A , Figure 48B is a diagram for explaining the job familiarization system.

[0600] Figure 48A is a diagram showing an example of the system configuration in the job familiarization system according to Embodiment 6 of the present disclosure. This job familiarization system includes a first humanoid robot 6020a that functions as a mobile robot and a second humanoid robot 6020b that functions as a job reproduction robot. It should be noted that the number of humanoid robots is not limited to two.

[0601] The first humanoid robot 6020a receives an instruction from a management control device 6060 (refer to Figure 49 ) described later, or moves near a first worker 400a (skilled worker 400a) who is working on the work line 201 in the work place 200 according to an instruction from an information processing device provided in the first humanoid robot 6020a. The skilled worker 400a is a worker who performs a prescribed action as a demonstration. Moreover, this job familiarization system senses the prescribed action of the skilled worker 400a through a first robot sensor 6023a (first robot imaging device 6024a) provided in the first humanoid robot 6020a. It should be noted that there are various cases of prescribed actions, such as assembling or moving parts, painting products, and moving the worker himself.

[0602] When the first robot sensor 6023a (first robot imaging device 6024a) senses the action of the skilled worker 400a, this job familiarization system operates a first moving mechanism or a first torso / head driving mechanism of the first humanoid robot 6020a so that the first robot sensor 6023a (first robot imaging device 6024a) senses the prescribed action of the skilled worker 400a. It should be noted that for the recognition of the prescribed action of the skilled worker 400a by each sensor, known image recognition technology can be used, or the learning unit 6663 (refer to Figure 50 ) can learn to recognize the prescribed action of the skilled worker 400a.

[0603] This job familiarization system stores a standard action model learned based on sensing information (first sensing information) corresponding to the prescribed action of the skilled worker 400a. The standard action model is a model representing an action that belongs to both the action corresponding to the prescribed action of the skilled worker 400a and the action designated as the action that should be performed in a prescribed job item.

[0604] This operation familiarization system refers to the standard action model to reproduce the actions of the second humanoid robot 6020b. Additionally, this operation familiarization system is based on sensing information (second sensing information) corresponding to the actions of the new operator 400b obtained by using a sensor (second robot camera 6024b) that can sense the actions of the second operator 400b (new operator 400b), and detects the differences between the actions of the new operator 400b and the standard action model. The new operator 400b is the operator who is the target of the familiarization operation.

[0605] According to this operation familiarization system, the second humanoid robot 6020b reproduces actions by referring to the standard action model generated based on the specified actions of the proficient operator 400a as a demonstration. Therefore, the second humanoid robot 6020b can perform faithful actions during the operation. That is to say, the reproduced actions of the second humanoid robot 6020b can serve as a demonstration. By making the reproduced actions of the second humanoid robot 6020b a reference for other operators (new operator 400b), operator familiarization with the operation can be achieved.

[0606] Additionally, in this operation familiarization system, the new operator 400b performs the specified actions of the operation, and the actions of the new operator 400b are sensed by the second robot sensor 6023b (second robot camera 6024b) of the second humanoid robot 6020b.

[0607] This operation familiarization system detects the differences between the actions of the new operator 400b and the standard action model based on the sensing information (second sensing information) obtained by the second robot sensor 6023b (second robot camera 6024b). Thereby, the operation of the proficient operator 400a as a demonstration can be compared with the operation of the new operator 400b, and operator familiarization with the operation for the new operator 400b can be achieved.

[0608] As an example, when it is desired to construct an operation line identical to that of a certain domestic factory in another factory (new factory) including overseas factories, through this operation familiarization system, the operation reproduction robot (second humanoid robot 6020b) that has learned the operations of the domestic factory can be sent to the new factory to teach the new operators in the new factory the operations.

[0609] Figure 48B It shows Figure 48AA diagram of an example of the humanoid robot shown. The humanoid robot 6020 that functions as a mobile robot and a task reproduction robot includes a robot main body 6021, a robot moving mechanism 6022, a robot sensor 6023, a robot imaging device 6024 included in the robot sensor 6023, an information processing device 6025, and a robot arm 6026.

[0610] The humanoid robot 6020 can move by the robot moving mechanism 6022 provided below the robot main body 6021. For example, it receives instructions from outside the humanoid robot 6020 such as the management control device 6060, or refers to a program stored in the information processing device 6025, and moves near the work line 201 of the work place 200 or executes a task.

[0611] The robot main body 6021 includes a robot torso 6211 and a robot head 6212. The robot torso 6211 and the robot head 6212 constitute a torso / head drive mechanism that can change the sensing area 6230 (imaging area 6240) of the robot sensor 6023 (robot imaging device 6024). The structure of the drive mechanism is not particularly limited. For example, it can be configured such that, using a servo motor (not shown), the robot head 6212 rotates a predetermined angle relative to the robot torso 6211, or the robot torso 6211 rotates a predetermined angle relative to the robot moving mechanism 22.

[0612] The robot moving mechanism 6022 is provided below the robot torso 6211, the robot arm 6026 is provided on the side of the robot torso 6211, and the robot sensor 6023 is provided on the robot head 6212. In addition, the information processing device 6025 is provided inside the robot main body 6021.

[0613] The robot moving mechanism 6022 can be of any structure. For example, it can be a structure in which a rotating body driven by a motor is provided, or a structure that mimics the shape of a human foot as the feet. As an example, when the robot moving mechanism 6022 is configured to mimic the shape of a human foot, servo motors are provided at positions corresponding to human joints, and the moving mechanism is constituted by rotating them by a predetermined angle.

[0614] The robot sensor 6023 is preferably arranged on the robot head 6212 to sense various operators such as a skilled operator 400a or a new operator 400b. In addition, the robot sensor 6023 sequentially obtains information that at least represents the distance and angle between an object to be operated by the humanoid robot 6020 located around the humanoid robot 6020 and the robot arm 6026. As an example of the robot sensor 6023, the highest-performance camera, thermal imaging camera, high-pixel / long-focus / ultra-wide-angle / 360-degree / high-performance camera, radar, solid-state LiDAR, LiDAR, multi-color laser coaxial displacement gauge, visual recognition, or other various sensor groups can be adopted. These are also an example of the robot photographing device 6024. In addition, in addition to these, as another example of the robot sensor 6023, a vibrometer, hardness tester, micro-vibrometer, ultrasonic measuring instrument, vibration measuring instrument, infrared measuring instrument, ultraviolet measuring instrument, electromagnetic wave measuring instrument, thermometer, hygrometer, fixed-point AI weather forecast, high-precision multi-channel GPS, low-altitude satellite information, or long-tail event AI data are listed.

[0615] As an example of the sensing information obtained from the robot sensor 6023, images, distances, vibrations, heat, odors, colors, sounds, ultrasonic waves, electric waves, ultraviolet rays, infrared rays, humidity, etc. are listed. Preferably, the robot photographing device 6024 is used to obtain the information of images and distances. As an example, the robot sensor 6023 (robot photographing device 6024) performs these senses per nanosecond. The sensing information is used, for example, for motion capture of the actions of each operator, 3D mapping of the work site 200, and analysis of the movement or actions of the operator 400 in the work site 200, such as navigation, steering, and speed.

[0616] The robot arm 6026 includes a right arm 6261 and a left arm 6262. In addition, the right arm 6261 includes a right gripping support portion 6263 and a right gripping portion 6265, and the left arm 6262 includes a left gripping support portion 6264 and a left gripping portion 6266. The right gripping support portion 6263 is a mechanism for supporting the right gripping portion 6265, and the left gripping support portion 6264 is a mechanism for supporting the left gripping portion 6266. As an example, it can be a mechanism that imitates the shape of a human wrist. The gripping portions 6265 and 6266 are mechanisms for gripping, for example, work components, etc. As an example, it can be a mechanism that imitates the shape of a human hand.

[0617] The robot arm 6026 constitutes an arm driving mechanism. The structure of the driving mechanism is not particularly limited. For example, when the robot arm 6026 imitates the shape of a human, a structure is adopted in which servo motors are provided at each joint part corresponding to the shoulder, elbow, wrist, knuckle, etc. of a human and are rotated by a specified angle.

[0618] It should be noted that, for example, the humanoid robot 6020 may also be provided with sensors in the robot trunk 6211 (refer to Figure 54B ). In this case, the height positions of the sensors are different from those of the robot sensors 6023 provided in the robot head 6212. Due to the different height positions, the sensors can sense each operator from different angles.

[0619] Figure 49 FIG. is a block diagram showing an example of the structure and functions in the operation familiarization system 6100 of the present embodiment.

[0620] The operation familiarization system 6100 is configured to include a first humanoid robot 6020a, a second humanoid robot 6020b, and a management control device 6060. The first humanoid robot 6020a and the second humanoid robot 6020b are respectively connected to the communication unit 6064 of the management control device 6060 via wireless communication or wired communication, receive instructions from the management control device 6060, and send information acquired by each sensor. It should be noted that the first humanoid robot 6020a and the second humanoid robot 6020b may also be connected via wireless communication or wired communication to transmit and receive information and instructions acquired by each sensor.

[0621] The second humanoid robot 6020b that functions as an operation reproduction robot includes a second robot moving mechanism 6022b, a second robot sensor 6023b, a second robot photographing device 6024b included in the second robot sensor 6023b, a second information processing device 6025b, a second torso / head driving mechanism 6021b, a second moving mechanism 6022b, and a second arm driving mechanism 6026b. In the present embodiment, the structure of the second humanoid robot 6020b is the same as that of the first humanoid robot 6020a that functions as a mobile robot.

[0622] The second information processing device 6025b according to the present embodiment includes a CPU (Central Processing Unit) 1212, a RAM (Random Access Memory) 1214, and a graphics controller 1216 that are interconnected via a host controller 1210. In addition, the second information processing device 6025b includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive and a DVD-RAM drive. The storage device 1224 may be a hard disk drive and a solid state drive, etc. In addition, the second information processing device 6025b includes a ROM (Read Only Memory) 1230 and input / output units such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0623] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 obtains image data generated by the CPU 1212 from a frame buffer or the like provided in the RAM 1214 or itself, and causes the image data to be displayed on the display device 1218.

[0624] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the second information processing device 6025b. In addition, the storage device 1224 may also store sensing information. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to the IC card.

[0625] The ROM 1230 stores therein a boot program or the like executed by the second information processing device 6025b at startup and / or a program dependent on the hardware of the second information processing device 6025b. In addition, the input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0626] The program is provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The program is read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. The information processing described in these programs is read by the first information processing device 4025a, and enables cooperation between the programs and the above-mentioned various types of hardware resources. The device or method may be configured by implementing operations or processing of information according to the use of the second information processing device 6025b.

[0627] For example, when communication is performed between the second information processing device 6025b and an external device, the CPU 1212 may execute a communication program loaded into the RAM 1214, and based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in the transmission buffer area provided in a recording medium such as the RAM 1214, the storage device 1224, the DVD-ROM, or the IC card, and transmits the read transmission data to the network, or writes the received data received from the network to the reception buffer area provided on the recording medium.

[0628] In addition, the CPU 1212 may cause all or a necessary part of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. Then, the CPU 1212 may write the processed data back to the external recording medium.

[0629] Various types of information such as various types of programs, data, tables, and databases may be stored in a recording medium to undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214. The various types of processing include various types of operations, information processing, conditional judgment, conditional branch, unconditional branch, information retrieval / replacement, etc. specified by the instruction sequence of the program described throughout this disclosure, and write the result back to the RAM 1214. In addition, the CPU 1212 may retrieve information in files, databases, etc. in the recording medium.

[0630] The programs or software modules described above can be stored on a computer-readable storage medium on or near the second information processing device 6025b. Additionally, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, and thereby the program can be provided to the second information processing device 6025b via the network.

[0631] The content described so far is the same for the first information processing device of the first humanoid robot 6020a.

[0632] The management control device 6060 is a control device that gives instructions to the humanoid robots 6020a and 6020b in order to implement the job familiarization system 6100. Additionally, the management control device 6060 acquires the sensing information stored in the respective storage devices of the humanoid robots 6020a and 6020b.

[0633] The management control device 6060 is composed of a CPU 6060A, a RAM 6060B, a ROM 6060C, an input / output unit (I / O) 6060D, a bus 6060E such as a data bus or a control bus that connects them, and a communication unit 6064. The I / O 6060D is connected to a storage medium 6062.

[0634] Additionally, a communication unit 6064 is connected to the I / O 6060D, and the communication unit 6064 transmits and receives sensing information, job manual information, process table information, etc. between the control systems of the humanoid robots 6020. It should be noted that the job manual information includes, for example, information such as the name and content of each job item, the order of job items, and the standard job time required for each job item. Additionally, the process table information includes, for example, information indicating the overall job time or start time / end time of the job, information indicating the job time or start time / end time of each job item, information indicating the operator of each job item, etc.

[0635] Figure 50 It is a block diagram showing an example of the functions of the management control device 6060 in the job familiarization system of the present embodiment.

[0636] The management control device 6060 includes a storage medium 6062, a communication unit 6064, and a processing unit 6066.

[0637] The storage medium 6062 includes, for example, at least one of a semiconductor storage device, a magnetic tape device, a magnetic disk device, or an optical disk device. The storage medium 6062 stores driver programs, operating system programs, application programs, data, etc. for processing in the processing unit 6066. For example, the storage medium 6062 stores sensing information. In addition, the storage medium 6062 stores operation manual information and / or process sheet information of the operator.

[0638] The communication unit 6064 has a wireless communication interface circuit such as Wi-Fi (registered trademark) and / or a wired communication interface circuit such as Ethernet (registered trademark). The communication unit 6064 transmits and receives various information through the humanoid robots 6020a, 6020b, and the interface circuit.

[0639] The processing unit 6066 has one or more processors and their peripheral circuits. The processing unit 6066 is a component that centrally controls the overall operation of the job familiarity system 6100, for example, a CPU. The processing unit 6066 executes processing by referring to programs (driver programs, operating system programs, application programs, etc.) stored in the storage medium 6062. In addition, the processing unit 6066 can execute multiple programs (application programs, etc.) in parallel.

[0640] The processing unit 6066 includes a determination unit 6661, a control unit 6662, a learning unit 6663, a generation unit 6664, and a detection unit 6665. These units are functional modules implemented by programs executed by the processors included in the processing unit 6066. Alternatively, these units can also be implemented as firmware in the processing unit 6066.

[0641] The determination unit 6661 determines whether a sensing target (skilled operator 400a or new operator 400b) is sensed. As a determination method, known image recognition technology can be used, and it is also a method that refers to the learning performed by the learning unit 6663 (reference Figure 51 ).

[0642] The control unit 6662 refers to the standard motion model to make the second robot 6020b perform a reproduction motion. In addition, when it is determined that the sensing target is not sensed, the control unit 6662 operates the torso / head drive mechanism 6021 or the robot movement mechanism 6022 of each humanoid robot 6020.

[0643] The learning unit 6663 learns a standard motion model based on first sensing information corresponding to the specified motions of the skilled worker 400a. The first sensing information is obtained by sensing the skilled worker 400a using the first robot sensor 6023a (the first robot imaging device 6024a). It should be noted that the learning by the learning unit 6663 is, for example, performed by automatic learning, which is learning that automatically creates a learned model or automatically performs determination / analysis using the learned model.

[0644] The generation unit 6664 generates a standard motion model with reference to the learning result of the learning unit 6663. In addition, the generation unit 6664 generates each sensing instruction or alarm instruction described later.

[0645] The detection unit 6665 detects the differences between the motions of the new worker 400b and the standard motion model based on second sensing information corresponding to the motions of the new worker 400b obtained using the second robot sensor 6023b (the second robot imaging device 6024b). In addition, the detection unit 6665 detects the differences between the motions of the new worker 400b and the work manual information or process sheet information based on the sensing information corresponding to the motions of the new worker 400b obtained using the second robot sensor 6023b (the second robot imaging device 6024b). It should be noted that as an example of the detection target, differences in the changes of each piece of information (data) over time or significant deviations between data when comparing data are cited.

[0646] (Processing of the work familiarization system according to Embodiment 6 of the present disclosure)

[0647] Figure 51 This is an example of a flowchart showing the processing of the work familiarization system of the present embodiment. This processing is mainly executed by the processing unit 6066 of the management control device 6060 in cooperation with the elements of the work familiarization system 6100 (the management control device 6060, the first humanoid robot 6020a, and the second humanoid robot 6020b) with reference to a control program stored in advance in the storage medium 6062, the storage device of the first information processing device, and / or the storage device 1224 of the second information processing device 6025b.

[0648] As Figure 51On the premise of the start of the processing described in [reference], according to the instruction of the processing unit 6066 or the instruction to read the program stored in the storage medium 6062 or the storage device of the first information processing device, the first information processing device gives an instruction to cause the first humanoid robot 6020a that functions as a mobile robot to move to the work site 200. The movement is performed based on the operation of the first robot movement mechanism 6022 of the first humanoid robot 6020a.

[0649] During the movement, an instruction is given so that each sensing area 6230 (imaging area 6240) of the first robot sensor 6023a (first robot imaging device 6024a) can sense the specified actions of the skilled operator 400a. The configuration of such a first humanoid robot 6020a is performed, for example, by storing the floor plan of the work site 200 in advance in the storage medium 6062 or the storage device of the first information processing device and making the position a of the first humanoid robot 6020 correspond to the stored floor plan. Alternatively, the configuration of the position a of the first humanoid robot 6020 can also be performed based on the position optimized by machine learning.

[0650] Initially, the management control device 6060 gives an instruction to sense the specified actions on the operation line 201 of the skilled operator 400a using the first robot sensor 6023a (first robot imaging device 6024a) (step S6101). Specifically, the generation unit 6664 generates a first sensing instruction for the purpose of sensing the specified actions of the skilled operator 400a, which causes the first robot sensor 6023a (first robot imaging device 6024a), the first robot movement mechanism, and / or the first torso / head drive mechanism of the first humanoid robot 6020a to operate, and sends the first sensing instruction to the first information processing device through the communication unit 6064. The CPU of the first information processing device receives the first sensing instruction through the communication interface of the first information processing device and starts the program that causes the first robot sensor 6...

Claims

1. A method for adjusting an operation robot, wherein, the method for adjusting the operation robot has the following processes: Moving a mobile robot with a sensor towards the environment where the operator performs actions; Using the sensor to record the actions of the operator; Based on the record, learning the actions of the operator; Based on the learning, making the operation robot perform the same actions as those of the operator; and Performing adjustment so that the actions of the operator are consistent with the actions of the operation robot.

2. The method for adjusting an operation robot according to claim 1, wherein, the mobile robot and the operation robot are the same robot.

3. The method for adjusting an operation robot according to claim 1 or 2, wherein, the sensor includes a photographing device, and the photographing device is arranged at a position higher than the mobile robot.

4. A sensing system, characterized in that, the sensing system includes: A first sensor for sensing a specified action of a sensing object; A second sensor for sensing the specified action of the sensing object from a position different from that of the first sensor; A first mobile robot having the first sensor and a first moving mechanism; and A management and control device capable of communicating with the first sensor, the second sensor, and the first moving mechanism, wherein the management and control device has: A determination unit for determining whether a specified part that can move when the sensing object performs a specified action is sensed based on first information obtained by the first sensor and second information obtained by the second sensor; and A control unit for, when it is determined by the determination unit that the specified part is not sensed, causing the first moving mechanism to operate so that the specified part is sensed.

5. The sensing system according to claim 4, wherein, the second sensor is arranged on a second mobile robot having a second moving mechanism, and when it is determined by the determination unit that the specified part is not sensed, the control unit causes the first moving mechanism and the second moving mechanism to operate so that the specified part is sensed.

6. The sensing system according to claim 5, wherein, the control unit causes the first moving mechanism to operate so that the first sensor senses a part of the specified part, and causes the second moving mechanism to operate so that the second sensor senses other parts of the specified part.

7. The sensing system according to any one of claims 4 to 6, wherein, the sensing system further includes an operation robot, and the management and control device has: A storage unit for storing the first information and the second information; A learning unit for learning the specified action with reference to the first information and the second information stored in the storage unit; and An action information generation unit, which generates action information for giving an action instruction to the work robot with reference to the learning result of the learning unit.

8. The sensing system according to claim 7, wherein, the storage unit stores in advance operation manual information or process sheet information of the sensing object, and the action information generation unit generates action information for giving an action instruction to the work robot with reference to the learning result of the learning unit and the operation manual information or process sheet information of the sensing object.

9. A sensing method, characterized in that it comprises: a first sensor for sensing a specified action of a sensing object; a second sensor for sensing the specified action of the sensing object from a position different from that of the first sensor; a first mobile robot having the first sensor and a first moving mechanism; and a management control device capable of communicating with the first sensor, the second sensor, and the first moving mechanism, in the sensing method, the management control device determines whether a specified part that can move when the sensing object performs a specified action is sensed based on first information obtained by the first sensor and second information obtained by the second sensor; and when it is determined by the determination that the specified part is not sensed, the first moving mechanism is made to operate so that the specified part is sensed.

10. A sensing system, characterized in that the sensing system comprises: a first sensor for sensing a specified action of a sensing object; a second sensor for sensing the specified action of the sensing object from a position different from that of the first sensor; a first mobile robot having the first sensor and a first moving mechanism; and a management control device capable of communicating with the first sensor, the second sensor, and the first moving mechanism, the management control device has: a determination unit that determines whether a specified part that can move when the sensing object performs a specified action is sensed based on first information obtained by the first sensor and second information obtained by the second sensor, and determines whether the specified part sensed by the first sensor is the same as the specified part sensed by the second sensor; and a control unit that, when it is determined that the specified part sensed by the first sensor is the same as the specified part sensed by the second sensor, causes the first moving mechanism to operate so that the specified part sensed by the first sensor is different from the specified part sensed by the second sensor.

11. A mobile robot, characterized in that the mobile robot comprises: a moving mechanism; a first sensor for sensing a sensing object; a second sensor for sensing the sensing object from a position different from that of the first sensor; ​ A drive mechanism that can move the position of the second sensor; And An information processing unit that controls the first sensor, the second sensor, the moving mechanism, and the drive mechanism. The information processing unit has: A determination unit that determines whether a specified part that can move when the sensed object performs a specified action is sensed based on first information obtained by the first sensor and second information obtained by the second sensor; And A control unit that, when it is determined by the determination unit that the specified part is not sensed, causes the moving mechanism or the drive mechanism to operate so that the specified part is sensed.

12. A mobile robot Characterized in that The mobile robot includes: A moving mechanism; A first sensor that senses a sensed object; A second sensor that senses the sensed object from a position different from that of the first sensor; A drive mechanism that can move the position of the second sensor; And An information processing unit that controls the first sensor, the second sensor, the moving mechanism, and the drive mechanism. The information processing unit has: A determination unit that determines whether a specified part that can move when the sensed object performs a specified action is sensed based on first information obtained by the first sensor and second information obtained by the second sensor, and determines whether the specified part sensed by the first sensor is the same as the specified part sensed by the second sensor; And A control unit that, when it is determined that the specified part sensed by the first sensor is the same as the specified part sensed by the second sensor, causes the moving mechanism or the drive mechanism to operate so that the specified part sensed by the first sensor is different from the specified part sensed by the second sensor.

13. A sensing system Characterized in that The sensing system includes: A first sensor for sensing a specified action of a sensed object; A work robot that operates according to an action instruction; A second sensor for sensing the robot action of the work robot; and A management control device that can communicate with the first sensor, the second sensor, and the work robot. The management control device has: A learning unit that learns the specified action by referring to first information obtained by the first sensor; An action information generation unit that generates action control information for giving the action instruction to the work robot by referring to the learning result of the specified action by the learning unit; And An adjustment unit that compares the first information with second information obtained by the second sensor and adjusts the action control information so that the robot action of the work robot approximates the specified action.

14. The sensing system according to claim 13, wherein, the management control device simultaneously acquires the first information by the first sensor and the second information by the second sensor.

15. The sensing system according to claim 13, wherein, the management control device separately acquires the first information by the first sensor and the second information by the second sensor.

16. The sensing system according to any one of claims 13 to 15, wherein, the management control device further has a storage unit that stores the first information, the second information, and operation manual information related to the specified action, and the action information generation unit generates the action control information with reference to the learning result of the specified action by the learning unit and the operation manual information.

17. The sensing system according to claim 16, wherein, when generating the action control information, the action information generation unit does not use the learning result of the specified action by the learning unit that is contrary to the operation manual information.

18. The sensing system according to claim 13, wherein, the first sensor or the second sensor is disposed on a mobile robot having a moving mechanism, and the mobile robot can communicate with the management control device.

19. A sensing method, characterized in that it includes: a first sensor for sensing a specified action of a sensing object; a work robot that performs an action according to an action instruction; a second sensor for sensing the robot action of the work robot; and a management control device that can communicate with the first sensor, the second sensor, and the work robot, wherein in the sensing method, the management control device learns the specified action with reference to the first information acquired by the first sensor; generates action control information for giving the action instruction to the work robot with reference to the learning result of the specified action through the learning; and compares the first information with the second information acquired by the second sensor, and adjusts the action control information so that the robot action of the work robot approximates the specified action.

20. A work robot, characterized in that the work robot performs an action according to an action instruction and includes: a first sensor for sensing a specified action of a sensing object; a second sensor for sensing the robot action of the work robot; and an information processing unit that can communicate with the first sensor and the second sensor, wherein the information processing unit has: a learning unit that learns the specified action with reference to the first information acquired by the first sensor; an action information generation unit that generates action control information for giving the action instruction to the work robot with reference to the learning result of the specified action by the learning unit; and ​ An adjustment unit that compares the first information with second information acquired by the second sensor and adjusts the motion control information so that the robot motion of the work robot approximates the specified motion.

21. A motion change system, characterized in that the motion change system includes: a work robot; a sensor; and a management control device capable of communicating with the work robot and the sensor, wherein the management control device includes: a learning unit that learns a standard motion model corresponding to the specified motion of the sensing object based on sensing information corresponding to the specified motion of the sensing object acquired using the sensor; a model generation unit that generates a changed motion model with reference to the standard motion model, the changed motion model setting the execution time of each motion in the standard motion model to be shorter than the required time of each motion when generating the standard motion model; and a control unit that causes the work robot to perform work with reference to the changed motion model.

22. The motion change system according to claim 21, wherein the management control device further has a storage unit that stores work manual information or process table information of the sensing object, and the learning unit generates the standard motion model with reference to the sensing information and the work manual information or process table information of the sensing object.

23. A motion change system, characterized in that the motion change system includes: a work robot; a plurality of sensors for respectively sensing a plurality of different sensing objects; and a management control device capable of communicating with the work robot and the plurality of sensors, wherein the management control device includes: a learning unit that learns each specified motion of the plurality of sensing objects and a plurality of standard motion models corresponding to each specified motion of the plurality of sensing objects based on a plurality of sensing information corresponding to the specified motions of the plurality of sensing objects acquired using the plurality of sensors; a model generation unit that generates a changed motion model with reference to the plurality of standard motion models, the changed motion model integrating at least a part of the specified motions of the plurality of sensing objects; and a control unit that causes the work robot to perform work with reference to the changed motion model.

24. The motion change system according to any one of claims 21 to 23, wherein a plurality of the work robots are provided, and the control unit causes the plurality of work robots to perform work.

25. A motion change method, characterized in that in the motion change method, a standard motion model corresponding to the specified motion of the sensing object is learned based on sensing information corresponding to the specified motion of the sensing object acquired using a sensor; a changed motion model is generated with reference to the standard motion model, the changed motion model setting the execution time of each motion in the standard motion model to be shorter than the required time of each motion when generating the standard motion model; and the work robot is caused to perform work with reference to the changed motion model.

26. A work robot, characterized in that, the work robot includes: a driving mechanism for causing the work robot to perform actions; a learning unit that learns a standard action model corresponding to a specified action of the sensing object based on sensing information corresponding to the specified action of the sensing object obtained by using a sensor; a model generation unit that refers to the standard action model and generates a modified action model, where the modified action model sets the execution time of each action in the standard action model to be shorter than the required time of each action when generating the standard action model; and a control unit that refers to the modified action model and controls the driving mechanism so that the work robot performs work.

27. A work reproduction system, characterized in that, the work reproduction system includes: a work reproduction robot; a sensor capable of sensing the actions of the work reproduction robot; and a management control device capable of communicating with the work reproduction robot and the sensor, the management control device includes: a learning unit that learns a standard action model corresponding to a specified action of the sensing object based on first sensing information corresponding to the specified action of the sensing object; a control unit that refers to the standard action model and causes the work reproduction robot to perform one or more reproduction actions; an input unit for inputting accident or malfunction information; and a detection unit that detects the occurrence of the accident or malfunction based on second sensing information corresponding to the reproduction actions of the work reproduction robot obtained by using the sensor.

28. A work reproduction system, characterized in that, the work reproduction system includes: a work reproduction robot; a sensor capable of sensing the actions of the work reproduction robot; and a management control device capable of communicating with the work reproduction robot and the sensor, the management control device includes: a learning unit that learns a standard action model corresponding to a specified action of the sensing object based on first sensing information corresponding to the specified action of the sensing object; a control unit that refers to the standard action model and causes the work reproduction robot to perform one or more reproduction actions; a storage unit that stores work manual information or process sheet information of the sensing object; and a detection unit that detects the occurrence of an action different from the work manual information or the process sheet information based on second sensing information corresponding to the reproduction actions of the work reproduction robot obtained by using the sensor.

29. A work reproduction method, characterized in that, it includes: a work reproduction robot; a sensor capable of sensing the actions of the work reproduction robot; and a management control device capable of communicating with the work reproduction robot and the sensor, in the work reproduction method, the management control device learns a standard action model corresponding to a specified action of the sensing object based on first sensing information corresponding to the specified action of the sensing object; Based on the reference standard action model, cause the operation reproduction robot to perform more than one reproduction action; Input accident or malfunction information; And Based on the second sensing information corresponding to the reproduction action of the operation reproduction robot obtained by using the sensor, detect the occurrence of the accident or malfunction.

30. An operation reproduction method, Characterized in that, Comprising: An operation reproduction robot; A sensor capable of sensing the actions of the operation reproduction robot; and A management control device capable of communicating with the operation reproduction robot and the sensor, In the operation reproduction method, the management control device Based on the first sensing information corresponding to the specified action of the sensing object, learn the standard action model corresponding to the specified action of the sensing object; Based on the reference standard action model, cause the operation reproduction robot to perform more than one reproduction action; Store the operation manual information or process sheet information of the sensing object; And Based on the second sensing information corresponding to the reproduction action of the operation reproduction robot obtained by using the sensor, detect the occurrence of an action different from the operation manual information or the process sheet information.

31. An operation reproduction robot, Characterized in that, Comprising: A sensor capable of sensing the actions of the operation reproduction robot and an information processing device capable of communicating, The information processing device comprises: A learning unit, which learns the standard action model corresponding to the specified action of the sensing object based on the first sensing information corresponding to the specified action of the sensing object; A control unit, which causes the operation reproduction robot to perform more than one reproduction action by referring to the standard action model; An input unit, which inputs accident or malfunction information; And A detection unit, which detects the occurrence of the accident or malfunction based on the second sensing information corresponding to the reproduction action of the operation reproduction robot obtained by using the external sensor.

32. An operation reproduction robot, Characterized in that, Comprising: A sensor capable of sensing the actions of the operation reproduction robot and an information processing device capable of communicating, The information processing device A learning unit, which learns the standard action model corresponding to the specified action of the sensing object based on the first sensing information corresponding to the specified action of the sensing object; A control unit, which causes the operation reproduction robot to perform more than one reproduction action by referring to the standard action model; A storage unit, which stores the operation manual information or process sheet information of the sensing object; And A detection unit, which detects the occurrence of an action different from the operation manual information or the process sheet information based on the second sensing information corresponding to the reproduction action of the operation reproduction robot obtained by using the sensor.

33. An operation familiarization system, Characterized in that, The operation familiarization system comprises: An operation reproduction robot; A sensor capable of sensing the actions of a new operator; and A management control device capable of communicating with the operation reproduction robot and the sensor, The management control device comprises: A storage unit that stores a standard action model learned based on first sensing information corresponding to the specified actions of a skilled operator; A control unit that refers to the standard action model and causes the operation reproduction robot to perform a reproduction action; And A detection unit that detects the differences between the actions of the new operator and the standard action model based on second sensing information corresponding to the actions of the new operator obtained using the sensor.

34. The operation familiarization system according to claim 33, wherein the storage unit further stores operation manual information or process sheet information, and the detection unit detects the differences between the actions of the new operator and the operation manual information or the process sheet information based on second sensing information corresponding to the actions of the new operator obtained using the sensor.

35. An operation familiarization method, characterized in that it includes: An operation reproduction robot; A sensor capable of sensing the actions of a new operator; and A management control device capable of communicating with the operation reproduction robot and the sensor, In the operation familiarization method, the management control device stores a standard action model learned based on first sensing information corresponding to the specified actions of a skilled operator; refers to the standard action model and causes the operation reproduction robot to perform a reproduction action; And detects the differences between the actions of the new operator and the standard action model based on second sensing information corresponding to the actions of the new operator obtained using the sensor.

36. An operation reproduction robot, characterized in that the operation reproduction robot includes: A sensor capable of sensing the actions of a new operator and an information processing device capable of communicating, The information processing device includes: A storage unit that stores a standard action model learned based on first sensing information corresponding to the specified actions of a skilled operator; A control unit that refers to the standard action model and causes the operation reproduction robot to perform a reproduction action; And A detection unit that detects the differences between the actions of the new operator and the standard action model based on second sensing information corresponding to the actions of the new operator obtained using the sensor.

Images