Collaborative work management device, method, and program
The collaborative work management system analyzes data from workers and robots to enhance performance metrics, addressing the challenge of improving human-robot collaboration by identifying and improving key contributing factors, thus efficiently supporting better work results.
Patent Information
- Application Number
- PCT/JP2024/018320
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-19
- Filing Date
- 2024-05-17
- Publication Date
- 2024-11-28
AI Technical Summary
Current systems fail to accurately improve performance when humans and work robots collaborate, as they cannot identify key factors contributing to poor performance, often resorting to replacing personnel rather than enhancing capabilities.
A collaborative work management device and method that includes a processing circuit to analyze data from both workers and work robots, calculating feature amounts and their contribution rates to evaluation values, enabling targeted improvements in worker and robot performance.
This approach allows for precise enhancement of performance metrics, efficiently supporting improved work results in human-robot collaborative tasks by identifying and improving key contributing factors.
Smart Images

Figure JP2024018320_28112024_PF_FP_ABST
Abstract
Description
Collaborative work management device, method, and program
[0001] The present disclosure relates to a collaborative work management device, method, and program.
[0002] Patent Literature 1 discloses a performance measurement device for objectively and quantitatively measuring a person's ability to process a task. This device analyzes sensing data obtained by measuring the person's activity while performing the task, and calculates a performance index that indicates the degree of performance for each elemental work achieved in the activity.
[0003] Japanese Patent Application Laid-Open No. 2020-35331
[0004] With this system, if a person's performance is poor, it is not possible to accurately identify the factors that will improve that performance, and the only option is to simply replace the person with another person who is performing better. In particular, a method that can effectively improve performance when humans and work robots work together is desired.
[0005] Therefore, one aspect of the present disclosure aims to efficiently support improving the evaluation of work results in collaborative work between humans and work robots.
[0006] A collaborative work management device according to one aspect of the present disclosure is a device for managing a collaborative work system including a work robot and a remote control terminal connected to the work robot via a communications network and operated by a worker, the device including a processing circuit configured to acquire data indicating at least one of information regarding the worker's perception, judgment, or instructions, and information regarding the operation of the work robot, calculate a plurality of feature amounts based on the acquired data, calculate an evaluation value for the work result of the work robot, and calculate the contribution rate of the plurality of feature amounts to the evaluation value.
[0007] A collaborative work management method according to one aspect of the present disclosure is a method for managing a collaborative work system including a work robot and a remote control terminal connected to the work robot via a communications network and operated by a worker, the method including acquiring data indicating at least one of information regarding the worker's perception, judgment, or instructions, and information regarding the operation of the work robot, calculating a plurality of feature amounts based on the acquired data, calculating an evaluation value for the work result of the work robot, and calculating the contribution rate of the plurality of feature amounts to the evaluation value.
[0008] A collaborative work management program according to one aspect of the present disclosure causes at least one processor to execute the collaborative work management method. The program may be stored in a computer-readable storage medium. The storage medium is a non-transitory, tangible medium. The storage medium may be built into or external to a computer (e.g., a mobile information terminal, a personal computer, a server, etc.). The storage medium may include RAM, ROM, EEPROM, storage, etc., and may be, for example, a hard disk, flash memory, or optical disk. The program stored in the storage medium may be executed on a computer to which the storage medium is directly connected, or on a computer connected to the storage medium via a network (e.g., the Internet).
[0009] According to one aspect of the present disclosure, it is possible to identify feature values that have a high contribution rate to the evaluation value of the work result of a work robot (e.g., work time). By improving the worker's ability with respect to those feature values, it is possible to accurately improve the evaluation value of the work result. Therefore, it is possible to efficiently support the improvement of the evaluation value of the work result in collaborative work between humans and work robots.
[0010] FIG. 1 is an overall view showing a collaborative work system equipped with a collaborative work management device according to an embodiment. FIG. 2 is a conceptual diagram illustrating the task execution process of a worker and a work robot in the collaborative work system of FIG. 1. FIG. 3 is a block diagram of the work robot of FIG. 1. FIG. 4 is a block diagram of the remote control terminal of FIG. 1. FIG. 5 is a block diagram of the collaborative work management server of FIG. 1. FIG. 6 is a diagram showing an example of a display screen of the remote control terminal of FIG. 4. FIG. 7 is a diagram illustrating elemental tasks performed by the work robot and worker of FIG. 1. FIG. 8 is a flowchart illustrating processing by the collaborative work management server of FIG. 5. FIG. 9 is a block diagram illustrating principal component analysis performed by the collaborative work management server of FIG. 5. FIG. 10 is a diagram illustrating a gaze plot obtained by tracking the gaze of a worker. FIG. 11 is a block diagram illustrating ability analysis performed by the collaborative work management server of FIG. 5.
[0011] Hereinafter, an embodiment will be described with reference to the drawings.
[0012] Fig. 1 is an overall view showing a collaborative work system 1 provided with a collaborative work management server 10 according to an embodiment. As shown in Fig. 1, the collaborative work system 1 includes a work robot 2, a remote control terminal 3 capable of communicating with the remote control terminal 3 via a communication network 5, and a collaborative work management server 10 capable of communicating with the work robot 2 and the remote control terminal 3 via the communication network 5. The communication network 5 is, for example, the Internet, but may also be an intranet or the like.
[0013] In this embodiment, the work robot 2 is exemplified as a work robot that performs harvesting work on the plant harvest target H (e.g., fruits, vegetables, etc.), which is the work object W, but the work robot 2 may be any robot that performs any type of work, and may also perform other types of work (e.g., visual inspection, sorting and collecting parts, etc.).
[0014] The work robot 2 has a body 2a, an articulated arm 2b protruding from the body 2a, an end effector 2c attached to the tip of the articulated arm 2b, and wheels 2d supporting the body 2a. As an example, the end effector 2c is a robotic hand capable of grasping an object. A basket for retrieving an object grasped by the robotic hand may be removably mounted on the work robot 2. As will be described later, the work robot 2 is equipped with sensors, a controller, actuators, and interfaces. Note that the work robot 2 is not limited to one that moves by running on the ground, and may be, for example, a drone that moves by flying through the air.
[0015] The remote control terminal 3 is operated by a human worker 4. That is, by operating the remote control terminal 3, the worker 4 remotely controls the work robot 2 connected to the remote control terminal 3 via a communication network 5. The collaborative work management server 10 receives data from the work robot 2 and data from the remote control terminal 3 via the communication network 5. The collaborative work system 1 may also include a management terminal 11 that can communicate with the collaborative work management server 10 via the communication network 5.
[0016] FIG. 2 is a conceptual diagram illustrating the task execution process of the worker 4 and the work robot 2 in the collaborative work system 1 of FIG. 1. As shown in FIG. 2, in the collaborative work system 1, the multiple task components that make up a task are shared between the work robot 2 and the worker 4. Specifically, a task is composed of the task components of "perception," "judgment," "instruction," and "action." "Perception" is performed based on information after the previous "action." "Judgment" is performed based on "perceived" information. "Instruction" is performed based on "judgment" information. "Action" is performed based on "instruction" information. In the collaborative work system 1, the work robot 2 and worker 4 work cooperatively, so the task components of "perception," "judgment," and "instruction" are performed by the worker 4, and the task component of "action" is performed by the work robot 2.
[0017] Figure 3 is a block diagram of the work robot 2 of Figure 1. As shown in Figure 3, the work robot 2 includes a processing circuit 20, a positioning sensor 24, a traveling speed sensor 25, a steering angle sensor 26, an image sensor 27, a force sensor 28, an arm acceleration sensor 29, a traveling actuator 30, a steering actuator 31, a braking actuator 32, an arm actuator 33, and a communication interface 34.
[0018] The processing circuit 20 includes a processor 21, a system memory 22, and a storage memory 23. The processor 21 may include a CPU (Central Processing Unit). The system memory 22 may include RAM. The storage memory 23 may include a hard disk, flash memory, or a combination thereof. The storage memory 23 stores a program P1. The configuration in which the processor 21 executes the program P1 read from the storage memory 23 to the system memory 22 is an example of a processing circuit of the work robot 2.
[0019] The positioning sensor 24 detects the position of the work robot 2 and may be, for example, a satellite positioning sensor such as a GPS sensor. The positioning sensor 24 may be a sensor other than a satellite positioning sensor. For example, the position of the work robot 2 on map information may be determined by matching the shape of the surroundings of the work robot 2 obtained by measuring the distance around the work robot 2 with the shape of the map information. In other words, the positioning sensor 24 may be realized by combining a ranging sensor with software that matches the shape detected by the ranging sensor with the map information. The ranging sensor detects position data of the outer surface of obstacles by emitting light, radio waves, or ultrasonic waves around the work robot 2 and receiving reflected waves. For example, the positioning sensor 24 may be a light detection and ranging (LIDAR) sensor, millimeter-wave radar, or depth-sensing camera. Alternatively, the positioning sensor 24 may acquire position coordinates by wirelessly communicating with multiple locators installed on the ground using radio waves, sound waves, light, or magnetism. The positioning principle using the locator may be, for example, the angle of arrival (AOA) method, the received signal strength indicator (RSSI) method, the time of arrival (TOA) method, or the time difference of arrival (TDOA) method.
[0020] The traveling speed sensor 25 detects the traveling speed of the work robot 2. The traveling speed sensor 25 detects, for example, the rotation speed of the wheels 2d of the work robot 2. The steering angle sensor 26 detects the steering angle of the wheels 2d. The image sensor 27 is capable of acquiring images of the work object W near the work robot 2 and may be, for example, a camera. The force sensor 28 detects the magnitude and direction of a force or moment and is provided on the articulated arm 2b or the end effector 2c. The arm acceleration sensor 29 detects the acceleration of the movement of the articulated arm 2b. The sensors mounted on the work robot 2 are not particularly limited and may include, for example, a gyro sensor, a distance measurement sensor, etc.
[0021] The travel actuator 30 drives the wheels 2d and is, for example, an electric motor. The steering actuator 31 drives a steering mechanism that steers the wheels 2d and is, for example, an electric motor. The braking actuator 32 drives a brake that brakes the wheels 2d and is, for example, an electric motor. The arm actuator 33 drives the articulated arm 2b or the end effector 2c and is, for example, an electric motor. The communication interface 34 includes a communication device that connects to the communication network 5 by wire or wirelessly.
[0022] In the work robot 2, the processor 21, which executes the program P1 read from the storage memory 23 to the system memory 22, controls at least one of the actuators 30 to 33 based on information input from at least one of the sensors 24 to 29.
[0023] FIG. 4 is a block diagram of the remote control terminal 3 of FIG. 1 . As shown in FIG. 4 , the remote control terminal 3 includes a processing circuit 40, a display 44, an operation interface 45, a line-of-sight sensor 46, and a communication interface 47. The processing circuit 40 includes a processor 41, a system memory 42, and a storage memory 43. The processor 41 is, for example, a CPU. The system memory 42 is, for example, a RAM. The storage memory 43 is an example of a computer-readable medium, and is a non-transitory, tangible medium. The storage memory 43 may include a ROM. The storage memory 43 may include a hard disk, a flash memory, or a combination thereof. The storage memory 43 stores a program P2. The configuration in which the processor 41 executes the program P2 read into the system memory 42 is an example of a processing circuit of the remote control terminal 3.
[0024] The display 44 is not particularly limited as long as it can output an image to the worker 4, and may be, for example, a liquid crystal display, an organic EL display, a head-mounted display, or a VR display. The operation interface 45 is a user interface operated by the worker 4. The operation interface 45 includes, for example, at least one selected from a keyboard, a mouse, a touch panel, a lever, a handle, and the like.
[0025] The gaze sensor 46 may be, for example, a combination of a camera that captures the eye movements of the worker 4 and software that processes the images captured by the camera to analyze the eye movements, but other types of gaze detection technology may also be used. The communication interface 47 includes a communication device that connects to the communication network 5 by wire or wirelessly.
[0026] FIG. 5 is a block diagram of the collaborative work management server 10 of FIG. 1 . As shown in FIG. 5 , the collaborative work management server 10 includes a processing circuit 60 and a communication interface 64. The processing circuit 60 includes a processor 61, a system memory 62, and a storage memory 63. The communication interface 64 includes a communicator that connects to the communication network 5 via a wired or wireless connection. The processor 61 may include a CPU (Central Processing Unit). The system memory 62 may include RAM. The storage memory 63 may include a hard disk, a flash memory, or a combination thereof. The storage memory 63 stores a collaborative work management program P3. The configuration in which the processor 61 executes the collaborative work management program P3 read from the storage memory 63 to the system memory 62 is an example of a processing circuit of the collaborative work management server 10.
[0027] Collaborative work management server 10 collects data from work robot 2 and remote control terminal 3 and stores it in storage memory 63. Program P3 stores a principal component analysis model 71 and a performance analysis model 72, which will be described later.
[0028] Fig. 6 is a diagram showing an example of a display screen 50 of the display 44 of the remote control terminal 3 of Fig. 4. The following control of the display screen 50 is executed by the processor 41 of the remote control terminal 3 operating in accordance with program P2. As shown in Fig. 6, the display screen 50 displays a harvest display area 51, a selection display area 52, a selection icon 53, a designated registration icon 54, a grip and recovery icon 55, a movement stop icon 56, and an emergency stop icon 57.
[0029] The processing circuitry 40 of the remote control terminal 3 receives in real time images of the work target W captured by the image sensor 27 of the work robot 2 via the communications network 5. The processing circuitry 40 displays the received images in the harvest display area 51. The harvest targets H1-H6 may include both ripe and suitable for harvesting and unripe and unsuitable for harvesting. To enhance the distinction of the harvest targets H1-H6 in the harvest display area 51, the processing circuitry 40 may highlight each of the harvest targets H1-H6, for example by displaying a frame around each of them.
[0030] When operator 4 uses operation interface 45 to designate harvest targets H1-H6 on the screen of harvest display area 51 (for example, by clicking or tapping with the mouse), processing circuitry 40 performs clustering using known image processing techniques to recognize the area of the designated harvest target on the image and displays the outline of the recognized area on the screen. When operator 4 selects designation registration icon 54 in this state, processing circuitry 40 adds a display indicating the designated harvest target (in FIG. 6, the numbers of harvest targets H1-H6) to selection display area 52.
[0031] When worker 4 uses operation interface 45 to select one of harvest targets H1 to H6 in selection display area 52 and selects selection icon 53, processing circuit 40 highlights (e.g., changes color) the number selected by selection icon 52. When worker 4 uses operation interface 45 to select grip and recovery icon 55 in this state, processing circuit 40 sends an instruction to work robot 2 via communication network 5 to grip and recover the harvest target selected by selection icon 53. Processing circuit 40 grays out the display indicating the harvest target that has been instructed by grip and recovery icon 55 in selection display area 52.
[0032] If the worker 4 selects the stop movement icon 56 using the operation interface 45 while the work robot 2 is stopped, the processing circuit 40 sends a command to the work robot 2 via the communication network 5 so that the work robot 2 heads toward the next work object W. If the worker 4 selects the stop movement icon 56 using the operation interface 45 while the work robot 2 is moving, the processing circuit 40 sends a command to the work robot 2 via the communication network 5 so that the work robot 2 stops moving and stops. If the worker 4 selects the emergency stop icon 57 using the operation interface 45, the processing circuit 40 sends a command to the work robot 2 via the communication network 5 so that the work robot 2 stops unconditionally.
[0033] 7 is a diagram explaining the elemental tasks E1 to E5 performed by the work robot 2 and worker 4 of FIG. 1. As shown in FIG. 7, a task for one work object W (for example, one plant) is divided into a task performed by the work robot 2 and a task performed by the worker 4. The entire task for one work object W (for example, one plant) is referred to as unit task V. Unit task V is made up of multiple elemental tasks E1, E3, and E5 performed by the work robot 2 and multiple elemental tasks E2 and E4 performed by the worker 4.
[0034] Specifically, first, the working robot 2 photographs the work object W (for example, a single plant) using the image sensor 27, sends the photographed image to the remote control terminal 3, and the remote control terminal 3 displays the received image on the display 44. This task element E1 is called a "photograph-display" task element.
[0035] Next, operator 4 looks at the image (harvest display area 51 in FIG. 6) displayed on display 44 of remote control terminal 3, decides which of harvest targets H1 to H6 (FIG. 6) to harvest, and operates icons 53 to 55 on display screen 50 (FIG. 6) to instruct working robot 2 to collect the selected harvest target. This task element E2 is called a "judgment-instruction" task element.
[0036] Next, the processing circuit 20 of the working robot 2 receives the instruction and drives the arm actuator 33 to grasp the selected harvest target and return it to a basket mounted on the working robot 2. This elemental task E3 is called the "grasp-recovery" elemental task. After that, the "judge-instruct" elemental task and the "grasp-recovery" elemental task are alternately repeated.
[0037] Next, when the worker 4 looks at the image (harvest display area 51 in FIG. 6 ) displayed on the display 44 of the remote control terminal 3 and determines that the harvesting work for the work object W is complete, the worker 4 selects the movement stop icon 56 on the display screen 50 and sends an instruction to the work robot 2 to move to the next work object W (for example, the adjacent plant). This elemental work E4 is called the "completion determination-movement instruction" elemental work.
[0038] Next, the processing circuit 20 of the working robot 2 receives the instruction and drives the travel actuator 30, etc., to move toward the next work object W (for example, the neighboring plant), stopping when it reaches a position suitable for working on the next work object W. This elemental task E5 is called the "move-stop" elemental task. The collection of elemental tasks from the "photograph-display" elemental task E1 to the "move-stop" elemental task E5 in this way constitutes unit task V.
[0039] The work time required to perform unit work V is referred to as unit work time Tv. The work time required to perform each of the element work E1 to E5 is referred to as element work time Te. Note that although the same symbol Te is used for each element work time, the element work times may actually differ from one another. The element work time Te and unit work time Tv are measured by the collaborative work management server 10.
[0040] With regard to the element task time Te for the "photograph-display" task element E1, the processing circuitry 20 of the work robot 2 transmits the time t1 when the image capture command was issued to the image sensor 27 to the collaborative work management server 10, and the processing circuitry 40 of the remote control terminal 3 transmits the time t2 when the image captured by the image sensor 27 was displayed on the display 44 to the collaborative work management server 10. The processing circuitry 60 of the collaborative work management server 10 obtains times t1 and t2 from the work robot 2 and the remote control terminal 3 via the communications network 5, and calculates the difference between these two times (t2 - t1) as the element task time Te for the "photograph-display" task element E1.
[0041] With regard to the element task time Te of the "judgment-instruction" element task E2, the remote control terminal 3 transmits to the collaborative work management server 10 the time t2 when the image captured by the image sensor 27 was displayed on the display 44, and the time t3 when the worker 4 selected the grasp and retrieve icon 55 on the display screen 50 by operating the operation interface 45 of the remote control terminal 3. The processing circuitry 60 of the collaborative work management server 10 obtains the times 2 and t3 from the remote control terminal 3 via the communication network 5, and calculates the difference between the two times (t3 - t2) as the element task time Te of the "judgment-instruction" element task E2.
[0042] With regard to the element task time Te of the "grasp-recovery" task element E3, the processing circuitry 20 of the working robot 2 transmits to the collaborative work management server 10 the time t4 when it receives the command to grasp and recover and begins driving the arm actuator 33, and the time t5 when it is determined that the harvest target H has been grasped and recovered into the basket based on the operation of the articulated arm 2b. The processing circuitry 60 of the collaborative work management server 10 obtains times t4 and t5 from the working robot 2 via the communications network 5, and calculates the difference between these two times (t5 - t4) as the element task time Te of the "grasp-recovery" task element E3.
[0043] With regard to the element task time Te for the "completion determination-move instruction" task element E4, the processing circuitry 20 of the work robot 2 transmits to the collaborative work management server 10 the time t6 at which it is determined that the last harvest target H has been grasped and returned to the basket, and the processing circuitry 40 of the remote control terminal 3 transmits to the collaborative work management server 10 the time t7 at which the worker 4 operates the operation interface 45 of the remote control terminal 3 to select the stop movement icon 56 on the display screen 50 to move toward the next task target W. The processing circuitry 60 of the collaborative work management server 10 obtains times t7 and t6 from the work robot 2 and the remote control terminal 3 via the communications network 5, and calculates the difference between these two times (t7 - t6) as the element task time Te for the "completion determination-move instruction" task element E4.
[0044] With regard to the element task time Te of the "move-stop" task element E5, the processing circuitry 20 of the work robot 2 transmits to the collaborative work management server 10 the time t8 when the work robot 2 starts moving towards the next task object W, and the time t9 when the work robot 2 reaches the next task object W and stops moving. The processing circuitry 60 of the collaborative work management server 10 obtains times t8 and t9 from the work robot 2 via the communications network 5, and calculates the difference between these two times (t9 - t8) as the element task time Te of the "move-stop" task element E5.
[0045] The processing circuit 60 of the collaborative work management server 10 calculates the unit work time Tv as the time required from the start time of the "photograph-display" element work E1, which is the first element work of the unit work V, to the end time of the "move-stop" element work E5, which is the last element work of the unit work V.
[0046] Fig. 8 is a flowchart illustrating the processing of the collaborative work management server 10 of Fig. 5. The following description will be given following the flow of Fig. 8 while also referring to other figures as appropriate. In the following description, the processing of the collaborative work management server 10 is executed by the processing circuitry 60.
[0047] In the collaborative work management server 10, an operator operates the management terminal 11 to perform initial settings for managing collaborative work between the work robot 2 and the remote control terminal 3 (step S1). When the worker 4 operates the remote control terminal 3 to cause the work robot 2 to start work, the collaborative work management server 10 acquires data from the work robot 2 and the remote control terminal 3 (step S2). This acquired data will be referred to as collected data.
[0048] Here, the collected data may include data detected by sensors equipped on the work robot 2 and the remote control terminal 3, data indicating the time (e.g., start time, end time, etc.) when a specific operation or process was performed on the work robot 2, and data indicating the time (e.g., operation time, start time, end time, etc.) when a specific operation, process, etc. was performed on the remote control terminal 3. The data collected by the collaborative work management server 10 from the remote control terminal 3 indicates information related to the perception, judgment, or instructions of the worker 4. The data collected by the collaborative work management server 10 from the work robot 2 indicates information related to the operation of the work robot 2.
[0049] The collaborative work management server 10 calculates an evaluation value for the work results of the work robot from the collected data (step S3). For example, the collaborative work management server 10 calculates the unit task time Tv from the collected data as the evaluation value. A shorter unit task time Tv indicates a higher evaluation, and a longer unit task time Tv indicates a lower evaluation.
[0050] The collaborative work management server 10 calculates the element work time Te of each element work E1 to E5 as a feature value from the collected data (step S4).The collaborative work management server 10 then performs principal component analysis using the calculated feature values (each element work time Te) and the calculated evaluation value (unit work time Tv) as inputs (step S5).
[0051] 9 is a block diagram illustrating principal component analysis performed by the collaborative work management server 10 of FIG. 5. As shown in FIG. 9, the collaborative work management server 10 has a principal component analysis model 71. The principal component analysis model 71 is a machine learning model that uses a known principal component analysis technique. When a plurality of feature quantities are input along with an evaluation value, the principal component analysis model 71 outputs the contribution rate of each feature quantity to a change in the evaluation value.
[0052] When performing principal component analysis on the work of worker 4 (step S5), for example, the element task times Te of the multiple "judgment-instruction" element tasks E2 in the unit task V and the element task time Te of the "completion judgment-movement instruction" element task E4 in the unit task V are input as feature quantities to the principal component analysis model 71, and the unit task time Tv of the unit task V is input as an evaluation value. When multiple unit tasks V have been performed, the element task times Te and unit task times Tv for these unit tasks V are similarly input to the principal component analysis model 71.
[0053] Then, the principal component analysis model 71 outputs the contribution rate of the element task time Te of the "judgment-instruction" element task E2 and the contribution rate of the element task time Te of the "completion judgment-instruction to move" element task E4. Here, it is assumed that the contribution rate of the element task time Te of the "judgment-instruction" element task E2 is the largest among all the contribution rates.
[0054] The collaborative work management server 10 calculates a plurality of detailed feature quantities by subdividing the feature quantity with the highest contribution rate (the element task time Te of the "judgment-instruction" element task E2) (step S6). The detailed feature quantities obtained by subdividing the element task time Te of the "judgment-instruction" element task E2 can be, for example, "eye point transition time," "eye point dwell time," and "eye point departure time."
[0055] 10 is a diagram illustrating a gaze plot obtained by tracking the gaze of the worker 4. As shown in FIG. 10, the collaborative work management server 10 uses known eye tracking technology to analyze the gaze movement of the worker 4 while working, based on image data 80 of the work object W displayed in the harvest display area 51 of the display screen 50 of the remote control terminal 3 and gaze trajectory data 81 indicating the trajectory of the worker's 4 gaze (point of gaze) on the image data screen 80. The trajectory of the worker's 4 gaze on the image data screen 80 is calculated from the eye movement detected by the gaze sensor 46 of the remote control terminal 3. The collaborative work management server 10 identifies the contours of the harvest objects H1 to H3 included in the image data 80 using known image recognition technology.
[0056] The "viewpoint transition time" is the time from when the viewpoint indicated by the viewpoint trajectory data 81 on the image data 80 leaves the outline of the first selected harvest target H1 until it enters the outline of the next selected harvest target H2. The "viewpoint dwell time" is the time from when the viewpoint indicated by the viewpoint trajectory data 81 on the image data 80 enters the outline of each of the selected harvest targets H1 to H3 until it exits the outline. The "viewpoint departure time" is the time from when the viewpoint indicated by the viewpoint trajectory data 81 on the image data 80 leaves the outline of each of the selected harvest targets H1 to H3 until it returns to the same outline.
[0057] Fig. 11 is a block diagram illustrating ability analysis by the collaborative work management server 10 of Fig. 5. As shown in Fig. 11, the collaborative work management server 10 has an ability analysis model 72. The ability analysis model 72 is a trained machine learning model. The collaborative work management server 10 uses the ability analysis model 72 to perform ability analysis on the calculated multiple detailed feature quantities (step S7). When the calculated multiple detailed feature quantities are input, the ability analysis model 72 outputs one or multiple ability scores.
[0058] The ability analysis model 72 receives, for example, the aforementioned "viewpoint transition time," "viewpoint dwell time," and "viewpoint departure time" as detailed feature quantities, and outputs one or more ability scores. Here, the ability scores include, for example, an "exploratory ability score," a "decisiveness score," and a "judgment ability score." The "exploratory ability score" tends to increase as the "viewpoint transition time" shortens, and is a value indicating the ability to search for harvest targets. The "decisiveness score" tends to increase mainly as the "viewpoint dwell time" shortens, and is a value indicating the decision-making ability from identifying a harvest target to issuing a collection instruction. The "judgment ability score" tends to increase as the "viewpoint departure time" shortens, and is a value indicating the judgment ability to determine without hesitation which harvest targets to harvest. Instead of the ability analysis model 72, a calculation formula may be used that receives multiple detailed feature quantities as input and outputs an ability score.
[0059] The collaborative work management server 10 evaluates the output ability scores by comparing them with predetermined standard scores (step S8). Specifically, the "exploratory ability score," "decisiveness score," and "judgment ability score" are evaluated by comparing each score with the corresponding standard score. Here, an ability score higher than the standard score indicates a high evaluation, and an ability score lower than the standard score indicates a low evaluation.
[0060] If the contribution rate of the element task time Te of the "completion judgment-move instruction" element task E4 is the largest among all the contribution rates in the principal component analysis, the collaborative work management server 10 can calculate multiple detailed feature values by subdividing the feature value with the largest contribution rate (the element task time Te of the "completion judgment-move instruction" element task E4) (step S6). The detailed feature values obtained by subdividing the element task time Te of the "completion judgment-move instruction" element task E4 can be, for example, "eye gaze transition time," "eye gaze dwell time," "eye gaze departure time," and "move instruction time." The "move instruction time" is the time from the completion of the final collection until the worker issues an instruction to move the work robot 2 to the next task object W. The collaborative work management server 10 can then use the ability analysis model 72 to perform an ability analysis on the calculated multiple detailed feature values (step S7). When the calculated multiple detailed feature values are input, the ability analysis model 72 can output one or more ability scores, such as a "search ability score," a "decisiveness score," and a "judgment ability score."
[0061] In the above steps S3 to S8, the work performed by the worker 4 has been described, but steps S3 to S8 can also be performed by the work robot 2.
[0062] When performing principal component analysis on the work of the work robot 2 (step S5), the element task time Te of the "photograph-display" element task E1 in unit task V, the element task time Te of the multiple "grasp-recovery" element tasks E3 in unit task V, and the element task time Te of the "move-stop" element task E5 in unit task V are input as feature quantities into principal component analysis model 71, and the unit task time Tv of that unit task V is input as an evaluation value. Note that although the learning content of the principal component analysis model for the work of the worker 4 and the principal component analysis model for the work of the work robot 2 differ, the principles are the same and so for simplicity they will be described using the same reference numeral (71).
[0063] In response to the above input, the principal component analysis model 71 outputs the contribution rate of the element task time Te of the "photograph-display" element task E1, the contribution rate of the element task time Te of the "grasp-recovery" element task E3, and the contribution rate of the element task time Te of the "move-stop" element task E5. Here, it is assumed that the contribution rate of the element task time Te of the "grasp-recovery" element task E3 is the largest among all the contribution rates.
[0064] The collaborative work management server 10 calculates a plurality of detailed feature amounts obtained by subdividing the feature amount with the highest contribution rate (the element task time Te of the "grasp-recovery" element task E3) (step S6). The detailed feature amounts obtained by subdividing the element task time Te of the "grasp-recovery" element task E3 can be, for example, "acceleration sensor time series data," "gyro sensor time series data," and "coordinate and time data of grasping and release."
[0065] "Acceleration sensor time series data" is time series data of the motion acceleration of the articulated arm 2b of the work robot 2. "Gyro sensor time series data" is time series data such as the angular velocity of the articulated arm 2b of the work robot 2. "Grip and release coordinate and time data" is data that indicates the coordinates and time when the articulated arm 2b of the work robot 2 grasps a harvest target, and the coordinates and time when the grasped harvest target is released. "Grip and release coordinate and time data" is data that indicates the coordinates and time when the work robot 2 grasps a harvest target, and the coordinates and time when the work robot 2 releases the grasped harvest target.
[0066] The collaborative work management server 10 uses the ability analysis model 72 to perform an ability analysis of the calculated multiple detailed feature quantities (step S7). Note that although the learning content of the ability analysis model for the work of the worker 4 and the ability analysis model for the work of the working robot 2 differ, the principles are the same and so for simplicity's sake, the same reference numeral (72) will be used in the explanation.
[0067] To the capability analysis model 72, for example, the above-mentioned "acceleration sensor time series data," "gyro sensor time series data," and "coordinate and time data of gripping and release" are input as detailed feature amounts, and one or more capability scores are output. Here, the capability score includes, for example, "arm acceleration," "arm speed," and "arm deceleration." Note that instead of the capability analysis model 72, a calculation formula may be used in which a plurality of detailed feature amounts are input and a capability score is output.
[0068] The collaborative work management server 10 evaluates the output ability score by comparing it with a predetermined standard score (step S8). Specifically, each score is evaluated by comparing "arm acceleration," "arm speed," and "arm deceleration" with the corresponding standard score. Here, if the ability score is higher than the standard score, it means that the evaluation is high, and if the ability score is lower than the standard score, it means that the evaluation is low.
[0069] The collaborative work management server 10 also executes steps S9 to S11 in addition to steps S3 to S8 described above. Specifically, the collaborative work management server 10 calculates the "oversight rate" and "defect rate" as feature quantities (step S9). The "oversight rate" is the value obtained by dividing the number of harvest targets that should have been harvested but were not collected by the total number of harvest targets that should have been harvested. The "defect rate" is the value obtained by dividing the number of harvest targets that should not have been collected by the total number of harvest targets that were collected.
[0070] The collaborative work management server 10 performs a capability analysis to calculate a capability score (e.g., a discrimination score) from these feature quantities (e.g., the oversight rate and the defect rate) (step S10). Specifically, the collaborative work management server 10 uses a predetermined arithmetic formula to calculate the discrimination score so that the discrimination score is lower when the "oversight rate" is high, and lower when the "defect rate" is high.
[0071] The collaborative work management server 10 evaluates the calculated ability score by comparing it with a predetermined standard score (step S11). Specifically, the collaborative work management server 10 evaluates the score by comparing the "discrimination score" with the standard score. Here, if the ability score is higher than the standard score, it means that the evaluation is high, and if the ability score is lower than the standard score, it means that the evaluation is low.
[0072] The collaborative work management server 10 also executes steps S12 to S14 in addition to steps S3 to S8 and steps S9 to S11 described above. Specifically, the collaborative work management server 10 calculates, as a feature, the element task times Te of the element tasks E2 and E4 for worker 4 from the collected data (step S12). These feature amounts are calculated for each of the multiple unit tasks V arranged in chronological order.
[0073] For example, if a first unit task V, a second unit task V, and a third unit task V are performed serially in this order, the collaborative work management server 10 calculates as feature quantities the element task time Te of the "judgment-instruction" element task E2 of the first unit task V, the element task time Te of the "judgment-instruction" element task E2 of the second unit task V, and the element task time Te of the "judgment-instruction" element task E2 of the third unit task. These times are the element task times required for the corresponding element tasks.
[0074] In addition, the collaborative work management server 10 may calculate the element work time Te of the “completion judgment-movement instruction” element work E4 of the first unit work V, the element work time Te of the “completion judgment-movement instruction” element work E4 of the second unit work V, and the element work time Te of the “completion judgment-movement instruction” element work E4 of the third unit work V as feature quantities.
[0075] The collaborative work management server 10 calculates the time-series increase / decrease trends of the element task times Te required for each of the corresponding element tasks ("judgment-instruction" element tasks E2) that make up the first to third unit tasks V (step S13). For example, if the element task times Te required for each of the corresponding element tasks that make up the first to third unit tasks V are on an increasing trend over time, it is considered that the concentration of the worker 4 is decreasing due to the continuous work of the first to third unit tasks V. On the other hand, if the element task times Te required for each of the corresponding element tasks that make up the first to third unit tasks V are not on an increasing trend over time, it is considered that the concentration of the worker 4 is being maintained even through the continuous work of the first to third unit tasks V.
[0076] Specifically, the collaborative work management server 10 calculates the time-series increase / decrease trends of the element task time Te of the "judgment-instruction" element task E2 of the first unit task V, the element task time Te of the "judgment-instruction" element task E2 of the second unit task V, and the element task time Te of the "judgment-instruction" element task E2 of the third unit task. For example, if the element task time Te of the "judgment-instruction" element task E2 of the second unit task is longer than the element task time Te of the "judgment-instruction" element task E2 of the first unit task V, and if the element task time Te of the "judgment-instruction" element task E2 of the third unit task is longer than the element task time Te of the "judgment-instruction" element task E2 of the second unit task V, the collaborative work management server 10 lowers the continuous task ability score of the worker 4. On the other hand, if there is little fluctuation in the element task time Te of the "judgment-instruction" element tasks E2 of the first to third unit tasks V, the collaborative work management server 10 raises the continuous task ability score of the worker 4.
[0077] The collaborative work management server 10 evaluates the continuous work ability score by comparing the output continuous work ability score with a predetermined standard score (step S14). Here, if the continuous work ability score is higher than the standard score, it means that the evaluation is high, and if the continuous work ability score is lower than the standard score, it means that the evaluation is low.
[0078] The collaborative work management server 10 outputs the evaluation results of the ability scores obtained in step S8, the evaluation results of the ability scores obtained in step S11, and the evaluation results of the continuous task ability scores obtained in step S14 (step S15). For example, the collaborative work management server 10 outputs the evaluation results to the management terminal 11 via the communication network 5. By viewing the evaluation results, the operator of the management terminal 11 can determine which ability needs to be improved optimally in order to achieve the target unit task time Tv.
[0079] The operator of the management terminal 11 then takes measures to improve the abilities corresponding to the low-rated ability scores. For example, the ability to be improved may be displayed on the remote control terminal 3 of the worker 4 being evaluated, or the worker may be replaced by a worker with relatively high ability. Such improvements can accurately improve the evaluation value of the work result (unit work time Tv). Therefore, in collaborative work between humans and work robots, improvement of the evaluation of the work result can be efficiently supported. However, a reference time (estimated time) for the total work time may be set, and ability improvement may be implemented only when it is determined that the total work time is longer than the reference time.
[0080] The technology of the present disclosure is not limited to the above-described embodiment. For example, while the collaborative work management server 10 has been exemplified as the collaborative work management device, the collaborative work management device is not limited to a server as long as it can obtain information on the remote control terminal 3 and the work robot 2. For example, the collaborative work management device may be a device that controls the work robot 2. Furthermore, the evaluation value is not limited to the work time, and may be another item (for example, defect rate), etc.
[0081] As described above, the above embodiments have been described as examples of the technology disclosed in this application. However, the technology of the present disclosure is not limited to these and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, the components described in the above embodiments can be combined to create new embodiments. For example, some configurations or methods in one embodiment may be applied to other embodiments, and some configurations in an embodiment can be separated and arbitrarily extracted from other configurations in that embodiment. Furthermore, the components described in the accompanying drawings and detailed description include not only components essential for solving the problem, but also components that are not essential for solving the problem, and are used to illustrate the technology.
[0082] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. Processors are considered processing circuits or circuits because they include transistors and other circuitry. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or processor.
[0083] Each of the following sections is a disclosure of a preferred embodiment.
[0084] [Item 1] A collaborative work management device that manages a collaborative work system including a work robot and a remote control terminal connected to the work robot via a communications network and operated by a worker, the collaborative work management device including a processing circuit configured to: acquire data indicating at least one of information regarding the worker's perception, judgment, or instructions, and information regarding the work robot's operation; calculate a plurality of feature amounts based on the acquired data; calculate an evaluation value for the work result of the work robot; and calculate the contribution rates of the plurality of feature amounts to the evaluation value.
[0085] This configuration makes it possible to identify feature values that contribute highly to the evaluation value of the work results of the work robot (e.g., work time). By improving the worker's ability with respect to these feature values, it is possible to accurately improve the evaluation value of the work results. This can efficiently support the improvement of the evaluation value of the work results in collaborative work between humans and work robots.
[0086] [Item 2] The collaborative work management device according to Item 1, wherein the processing circuit is further configured to calculate a capability score corresponding to the feature quantity having the largest contribution rate among the plurality of feature quantities, compare the capability score with a reference score, and output a result of comparing the capability score with the reference score.
[0087] This configuration makes it possible to easily grasp the capabilities of a worker or a working robot required to improve a feature with a high contribution rate.
[0088] [Item 3] The collaborative work management device according to Item 2, wherein calculating the ability score includes: calculating a plurality of detailed feature amounts obtained by subdividing the feature amount having the largest contribution rate; and calculating the ability score based on the detailed feature amounts.
[0089] This configuration makes it possible to accurately determine the capabilities of a worker or a working robot required to improve a feature with a high contribution rate.
[0090] [Item 4] The collaborative work management device according to any one of items 1 to 3, wherein the evaluation value includes a task time.
[0091] This configuration makes it possible to accurately improve the capabilities of the worker or work robot, thereby contributing to achieving the target work time.
[0092] [Item 5] The collaborative work management device according to Item 4, wherein the work time of the evaluation value includes at least one unit work time required for at least one unit work configured by a plurality of element works, and the plurality of feature amounts include a plurality of element work times required for each of the plurality of element works.
[0093] With this configuration, it is possible to grasp which of the multiple elemental tasks that make up a unit task should be improved to contribute to achieving the target task time.
[0094] [Item 6] The collaborative work management device according to any one of Items 1 to 5, wherein the evaluation value includes a plurality of unit work times required for a unit work that is made up of a plurality of element works and is arranged in chronological order, the plurality of unit works including a plurality of element works that correspond to each other, and the processing circuit is further configured to: calculate a time-series increase / decrease trend of the plurality of element work times required for the plurality of element works that correspond to each other; calculate a continuous work ability score based on the increase / decrease trend; compare the continuous work ability score with a continuous work standard score; and output a result of comparing the continuous work ability score with the continuous work standard score.
[0095] This configuration makes it easy to grasp the capabilities of a worker or a work robot in relation to changes in element work times over time.
[0096] [Item 7] A method for managing a collaborative work system comprising a work robot and a remote control terminal connected to the work robot via a communications network and operated by a worker, the method comprising: acquiring data indicating at least one of information regarding the worker's perception, judgment, or instructions, and information regarding the work robot's operation; calculating a plurality of feature amounts based on the acquired data; calculating an evaluation value for the work result of the work robot; and calculating the contribution rate of the plurality of feature amounts to the evaluation value.
[0097] This method makes it possible to identify feature values that contribute significantly to the evaluation value of the work results of a work robot (e.g., work time). By improving the worker's ability with respect to these feature values, it is possible to accurately improve the evaluation value of the work results. This method therefore efficiently supports the improvement of the evaluation value of work results in collaborative work between humans and work robots.
[0098] [Item 8] A collaborative work management program that causes at least one processor to execute the collaborative work management method according to Item 7.
[0099] Similarly to the above, this program can efficiently support improved evaluation of work results in collaborative work between humans and work robots.
[0100] REFERENCE SIGNS LIST 1 Collaborative work system 2 Work robot 3 Remote control terminal 4 Worker 5 Communication network 10 Collaborative work management server (collaborative work management device) 60 Processing circuit 61 Processor 71 Principal component analysis model E1 to E5 Element work Te Element work time Tv Unit work time V Unit work
Claims
1. A collaborative work management device that manages a collaborative work system including a work robot and a remote control terminal connected to the work robot via a communications network and operated by a worker, the collaborative work management device having a processing circuit configured to: acquire data indicating at least one of information regarding the perception, judgment or instructions of the worker, or information regarding the operation of the work robot; calculate a plurality of feature amounts based on the acquired data; calculate an evaluation value for the work result of the work robot; and calculate the contribution rate of the plurality of feature amounts to the evaluation value.
2. The collaborative work management device of claim 1, wherein the processing circuit is further configured to: calculate a capability score corresponding to the feature among the plurality of features having the maximum contribution rate; compare the capability score with a standard score; and output a result of comparing the capability score with the standard score.
3. The collaborative work management device of claim 2, wherein calculating the ability score includes: calculating a plurality of detailed features by subdividing the feature with the largest contribution rate; and calculating the ability score based on the detailed features.
4. The collaborative work management device according to claim 1, wherein the evaluation value includes a work time.
5. A collaborative work management device as described in claim 4, wherein the work time of the evaluation value includes at least one unit work time required for at least one unit work composed of a plurality of element works, and the plurality of feature amounts include a plurality of element work times required for each of the plurality of element works.
6. A collaborative work management device as claimed in any one of claims 1 to 3, wherein the evaluation value includes a plurality of unit work times each required for a unit work composed of a plurality of element works arranged in chronological order, the plurality of unit works including a plurality of element works each corresponding to each other, and the processing circuit is further configured to: calculate a chronological increase / decrease trend of the plurality of element work times each required for the plurality of element works corresponding to each other, calculate a continuous work ability score based on the increase / decrease trend, compare the continuous work ability score with a continuous work standard score, and output a result of comparing the continuous work ability score with the continuous work standard score.
7. A method for managing a collaborative work system comprising a work robot and a remote control terminal connected to the work robot via a communications network and operated by a worker, the collaborative work management method including: acquiring data indicating at least one of information regarding the worker's perception, judgment or instructions, or information regarding the operation of the work robot; calculating a plurality of feature amounts based on the acquired data; calculating an evaluation value for the work results of the work robot; and calculating the contribution rate of the plurality of feature amounts to the evaluation value.
8. A collaborative work management program that causes at least one processor to execute the collaborative work management method according to claim 7.
Citation Information
Patent Citations
Network flow management device
JP1995141426A
Process improving and supporting method and device thereof
JP1998249683A
Evaluation program, evaluation device, and evaluation method
JP2016038754A
Information processing device, information processing method, and information processing program
JP7154468B2
Operation sequence identification device, operation sequence identification system, operation sequence identification method, and program
WO2020017160A1
