Skills teaching system

Abstract

This system provides a skills teaching system that facilitates the comparison and learning of the differences between skilled workers and trainee workers, and enables the easy transfer of skills. [Solution] The skill teaching system 1 teaches the skills required to operate the hydraulic excavator 2 to the learning worker. The skill teaching system 1 comprises a model generation unit 10b, an estimation unit 10c, and a comparison display unit 10d. The model generation unit 10b obtains a first learning model 7a by performing machine learning on training data, which is input data of skilled operator operation trajectory data and working conditions, and output data of skilled operator's viewpoint trajectory data. The estimation unit 10c inputs worker operation trajectory data and working conditions as estimation input data into the first learning model 7a to obtain skilled operator's viewpoint estimated trajectory data. The comparison display unit 10d displays the skilled operator's viewpoint estimated trajectory data and the worker's viewpoint trajectory data overlaid on the worker's video data for comparison.

Description

Technical Field

[0001] The present invention relates to a skill teaching system that can efficiently teach learning workers the skills of skilled workers when operating construction machinery.

Background Art

[0002] Conventionally, it has been generally known that it is difficult to systematically transmit the skills of skilled workers to others, and a great deal of time and effort are required to acquire them. In particular, when operating construction machinery at a construction site, it is difficult to operate while guiding others in the cab, and there has been a problem that it is difficult to inherit skills.

[0003] To address this, it is conceivable to have learning workers watch the work scenes of skilled workers to learn their skills. For example, the learning support device of Patent Document 1 is for efficiently acquiring the demonstration operations of skilled workers in medical care and nursing. It includes a camera that shoots a video of the demonstration operation, a model video storage area that stores a model video of the exemplary demonstration operation, and a practice video storage area that stores a practice video of the learning worker's demonstration operation. On the display device, a first partial display screen for displaying the model video and a second partial display screen for displaying the practice video are arranged and displayed simultaneously. In the model video, the position of the skilled worker's line of sight is displayed, while in the practice video, the position of the learning worker's line of sight is displayed, enabling comparative learning using the two images.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, as described in Patent Document 1, the detailed movements and demonstration speed of skilled workers are not exactly the same as those of learners. Therefore, when learners try to compare the demonstration points of the skilled workers they are interested in with their own practice points, they face the problem that the practice points are not displayed at the same time, making it difficult to learn.

[0006] This invention has been made in view of the above, and its purpose is to provide a skills teaching system that facilitates the comparison and learning of the differences between skilled workers and trainee workers, and facilitates the transfer of skills. [Means for solving the problem]

[0007] To achieve the above objectives, the present invention is characterized by its ingenuity in enabling the differences between skilled workers and trainee workers to be displayed on the same screen at the same time.

[0008] Specifically, the following measures were taken regarding a skills training system that allows trainee workers to learn the skills used by experienced operators when operating construction machinery.

[0009] In other words, in the first invention, the skilled operator operation trajectory data is extracted from skilled operator video data obtained by filming a skilled operator performing a predetermined task by operating the operating part of the construction machine, and the working conditions at the time the skilled operator video data was obtained are also included. and A model generation unit creates a first learning model by performing machine learning on training data that takes the following as input data and expert eye-tracking trajectory data, which is a record of the expert's eye-tracking trajectory when the expert's video data was obtained, as output data; and worker operation trajectory data obtained by extracting the posture and position trajectory of the operating part from worker video data obtained by filming the learning worker operating the operating part of the construction machine to perform the work, and the work conditions when the worker video data was obtained. andThe system is characterized by comprising: an estimation unit that inputs the data as estimation input data to the first learning model to estimate the trajectory of the skilled worker's gaze in the worker video data and obtains skilled worker gaze-line estimation trajectory data; and a comparison display unit that can overlay and display the worker gaze-line trajectory data, which is a record of the trajectory of the learned worker's gaze when the worker video data was obtained, and the skilled worker gaze-line estimation trajectory data on the worker video data for comparison.

[0010] The second invention is characterized in that, in the first invention, it comprises: a calculation unit that calculates the distance between eye lines between worker eye line trajectory data and the estimated eye line trajectory data of the skilled worker; a storage unit that stores a first evaluation table in which evaluation points corresponding to the distance between eye lines are predetermined; and a skill evaluation unit that calculates evaluation points based on the first evaluation table from the distance between eye lines calculated by the calculation unit.

[0011] In the third invention, in the second invention, the model generation unit takes the working conditions, the starting position of the operating unit, and the work start area of ​​the operating unit in the skilled worker video data as input data, and the skilled worker movement operation trajectory data, which is a record of the trajectory of the attitude and position of the operating unit during a movement operation in which the skilled worker operates the operating unit of the construction machine from the starting position to the work start area, as training data to create a second learning model by performing machine learning from the training data. The estimation unit inputs the working conditions, the starting position of the operating unit, and the work start area of ​​the operating unit in the worker video data as estimation input data to the second learning model to estimate the attitude and position of the operating unit as operated by the skilled worker in the worker video data. The system acquires estimated trajectory data of the movement of a skilled worker, and the calculation unit calculates the time it takes for the trainee worker to move from the starting position to the work start area based on the worker movement trajectory data which records the trajectory of the posture and position of the operating part when the trainee worker performs the movement operation, and also calculates the time it takes for the trainee worker to move from the starting position to the work start area based on the estimated trajectory data of the trainee worker's movement operation, calculates the time difference between the trainee worker's operation time and the trainee worker's operation time, the storage unit stores a second evaluation table in which evaluation points corresponding to the time difference are predetermined, and the skill evaluation unit calculates an evaluation point based on the second evaluation table from the time difference calculated by the calculation unit.

[0012] In the fourth invention, in the second invention, the model generation unit takes expert specific work operation trajectory data as input data, which is a record of the trajectory of the posture and position of the operating part when the expert operates the operating part of the construction machine to perform a specific operation on the work area in the expert video data, and performs machine learning from training data which is the number of times the expert performed the specific operation in the work content as output data, to create a third learning model, and the estimation unit takes the worker video data as input data when the learning worker operates the operating part of the construction machine to perform the specific operation on the work area The operator-specific operation trajectory data, which records the trajectory of the posture and position of the operating part when performing the operation, is input to the third learning model as estimation input data to estimate the number of times the learning operator performed the specific operation. The calculation unit calculates the difference between the number of times the skilled worker performed the specific operation and the estimated number of times the learning operator performed the specific operation. The storage unit stores a third evaluation table in which evaluation points corresponding to the difference in the number of times are predetermined. The skill evaluation unit calculates an evaluation point based on the third evaluation table from the difference in the number of times calculated by the calculation unit.

[0013] The fifth invention is characterized in that, in any one of the first to fourth inventions, the working conditions are at least one of the type of work, the work location, and the object of work. [Effects of the Invention]

[0014] In the first invention, the learning worker can see the trajectory of their own eye position during the work and the trajectory of the estimated eye position of the expert in the same video. Therefore, the learning worker can not only understand where the expert would have looked when working on the operating part of the construction machine, but can also compare the difference between their own eye position and the estimated eye position where the expert would have looked in the same video, thus efficiently learning the differences between the expert and the learning worker.

[0015] The second invention makes it possible to determine how much the eye position of a learning worker differs from the estimated eye position of a skilled worker during work. Therefore, it becomes possible to quantitatively determine the skill level of the learning worker's eye position during work, which can be used to help improve the learning worker's skills.

[0016] In the third invention, it becomes possible to determine the difference between an experienced worker and a trainee worker in moving the operating part of a construction machine from the starting position to the work start area, and in positioning the operating part to the optimal posture and position for starting work in that work start area. Therefore, it becomes possible to quantitatively determine the trainee worker's level of proficiency in operating construction machines, which can be used to improve the trainee worker's skills.

[0017] The fourth invention makes it possible to determine how many more times a trainee worker performs a specific operation on the work area using the operating parts of a construction machine compared to a skilled worker. In other words, it becomes possible to determine the number of reworks in a given task. Therefore, it becomes possible to determine whether the trainee worker is able to operate the operating parts of the construction machine smoothly in a series of movements, which can be used to help trainee workers improve their skills.

[0018] In the fifth invention, since a learning model is created using machine learning with the working conditions specifically identified, the accuracy of estimated data can be improved, such as the estimated eye position of a skilled worker. [Brief explanation of the drawing]

[0019] [Figure 1] This is a block diagram of a skills teaching system in an embodiment of the present invention. [Figure 2] This is a schematic side view of a hydraulic excavator operated by a skilled worker or a trainee worker. [Figure 3] This figure, equivalent to Figure 2, schematically shows the trajectory of the posture and position of the excavation attachment and the trajectory of the operator's line of sight during the movement operation in which a skilled or trainee operator moves the excavation attachment from the starting position to the work area. [Figure 4] This is a diagram schematically showing the posture and position trajectory of the excavation attachment and the trajectory of the line of sight during operation, immediately after a skilled operator or trainee operator starts operating the excavation attachment of the hydraulic excavator to perform a specific operation on the work target, after FIG. 3. [Figure 5] This is a diagram schematically showing the posture and position trajectory of the excavation attachment and the trajectory of the line of sight during operation, immediately before a skilled operator or trainee operator finishes operating the excavation attachment of the hydraulic excavator to complete a specific operation on the work target, after FIG. 4. [Figure 6] This is a flowchart showing the procedure of the skill evaluation process implemented by the skill teaching system according to an embodiment of the present invention. [Figure 7] This is a flowchart showing the procedure of the work implemented in step S1 of FIG. 6. [Figure 8] This is a flowchart showing the procedure of the work implemented in step S2 of FIG. 6. [Figure 9] This is a flowchart showing the procedure of the work implemented in step S3 of FIG. 6. [Figure 10] This is a flowchart showing the procedure of the work implemented in step S4 of FIG. 6. [Figure 11] This is a flowchart showing the procedure of the work implemented in step S5 of FIG. 6. [Figure 12] This is a diagram showing the first teacher data used when creating the first learning model in the model generation unit. [Figure 13] This is a diagram showing the second teacher data used when creating the second learning model in the model generation unit. [Figure 14] This is a diagram showing the third teacher data used when creating the third learning model in the model generation unit. [Figure 15]This is an evaluation table used to assess the difference between an experienced operator and a trainee operator when operating a hydraulic excavator. (a) is used to evaluate the distance between eye levels, (b) is used to evaluate the time difference in movement operations to a predetermined position, (c) is used to evaluate the number of reworks in a specific operation, and (d) is used to evaluate the time difference from the start to the end of the work. [Figure 16] This is an example of a video showing the eye position of a trainee worker and the estimated eye position of a skilled worker on the same screen during rock crushing work using a crusher. (a) is a picture taken immediately after the trainee worker started crushing the rock using the crusher, and (b) is a picture taken immediately before the trainee worker finished crushing the rock using the crusher. [Figure 17] This diagram shows a video of a trainee worker's eye level during rock crushing using a crusher, along with a diagram illustrating the trainee worker's skill assessment at the time the video was obtained. [Modes for carrying out the invention]

[0020] The embodiments of the present invention will be described in detail below with reference to the drawings. Note that the following description of preferred embodiments is essentially illustrative.

[0021] Figure 1 shows a skills teaching system 1 according to an embodiment of the present invention. This skills teaching system 1 consists of a hydraulic excavator 2 (construction machine) and a control system 3, and is designed to allow a skilled worker H to teach a learning worker h the skills required to operate the hydraulic excavator 2.

[0022] As shown in Figure 2, the hydraulic excavator 2 comprises a crawler-type lower traveling body 2A and an excavator body 2B mounted above the lower traveling body 2A via a slewing mechanism, and the excavator body 2B is capable of slewing around a slewing axis that extends in the vertical direction.

[0023] On one side in the width direction of the shovel body 2B, a box-shaped cab 21 is provided to protect the skilled operator H or trainee worker h operating the hydraulic shovel 2.

[0024] On the other side of the shovel body 2B in the width direction, an excavation attachment 22 (operating part) is provided, and this excavation attachment 22 is equipped with a boom 22a, an arm 22b, and a bucket 22c in that order from the base end.

[0025] Inside the cab 21, there is a cockpit 23 equipped with an operating lever 23a, allowing a skilled worker H or a trainee worker h to operate the excavation attachment 22 using the operating lever 23a from the cockpit 23 to perform predetermined tasks.

[0026] For example, as shown in Figure 3, an experienced worker H or a trainee worker h can perform a movement operation M to move the excavation attachment 22 from the starting position P1, where the excavation attachment 22 is bent in a roughly V-shape, to the work start area P2, which is set to be diagonally downward and forward.

[0027] Furthermore, as shown in Figures 4 and 5, for example, an experienced worker H or a trainee worker h can perform a specific operation X: first, rotate the bucket 22c of the excavation attachment 22 towards the front from the work start area P2 to the excavation position P3 to excavate soil in the excavation area R1 (work area); and then rotate the bucket 22c away from the excavation position P3 to the embankment position P4 to drop the excavated soil in the bucket 22c into the embankment area R2.

[0028] As shown in Figure 2, a work camera 2a capable of capturing images of a predetermined range F in front of the cab 21 is suspended above the cockpit 23.

[0029] This work-recording camera 2a is designed to capture video data V1 of the skilled worker H operating the excavation attachment 22 of the hydraulic excavator 2 to perform a predetermined task.

[0030] Furthermore, the work recording camera 2a is designed to capture worker video data v1 by recording the trainee worker h operating the excavation attachment 22 of the hydraulic excavator 2 to perform predetermined tasks.

[0031] Furthermore, a wearable eye-tracking device 2b is installed in the cockpit 23. A skilled worker H or a trainee worker h can operate the excavation attachment 22 of the hydraulic excavator 2 while wearing the eye-tracking device 2b.

[0032] The eye-tracking device 2b measures the coordinate data of the eye position E of the expert H in the expert video data V1 at predetermined time intervals to acquire expert eye trajectory data ET1 of the expert H.

[0033] Furthermore, the eye-tracking device 2b measures the coordinate data of the eye position e of the trainee worker h in the worker video data v1 at predetermined time intervals to acquire the trainee worker h's eye-tracking trajectory data ET2.

[0034] Furthermore, if the time interval for measuring the coordinate data of the eye position E of the expert H and the eye position e of the learning worker h is too short, the eye position E and eye position e will appear to fluctuate significantly when displayed on the monitor 11 described later. Therefore, the expert eye trajectory data ET1 and the worker eye trajectory data ET2 are adjusted by thinning out the measured coordinate data so that the eye position E and eye position e do not change significantly.

[0035] While Microsoft's "HoloLens 2" is a specific example of an eye-tracking device 2b, eye-tracking trajectory data may also be acquired using other devices with equivalent performance.

[0036] As shown in Figure 1, the construction machine-side communication unit 2c is a communication device capable of communicating with the control system 3 via a communication network N, and is configured to send and receive information between it and the work camera 2a and the eye-tracking device 2b. The communication network N includes, for example, the Internet, Wi-Fi, Wi-SUN (Wireless Smart Utility Network), WAN (Wide Area Network), LAN (Local Area Network), etc.

[0037] The control system 3 consists of a system-side communication unit 4, a storage unit 5, a control unit 10, and a monitor 11. The system-side communication unit 4 is connected to the construction machine-side communication unit 2c of the hydraulic excavator 2 via a communication network N.

[0038] The system-side communication unit 4 is configured to send and receive information with the storage unit 5, the control unit 10, and the monitor 11.

[0039] The monitor 11 is a display device connected to the storage unit 5 and the control unit 10, respectively, and is capable of displaying images captured by the work camera 2a and various other information.

[0040] The memory unit 5 includes a training data storage unit 6 for storing training data for machine learning, a model storage unit 7 for storing a learning model created by machine learning the training data, an evaluation data storage unit 8 for storing an evaluation table used when evaluating the skills of the learning worker h, and an evaluation result storage unit 9 for storing the evaluation results from the skills evaluation unit 10f, which will be described later.

[0041] The training data storage unit 6 stores the following: first training data 6a for machine learning to estimate the trajectory of the eye position E of the skilled worker H; second training data 6b for machine learning to estimate the movement operation M of the excavation attachment 22 of the hydraulic excavator 2 performed by the skilled worker H; third training data 6c for machine learning to estimate the number of times a specific operation X of the excavation attachment 22 of the hydraulic excavator 2 is performed by the learning worker h; and fourth training data 6d for machine learning to output the posture and position of the excavation attachment 22 of the hydraulic excavator 2 from the video.

[0042] The model memory unit 7 stores the following: a first learning model 7a created by machine learning the first training data 6a; a second learning model 7b created by machine learning the second training data 6b; a third learning model 7c created by machine learning the third training data 6c; and a fourth learning model 7d created by machine learning the fourth training data 6d.

[0043] As shown in Figure 12, the first training data 6a is created by extracting the trajectory of the posture and position of the excavation attachment 22 from the expert video data V1, and using the working conditions W at the time the expert video data V1 was obtained as input data, while using the expert's eye-line trajectory data ET1 as output data, and is stored linked together.

[0044] The aforementioned working conditions W consist of the type of work, the work location, and the work object. The type of work may be, for example, excavation or crushing. The type of work may also be the ground or rock. Furthermore, the work object may be an excavation site or a quarry.

[0045] As shown in Figure 13, the second training data 6b uses the working conditions W, the starting position P1 of the excavation attachment 22, and the working start area P2 of the excavation attachment 22 from the expert video data V1 as input data, while the expert movement operation trajectory data PT1, which records the trajectory of the posture and position of the excavation attachment 22 during the movement operation M in which the expert H operates the excavation attachment 22 of the hydraulic excavator 2 from the starting position P1 to the working start area P2, is used as output data and is stored linked together.

[0046] As shown in Figure 14, the third training data 6c uses expert specific operation trajectory data PT2, which records the trajectory of the posture and position of the excavation attachment 22 when an expert H operates the excavation attachment 22 of the hydraulic excavator 2 to perform a specific operation X on the work object, as input data, while the number of times expert H performed the specific operation X in the work content is used as output data, and these are stored linked together.

[0047] The fourth training data 6d uses video footage of the excavation attachment 22 of the hydraulic excavator 2 as input data, while outputting the attitude information and position information of the excavation attachment 22 in the video, and these are stored linked together.

[0048] As shown in Figure 15, the evaluation data storage unit 8 stores the first evaluation table 8a, the second evaluation table 8b, the third evaluation table 8c, and the fourth evaluation table 8d.

[0049] The first evaluation table 8a is an evaluation table in which evaluation points are predetermined according to the inter-eye distance D1 (see Figure 16) between the estimated eye position E' of the skilled worker H and the eye position e of the learning worker h in the worker video data v1.

[0050] The second evaluation table 8b is an evaluation table in which evaluation points are predetermined according to the time difference TD1 between the learning worker operation time T1 taken when the learning worker h performs the movement operation M shown in Figure 3, and the expert operation time T2 estimated when the expert H performs the operation in the worker video data v1.

[0051] The third evaluation table 8c is an evaluation table in which evaluation points are predetermined according to the difference C between the number of times the skilled worker H performed the specific operation X shown in Figures 4 and 5 and the estimated number of times the learning worker h performed the specific operation X.

[0052] The fourth evaluation table 8d is an evaluation table in which evaluation points are predetermined according to the time difference TD2 between the time t1 taken for a learning worker h to perform the work and the standard time t2 taken for a skilled worker H to perform the work.

[0053] As shown in Figure 1, the control unit 10 processes various types of information received from the construction machine-side communication unit 2c and the storage unit 5 of the hydraulic excavator 2 via the communication network N.

[0054] The control unit 10 includes a data acquisition unit 10a, a model generation unit 10b, an estimation unit 10c, a comparison display unit 10d, a calculation unit 10e, and a skill evaluation unit 10f.

[0055] The data acquisition unit 10a acquires expert video data V1, which shows the work of an expert H operating the excavation attachment 22, and worker video data v1, which shows the work of a learning worker h operating the excavation attachment 22, via the system-side communication unit 4 and the construction machine-side communication unit 2c.

[0056] Furthermore, the data acquisition unit 10a acquires the expert's eye-tracking trajectory data ET1 of the expert H and the worker's eye-tracking trajectory data ET2 of the trainee worker h, obtained by the eye-tracking device 2b, via the system-side communication unit 4 and the construction machine-side communication unit 2c.

[0057] Furthermore, the data acquisition unit 10a organizes the expert video data V1, expert operation trajectory data PT, expert movement operation trajectory data PT1, expert specific operation trajectory data PT2, expert eye-line trajectory data ET1, and the number of times the specific operation X performed by expert H was performed to generate first teacher data 6a, second teacher data 6b, and third teacher data 6c, and the generated first teacher data 6a, second teacher data 6b, and third teacher data 6c are stored in the teacher data storage unit 6.

[0058] The model generation unit 10b retrieves the first training data 6a stored in the training data storage unit 6 and generates a first learning model 7a by performing machine learning based on this data. This first learning model 7a is capable of estimating the trajectory of the eye position E of the skilled worker H in the worker video data v1. The generated first learning model 7a is stored in the model storage unit 7.

[0059] The model generation unit 10b retrieves the second training data 6b stored in the training data storage unit 6 and generates a second learning model 7b by performing machine learning based on this data. This second learning model 7b is capable of estimating the trajectory of the posture and position of the excavation attachment 22 when a skilled worker H performs a movement operation M in the worker video data v1. The generated second learning model 7b is stored in the model storage unit 7.

[0060] The model generation unit 10b retrieves the third training data 6c stored in the training data storage unit 6 and generates a third learning model 7c by performing machine learning based on this data. This third learning model 7c is capable of estimating the number of times a learning operator h has performed a specific operation X. The generated third learning model 7c is stored in the model storage unit 7.

[0061] The model generation unit 10b retrieves the fourth training data 6d stored in the training data storage unit 6 and generates a fourth learning model 7d based on this data through machine learning. This fourth learning model 7d is capable of estimating the trajectory of the posture and position of the excavation attachment 22 operated by the expert H or the learning worker h in the expert video data V1 or the worker video data v1. The generated fourth learning model 7d is stored in the model storage unit 7.

[0062] The estimation unit 10c inputs the worker operation trajectory data pt, obtained by extracting the trajectory of the posture and position of the excavation attachment 22 from worker video data v1 obtained by filming a trainee worker h operating the excavation attachment 22 of the hydraulic excavator 2 and the working conditions W at the time the worker video data v1 was obtained, as estimation input data to the first learning model 7a, and estimates the trajectory of the estimated eye position E' of the skilled worker H in the worker video data v1 to obtain the skilled worker eye position estimated trajectory data ES1.

[0063] The estimation unit 10c inputs the working conditions W, the starting position P1 of the excavation attachment 22, and the working start area P2 of the excavation attachment 22 from the worker video data v1 into the second learning model 7b as estimation input data to obtain the estimated skilled worker movement trajectory data ES2, which estimates the trajectory of the posture and position of the excavation attachment 22 operated by the skilled worker H in the worker video data v1.

[0064] The estimation unit 10c inputs worker-specific operation trajectory data pt2, which records the trajectory of the posture and position of the excavation attachment 22 when the learning worker h operates the excavation attachment 22 of the hydraulic excavator 2 to perform a specific operation X on the excavation area R1 in the worker video data v1, into the third learning model 7c as estimation input data to estimate the number of times the learning worker h performed the specific operation X.

[0065] The estimation unit 10c inputs the expert operator video data V1 or worker video data v1 as estimation input data to the fourth learning model 7d and estimates the expert operator operation trajectory data PT or worker operation trajectory data pt, which represent the trajectory of the posture and position of the excavation attachment 22 in the expert operator video data V1 or worker video data v1.

[0066] The comparison display unit 10d overlays the worker's eye-line trajectory data ET2 and the expert's eye-line estimated trajectory data ES1 onto the worker's video data v1 and displays them for comparison on the monitor 11.

[0067] The calculation unit 10e calculates the eye-line distance D1 between the worker's eye-line trajectory data ET2 and the expert's eye-line estimated trajectory data ES1 in a time series.

[0068] Furthermore, the calculation unit 10e calculates the learning worker operation time T1 from the starting position P1 to the work start area P2 based on the worker movement operation trajectory data pt1, which records the trajectory of the posture and position of the excavation attachment 22 when the learning worker h performs the movement operation M. It also calculates the expert operator operation time T2 from the starting position P1 to the work start area P2 based on the expert operator movement operation estimated trajectory data ES2, and calculates the time difference TD1 between the learning worker operation time T1 and the expert operator operation time T2.

[0069] Furthermore, the calculation unit 10e calculates the difference C between the number of times the skilled worker H performed a specific operation X and the estimated number of times the trainee worker h performed the specific operation X.

[0070] Furthermore, the calculation unit 10e calculates the time difference TD2 between the time t1 taken for the learning worker h to perform the work and the standard time t2 taken for the skilled worker H to perform the work.

[0071] The skills evaluation unit 10f is designed to perform skills evaluations for skills 1 through 5.

[0072] The skill evaluation unit 10f calculates an evaluation score based on the first evaluation table 8a from the eye-to-eye distance D1 obtained by the calculation unit 10e (first skill evaluation).

[0073] Specifically, as shown in Figure 15(a), for example, if the maximum value of the eye-to-eye distance D1 obtained through calculation is 90 mm, the evaluation score will be calculated as "5".

[0074] The skills evaluation unit 10f calculates an evaluation score based on the second evaluation table 8b from the time difference TD1 obtained by the calculation unit 10e (second skills evaluation).

[0075] Specifically, as shown in Figure 15(b), for example, if the time difference TD1 is 50s, the evaluation score will be calculated as "3".

[0076] The skill evaluation unit 10f calculates an evaluation score based on the third evaluation table 8c from the difference in the number of repetitions C obtained by the calculation unit 10e (third skill evaluation).

[0077] Specifically, as shown in Figure 15(c), for example, if the difference in the number of occurrences C is 5, the evaluation score is calculated as "1".

[0078] The skills evaluation unit 10f calculates an evaluation score based on the fourth evaluation table 8d from the time difference TD2 obtained by the calculation unit 10e (fourth skills evaluation).

[0079] Specifically, as shown in Figure 15(d), for example, if the time difference TD2 is 200s, the evaluation score will be calculated as "4".

[0080] The skills evaluation unit 10f calculates the average value for each evaluation score of the first to fourth skills evaluations, calculates the standard deviation, and then calculates the evaluation score by subtracting the standard deviation from the average value (fifth skills evaluation).

[0081] The first skill evaluation assesses "eye alignment," the second skill evaluation assesses "work proficiency," the third skill evaluation assesses "work efficiency," the fourth skill evaluation assesses "work time," and the fifth skill evaluation assesses "overall balance."

[0082] Next, the workflow performed in the skills teaching system 1 according to an embodiment of the present invention will be described in detail.

[0083] As shown in Figure 6, first, in step S1, the first skill evaluation process is performed. Specifically, based on the flowchart shown in Figure 7, the process is performed so that the eye position e of the learning worker h and the estimated eye position E' of the skilled worker H are displayed simultaneously in the worker video data v1.

[0084] First, in step SA1, the work scene of skilled worker H operating the excavation attachment 22 of the hydraulic excavator 2 to perform a predetermined task is captured by the work camera 2a, and the data acquisition unit 10a acquires the captured video as skilled worker video data V1. The acquired skilled worker video data V1 is stored in the training data storage unit 6. In addition, the work conditions W during the skilled worker H's work are input to the control system 3 using an input device such as a mouse or keyboard and are stored in the training data storage unit 6.

[0085] In step SA2, the model memory unit 7 reads the expert operator video data V1 from the training data memory unit 6 and inputs it into the fourth learning model 7d to obtain the expert operator operation trajectory data PT. The obtained expert operator operation trajectory data PT is stored in the training data memory unit 6, and the process proceeds to step SA3.

[0086] In step SA3, the eye-tracking device 2b measures the eye position E in the expert's video data V1 when the expert H operates the excavation attachment 22 of the hydraulic excavator 2 to perform the above-mentioned work. The data acquisition unit 10a acquires expert's eye-tracking trajectory data ET1, which is obtained by continuously capturing the coordinate data of the eye position E, and proceeds to step SA4.

[0087] In step SA4, the model generation unit 10b retrieves the first training data 6a stored in the training data storage unit 6, performs machine learning based on this data to generate the first learning model 7a, and then proceeds to step SA5.

[0088] In step SA5, the work scene is filmed by the work camera 2a as the learning worker h operates the excavation attachment 22 of the hydraulic excavator 2 to perform a predetermined task, and the data acquisition unit 10a acquires the video footage as worker video data v1. The acquired worker video data v1 is stored in the teacher data storage unit 6. In addition, the work conditions W during the learning worker h's work are input to the control system 3 using an input device such as a mouse or keyboard and are stored in the teacher data storage unit 6.

[0089] In step SA6, the model generation unit 10b reads worker video data v1 from the training data storage unit 6 and inputs it into the first learning model 7a along with the work conditions W to estimate the trajectory of the skilled worker H's eye position E in the worker video data v1, obtains the skilled worker eye position estimation trajectory data ES1, and proceeds to step SA7.

[0090] In step SA7, the eye-tracking device 2b measures the eye position e in the worker video data v1 when the learning worker h operates the excavation attachment 22 of the hydraulic excavator 2 to perform the above-described work. The data acquisition unit 10a acquires worker eye-tracking trajectory data ET2, which is obtained by continuously capturing the coordinate data of the eye position e, and proceeds to step SA8.

[0091] In step SA8, the comparison display unit 10d overlays the worker's eye-line trajectory data ET2 and the expert's eye-line estimated trajectory data ES1 onto the worker's video data v1 and displays them for comparison on the monitor 11, then proceeds to step SA9.

[0092] In step SA9, the calculation unit 10e calculates the eye-line distance D1 between the worker's eye-line trajectory data ET2 and the expert's eye-line estimated trajectory data ES1 in a time series. Then, the skill evaluation unit 10f calculates the evaluation score from the eye-line distance D1 based on the first evaluation table 8a, and then proceeds to step S2 in Figure 6.

[0093] In step S2, a second skill evaluation process is performed. Specifically, based on the flowchart shown in Figure 8, a process is performed to output the time difference TD1 when the skilled worker H and the learning worker h perform a predetermined movement operation M of the excavation attachment 22 of the hydraulic excavator 2.

[0094] In step SB1, the starting position P1 and work start area P2 of the excavation attachment 22 of the hydraulic excavator 2 in the expert video data V1 are set to the control system 3 using an input device such as a mouse or keyboard, and then the user proceeds to step SB2.

[0095] In step SB2, the control unit 10 extracts data from the skilled operator operation trajectory data PT corresponding to the starting position P1 of the excavation attachment 22 of the hydraulic excavator 2 to the work start area P2, obtains the skilled operator movement trajectory data PT1, and proceeds to step SB3.

[0096] In step SB3, the model generation unit 10b retrieves the second training data 6b stored in the training data storage unit 6, performs machine learning based on this data to generate a second learning model 7b, and then proceeds to step SB4.

[0097] In step SB4, the model generation unit 10b reads worker video data v1 from the training data storage unit 6 and inputs it into the second learning model 7b along with the information of the starting position P1, the work starting area P2, and the work conditions W. The model then estimates the trajectory of the posture and position of the excavation attachment 22 when the skilled worker H performs the movement operation M in the worker video data v1, obtains the skilled worker movement operation estimated trajectory data ES2, and proceeds to step SB5.

[0098] In step SB5, the control unit 10 extracts data from the worker operation trajectory data pt corresponding to the starting position P1 of the excavation attachment 22 of the hydraulic excavator 2 to the work start area P2, obtains the worker movement operation trajectory data pt1, and proceeds to step SB6.

[0099] In step SB6, the calculation unit 10e calculates the learned worker operation time T1, which is the time it takes to travel from the starting position P1 to the work start area P2, based on the worker movement operation trajectory data pt1. The calculation unit 10e also calculates the expert worker operation time T2, which is the time it takes to travel from the starting position P1 to the work start area P2, based on the expert worker movement operation estimated trajectory data ES2, and then proceeds to step SB7.

[0100] In step SB7, the calculation unit 10e calculates the time difference TD1 between the learning worker's operation time T1 and the skilled worker's operation time T2. Then, the skill evaluation unit 10f calculates an evaluation score based on the second evaluation table 8b using the time difference TD1 obtained by the calculation unit 10e, and then proceeds to step S3 in Figure 6.

[0101] Step S3 performs a third skill evaluation process. Specifically, based on the flowchart shown in Figure 9, the process outputs the difference C in the number of times an expert H and a trainee worker h perform a predetermined specific operation X on the excavation attachment 22 of the hydraulic excavator 2.

[0102] In step SC1, using an input device such as a mouse or keyboard, the user sets the operation X in the expert video data V1 as the period from the starting work area P2 to the embankment position P4 via the excavation position P3, and then proceeds to step SC2.

[0103] In step SC2, the control unit 10 extracts data from the skilled operator operation trajectory data PT from the data of the excavation attachment 22 of the hydraulic excavator 2 from the work start area P2 through the excavation position P3 to the embankment position P4, thereby obtaining skilled operator identification operation trajectory data PT2 and proceeding to step SC3.

[0104] In step SC3, the number of times a specific operation X is performed by the expert H in the expert video data V1 is input to the control system 3 using an input device such as a mouse or keyboard, stored in the teacher data storage unit 6, and the process proceeds to step SC4.

[0105] In step SC4, the model generation unit 10b retrieves the third training data 6c stored in the training data storage unit 6, performs machine learning based on this data to generate the third learning model 7c, and proceeds to step SC5.

[0106] In step SC5, the control unit 10 extracts data from the worker operation trajectory data pt showing the excavation attachment 22 of the hydraulic excavator 2 moving from the work start area P2 through the excavation position P3 to the embankment position P4, thereby obtaining worker-specific operation trajectory data pt2 and proceeding to step SC6.

[0107] In step SC6, the model generation unit 10b inputs the worker identification operation trajectory data pt2 into the third learning model 7c to estimate the number of times the learned worker h performed the specific operation X in the worker video data v1, obtains the number of specific operations, and proceeds to step SC7.

[0108] In step SC7, the difference C between the number of times the skilled worker H performed a specific operation X and the estimated number of times the learning worker h performed the specific operation X is calculated. Then, the skill evaluation unit 10f calculates an evaluation score based on the third evaluation table 8c from the difference C obtained by the calculation unit 10e, and then proceeds to step S4 in Figure 6.

[0109] Step S4 performs the fourth skill assessment process. Specifically, based on the flowchart shown in Figure 10, the process outputs the difference in the time it takes for the skilled worker H and the learning worker h to complete all tasks.

[0110] In step SD1, the control unit 10 extracts the reference time t2 for the excavation attachment 22 of the hydraulic excavator 2 to move from the starting position P1 to the embankment position P4 and perform the work from the skilled operator trajectory data PT, and proceeds to step SD2.

[0111] In step SD2, the control unit 10 extracts the execution time t1 during which the excavation attachment 22 of the hydraulic excavator 2 moves from the starting position P1 to the embankment position P4 and performs the work from the worker operation trajectory data pt, and proceeds to step SD3.

[0112] In step SD3, the calculation unit 10e calculates the time difference TD2 between the time t1 taken for the learning worker h to perform the work and the standard time t2 taken for the skilled worker H to perform the work. Then, the skill evaluation unit 10f calculates the evaluation score based on the fourth evaluation table 8d from the time difference TD2 calculated by the calculation unit 10e, and then proceeds to step S5 in Figure 6.

[0113] Step S5 involves the assessment of the fifth skill. Specifically, the variability of the first to fourth skill assessments is evaluated based on the flowchart shown in Figure 11.

[0114] In step SE1, the skills evaluation unit 10f reads the evaluation results for the first to fourth skills and proceeds to step SE2.

[0115] In step SE2, the skills evaluation unit 10f calculates the average value and standard deviation for each evaluation score of the first to fourth skills evaluations, and then calculates the evaluation score by subtracting the standard deviation from the average value, after which it proceeds to step S6 in Figure 6.

[0116] In step S6, as shown in Figure 17, the work in the skills teaching system 1 is completed by presenting the results of the first to fifth skills evaluations as a radar chart.

[0117] As described above, according to the embodiment of the present invention, the learning worker h can see the trajectory of their own eye position e during the work they performed and the trajectory of the estimated eye position E' of the skilled worker H within the same video. Therefore, the learning worker h can not only understand where the skilled worker H would have been looking relative to the position of the excavation attachment 22 of the hydraulic excavator 2, but can also compare the difference between their own eye position e and the estimated eye position E' where the skilled worker H is expected to have looked within the same video, thus efficiently learning the differences between the skilled worker H and the learning worker h.

[0118] Furthermore, since the distance D1 between the estimated eye position E' of the skilled worker H and the eye position e of the learning worker h can be determined, it becomes possible to determine how much the learning worker h's eye position e differed from the skilled worker H's eye position E during the work. Therefore, it becomes possible to quantitatively determine the skill level of the learning worker h's eye position e during the work, which can be used to improve the learning worker h's skills.

[0119] Furthermore, the difference between an experienced operator H and a learning worker h in moving the excavation attachment 22 of the hydraulic excavator 2 from the starting position P1 to the work start area P2, and in positioning the excavation attachment 22 to the optimal posture and position for starting work in the work start area P2, can be determined. Therefore, the level of proficiency of the learning worker h in operating the hydraulic excavator 2 can be quantitatively determined, and this can be used to improve the skills of the learning worker h.

[0120] Furthermore, it becomes possible to determine how many more times the learning worker h performed a specific operation X on the excavation area R1 using the excavation attachment 22 of the hydraulic excavator 2 compared to the skilled worker H. In other words, it becomes possible to determine the number of reworks in a given operation. Therefore, it becomes possible to determine whether the learning worker h is able to operate the excavation attachment 22 of the hydraulic excavator 2 smoothly in a series of movements, which can be used to improve the skills of the learning worker h.

[0121] Furthermore, since the learning model is created using machine learning with the working conditions W specifically identified, the accuracy of estimated data can be improved, such as the estimated eye position E' of the skilled worker H.

[0122] The skill teaching system 1 in this embodiment of the present invention allows a learning worker h to learn how to operate the excavation attachment 22 of a hydraulic excavator 2, but it can also be applied to other construction machinery. For example, Figure 16 shows the skill teaching system 1 applied to the operation of a crusher, with the worker video data v1 displayed on the monitor 11. It can be seen that the estimated eye position E' of the skilled worker H and the eye position e of the learning worker h can be compared on the screen.

[0123] Furthermore, as shown in Figure 17, the learning status of worker h may be displayed in a radar chart next to the worker video data v1 display. This allows for a quick comparison of each evaluation, such as "gaze misalignment," "work proficiency," "work efficiency," "work time," and "overall balance," making it easy to understand the learning rate of worker h.

[0124] In the embodiment of the present invention, when evaluating the movement operation M of the learning worker h, the operation immediately after the start of the work (movement from the starting position P1 to the work start area P2) is evaluated. However, the invention is not limited to this, and for example, other timings towards the end of the work may be evaluated as movement operations M.

[0125] Furthermore, in the embodiments of the present invention, when evaluating the specific operation X of the learning worker h, the operation during excavation is evaluated. However, the invention is not limited to this, and for example, operations such as setting the bucket 22c to a predetermined position or the crushing operation of the crusher may be evaluated as the specific operation X.

[0126] Furthermore, in the embodiments of the present invention, the learning model is trained using multiple data sets with different types of work, work targets, and work locations as training data. However, the invention is not limited to this, and a large number of data sets of the same type may be used for training. Also, it is not necessary to use all of the work type, work target, and work location as input data; it is sufficient to use at least one of them as input data.

[0127] Furthermore, the numerical values ​​and ranges set in each evaluation table of the embodiments of the present invention are examples only, and other numerical values ​​and ranges may be set. Also, the evaluation method is not limited to the method shown in Figure 15. [Industrial applicability]

[0128] This invention is suitable for a skills teaching system that can efficiently teach the skills that experienced operators use when operating construction machinery to trainee workers. [Explanation of Symbols]

[0129] 1…Skills Teaching System 2…Hydraulic Excavator 2A…Lower Traveling Body 2B…Excavator Body 2a…Camera for Filming Work 2b…Eye Tracking Device 2c…Construction Machine Side Communication Unit 3…Control System 4…System Side Communication Unit 5…Storage Unit 6…Training Data Storage Unit 6a…First Training Data 6b…Second Training Data 6c…Third Training Data 6d…Fourth Training Data 7…Model Storage Unit 7a…First Learning Model 7b…Second Learning Model 7c…Third Learning Model 7d…Fourth Learning Model 8…Evaluation Data Storage Unit 8a…First Evaluation Table 8b…Second Evaluation Table 8c…Third Evaluation Table 8d…Fourth Evaluation Table 9…Evaluation Result Storage Unit 10…Control Unit 10a…Data Acquisition Unit 10b…Model Generation Unit 10c…Estimation Unit 10d…Comparison Display Unit 10e…Calculation Unit 10f…Skills Evaluation Unit 11…Monitor 21…Cab 22…Excavation attachment 22a…Boom 22b…Arm 22c…Bucket 23…Driver's seat 23a…Operating lever C…Difference in number of times D1…Eye line distance E…Eye line position E'…Estimated eye line position ES1…Estimated trajectory data of expert's eye line ES2…Estimated trajectory data of expert's movement operation ET1…Estimated trajectory data of expert's eye line ET2…Worker's eye line trajectory data F…Shooting range H…Expert M…Movement operation N……Communication network P1…Starting position P2…Work start area P3…Excavation position P4…Embankment position PT…Expert operation trajectory data PT1…Expert movement operation trajectory data PT2…Expert specific operation trajectory data R1…Excavation area R2…Embankment area T1…Learning worker operation time T2…Expert operation time TD1…Time difference TD2…Time difference V1…Expert video data W…Working conditions X…Specific operation e...Eye position h...Learning worker pt...Worker operation trajectory data pt1...Worker movement operation trajectory data pt2...Worker specific operation trajectory data t1...Implementation time t2...Reference time v1...Worker video data

Claims

1. A skills teaching system capable of teaching the skills of experienced operators when operating construction machinery to trainee workers, A model generation unit creates a first learning model by performing machine learning on training data, which includes expert operation trajectory data extracted from expert video data obtained by filming an expert operating the operating part of the construction machine to perform a predetermined task, and the working conditions at the time the expert video data was obtained as input data, and expert eye-line trajectory data recorded from the expert's gaze at the time the expert video data was obtained as output data. An estimation unit obtains estimated expert gaze trajectory data by inputting the worker operation trajectory data obtained by extracting the posture and position trajectory of the operating part from worker video data obtained by filming the worker operating the operating part of the construction machine to perform the work, and the work conditions at the time the worker video data was obtained, into the first learning model as estimation input data, and estimating the trajectory of the expert's gaze in the worker video data, A skills teaching system characterized by comprising a comparison display unit capable of overlaying and displaying worker eye-tracking trajectory data, which is worker eye-tracking trajectory data recorded when the worker video data was obtained, and the expert eye-tracking estimated trajectory data, on the worker video data for comparison.

2. In the skill teaching system described in claim 1, A calculation unit that calculates the distance between the line of sight between the worker's line of sight trajectory data and the expert's estimated line of sight trajectory data, A storage unit that stores a first evaluation table in which evaluation points corresponding to the distance between the eyes are predetermined, A skills teaching system characterized by comprising: a skills evaluation unit that calculates an evaluation score based on the first evaluation table from the eye-to-eye distance calculated by the calculation unit.

3. In the skill teaching system described in claim 2, The model generation unit uses the working conditions, the starting position of the operating unit, and the work start area of ​​the operating unit in the expert's video data as input data, and expert's movement trajectory data, which records the trajectory of the attitude and position of the operating unit during the movement operation in which the expert operates the operating unit of the construction machine from the starting position to the work start area, as output data to perform machine learning and create a second learning model. The estimation unit inputs the working conditions, the starting position of the operating unit, and the work start area of ​​the operating unit in the worker video data as estimation input data into the second learning model to obtain estimated expert movement trajectory data, which estimates the trajectory of the attitude and position of the operating unit due to the operation of the expert in the worker video data. The calculation unit calculates the time taken by the learning worker from the starting position to the work start area based on the worker movement operation trajectory data which records the trajectory of the posture and position of the operating part when the learning worker performs the movement operation, and calculates the time taken by the expert from the starting position to the work start area based on the expert movement operation estimated trajectory data, and calculates the time difference between the learning worker's operation time and the expert's operation time. The storage unit stores a second evaluation table in which evaluation points corresponding to the time difference are predetermined. The skills teaching system is characterized in that the skills evaluation unit calculates an evaluation score based on the second evaluation table from the time difference calculated by the calculation unit.

4. In the skill teaching system described in claim 2, The model generation unit creates a third learning model by performing machine learning on training data, which is trained data consisting of trained worker-specific work operation trajectory data, which is a record of the posture and position of the operating part of the construction machine when the trained worker operates the operating part of the construction machine to perform a specific operation on the work area in the trained worker video data, and the number of times the trained worker performed the specific operation in the work content, as output data. The estimation unit inputs worker-specific operation trajectory data, which is recorded in the worker video data, the trajectory of the posture and position of the operating part when the learning worker operates the operating part of the construction machine to perform the specific operation on the work area, into the third learning model as estimation input data to estimate the number of times the learning worker performed the specific operation. The calculation unit calculates the difference between the number of times the skilled worker performed the specific operation and the estimated number of times the trainee worker performed the specific operation. The memory unit stores a third evaluation table in which evaluation points corresponding to the difference in the number of times are predetermined. The skills teaching system is characterized in that the skills evaluation unit calculates an evaluation score based on the third evaluation table from the difference in the number of repetitions calculated by the calculation unit.

5. In the skills teaching system according to any one of claims 1 to 4, The aforementioned work conditions are characterized by being at least one of the type of work, the work location, and the object of work.

Images