Proficiency support system

Abstract

This system quantitatively demonstrates differences in work proficiency and supports workers in improving their skills. [Solution] The proficiency support system for assisting mastery of a predetermined task comprises: an acquisition unit that acquires the actions of the worker to be supported; a recognition unit that recognizes the worker's behavior from the acquired actions according to predetermined categories; an identification unit that compares the supported worker's trajectory, which is the progression of categories of the worker's behavior, with the proficiency worker's trajectory, which is the progression of categories of actions of a person who has mastered the predetermined task; and a presentation unit that presents the results of the comparison.

Description

[Technical Field]

[0001] The present disclosure relates to a proficiency support system that supports the acquisition of technical proficiency. [Background Art]

[0002] Conventionally, there has been known a learning support device that displays digitized work performed by an expert as a model video to allow inexperienced workers to efficiently master work (see, for example, Patent Document 1 below). [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Unexamined Patent Publication No. 2020-144233 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, although such devices may sometimes be useful for supporting skill learning, they merely superimpose the actual movements of a person learning from a model video (hereinafter referred to as a beginner) on the model video, and do not quantitatively handle the differences between a beginner and an expert. For this reason, it has been difficult to grasp the gap between a beginner and an expert, or to provide instruction on what needs to be improved and to what extent for proficiency. [Means for Solving the Problem]

[0005] The present disclosure can be implemented as the following modes or application examples.

[0006] (1) The proficiency support system of this disclosure can be in the following forms. This proficiency support system is a proficiency support system that supports proficiency in a predetermined task, and comprises: an acquisition unit that acquires the actions of a worker who is the target of proficiency support; a recognition unit that recognizes the worker's behavior from the acquired actions according to predetermined classifications; an identification unit that compares the trajectory of the supported worker, which is the change in the classification of the worker's behavior, with the trajectory of a skilled worker, which is the change in the classification of the behavior of a skilled worker who has become proficient in the predetermined task; and a presentation unit that presents the results of the comparison. In this way, by comparing the classified behavior trajectories, it is easy to understand the behavior of the supported worker who has a low level of proficiency, and the behavior of the low level of proficiency can be grasped quantitatively, so that support toward proficiency can be easily realized. Such a proficiency support system may further include a learning unit that generates a learning model using supervised machine learning with a recurrent neural network, and a recognition unit that recognizes the worker's behavior using the trained learning model. In this way, the worker's behavior can be easily recognized according to predetermined categories. Such a learning unit may be included in the proficiency support system or may be provided independently of the proficiency support system. (2) In the above configuration, the recognition unit recognizes the worker's behavior by associating it with a predetermined label, and the identification unit generates the worker's trajectory based on the label. By selecting an appropriate label, the differences between the behavior of a worker being supported and the behavior of a skilled worker can be easily grasped, making it easier to improve the worker's proficiency. (3) In the configurations of (1) and (2) above, the labels associated with the worker's actions may include a noun that identifies the object to which the worker's actions are directed and a verb to which the action belongs, and the recognition unit may recognize the worker's behavior based on the combination of the noun and the verb. This makes it easier to understand the content of the work, such as what to do and how to do it. It also makes it easier to understand which parts are problematic and where the supported worker's proficiency is lacking. Consequently, it becomes easier to improve tasks where proficiency is insufficient. (4) In the configurations of (1) to (3) above, the trajectory of the skilled worker may be generated in advance prior to the comparison in the specific unit by acquiring the skilled worker's movements using the acquisition unit and recognizing them using the recognition unit. In this way, some of the configurations for acquiring the supported worker's trajectory and the skilled worker's trajectory can be shared, and the configuration of the proficiency support system can be simplified. (5) In the configurations described in (1) to (4) above, the comparison results may be obtained as differences for each category. This makes it easy to grasp the differences between the two. [Brief explanation of the drawing]

[0007] [Figure 1] A schematic diagram of the learning support system of the first embodiment. [Figure 2] A flowchart illustrating the worker trajectory generation processing routine. [Figure 3] An explanatory diagram showing how worker trajectories are generated. [Figure 4] A flowchart illustrating the work proficiency support processing routine. [Figure 5] An explanatory diagram showing an example of the trajectories of workers and the results of their comparison. [Modes for carrying out the invention]

[0008] A. First Embodiment: (A1) Device configuration: Figure 1 is a schematic diagram of the learning support system 10 of an embodiment. As shown in the figure, this learning support system 10 comprises a learning support device 20 that performs data processing for learning support and supports the learning of the subject's work, and a learning device 100 that learns work categories and other information and generates a learning model prior to support by the learning support device 20. Of course, the generation of the learning model by the learning device 100 may be performed outside the system, and the learning model may be introduced into the learning support device 20. The learning device 100 is equipped with an RNN (Recurrent Neural Network) and generates a learning model TM. The learning of the learning model TM by the learning device 100 is performed before support is provided by the learning support device 20. The learning of the learning model TM will be explained later.

[0009] The proficiency support system 10 acquires the actions of workers who are not yet proficient in a task and are using the system to learn the task (hereinafter referred to as "supported workers"), and compares them with workers who are already proficient in that task (hereinafter referred to as "proficient workers"). In Figure 1, for ease of understanding, supported workers are shown as code USR-A and skilled workers as code USR-B. In the explanation of the configuration and functions of the proficiency support system 10, when there is no need to distinguish between the two, they are simply referred to as worker USR.

[0010] The proficiency support system 10 comprises a proficiency support device 20 and work parameter sensors 11 and 12 connected thereto, as well as a wearable camera 15 attached to the head of the worker USR. In this embodiment, the work parameter sensors 11 are provided on each of the pair of gloves GVL and GVR worn on both hands by the worker USR. The worker USR performs work on an object such as a bolt or nut held through one glove GVL, using a tool or jig held through the other glove GVR. The work parameter sensors 11 and 12 are used to detect these various actions by the worker USR using pressure, displacement, etc.

[0011] In this embodiment, images captured by the wearable camera 15 are also used to detect various movements. The work parameter sensors 11 and 12 may be installed on only one of the grommets. Alternatively, instead of the work parameter sensors 11 and 12, various movements may be detected by detecting the position and posture of the worker USR's hands. In this case, the posture and finger positions may be recognized from images captured by the wearable camera 15, or sensors for detecting the position and movement of the fingers may be installed on the work clothes worn by the worker USR. Of course, easily recognizable marks or light-emitting objects may be provided on the clothing, and the movements of the worker USR may be recognized by directly recognizing the marks or light-emitting objects.

[0012] The wearable camera 15, attached to the head of the worker USR, basically continuously images the area around the worker USR's hands where they are performing various tasks. A fixed camera mounted on the ceiling or wall of the workplace is also acceptable, as long as it can capture images of the worker USR's hands, i.e., the area around the gloves GVL and GVR. Furthermore, continuous imaging is not necessary; imaging can be initiated at the start of a task being learned, upon instruction from the worker USR, or intermittently, such as every second.

[0013] The signals from these work parameter sensors 11 and 12 are input to the data input unit 21 provided in the proficiency support device 20, and the signals from the wearable camera 15 are input to the imaging input unit 22. In this embodiment, the work parameter sensors 11 and 12 and the wearable camera 15, along with the data input unit 21 and the imaging input unit 22, function as an acquisition unit that acquires the behavior of the worker USR. Since the data acquired by the acquisition unit is obtained from the actual movements of the worker USR, both are collectively called real data LD. When the proficiency support system 10 provides proficiency support to the worker, the real data LD is output from the proficiency support device 20. However, using the same device configuration, real data LD can be obtained from the movements of a worker who has become proficient in the work, and this can be used for machine learning by the learning device 100 described later.

[0014] As shown in the figure, the proficiency support device 20 includes a data input unit 21 that receives signals from work parameter sensors 11 and 12, an imaging input unit 22 that receives video signals from a wearable camera 15, as well as a recognition unit 30, an identification unit 40, a storage unit 45, and a display unit 50. The recognition unit 30 receives the actions of the worker USR acquired based on inputs from the work parameter sensors 11 and 12 and the wearable camera 15, and recognizes the behavior of the worker USR using a learning model TM. The recognition unit 30 performs this recognition using pre-prepared work category data. The data indicating the work category is hereinafter referred to as a label. This label includes a noun that identifies the object to which the worker's actions are directed, and a verb to which the actions belong. The recognition unit 30 recognizes the worker's behavior based on the combination of this noun and verb.

[0015] The identification unit 40 stores the changes in the behavior categories of the worker USR recognized by the recognition unit 30, that is, the time series of behavior categories, as a trajectory for each worker in the storage unit 45. This trajectory for each worker is created and stored for the supported worker USR-A, but the trajectory CSB of the accustomed worker USR-B, which is compared with the trajectory CSA of supported worker USR-A, is also stored in the storage unit 45 for comparison. This accustomed worker trajectory CSB is obtained by acquiring the behavior of an accustomed worker in advance, having the accreditation support device 20 recognize that behavior, and then calculating the trajectory and storing it in the storage unit 45. The trajectory may be generated by a system or device other than the accreditation support system 10 shown in Figure 1, and the result may be saved in the storage unit 45.

[0016] The memory unit 45 stores the supported worker trajectory CSA, which is the progression of behavioral categories of supported worker USR-A recognized using the recognition unit 30, and the skilled worker trajectory CSB, which is prepared in advance for skilled worker USR-B. The identification unit 40 then compares the two. The result of this comparison is displayed on the display unit 50, which is a form of the presentation unit. Here, the comparison result is presented by displaying it on the display unit 50, but it may also be presented by voice output, by printing the comparison result on paper, or by presenting it as the movement of a three-dimensional worker model.

[0017] (A2) Generation of learning model TM: The learning model TM used when the recognition unit 30 recognizes the behavior of the worker USR is generated by machine learning performed by the learning device 100. For convenience of illustration, the learning device 100 is depicted in FIG. 1 together with the proficiency support device 20. The learning device 100 is prepared prior to the process of recognizing the behavior of the supported worker USR-A, and performs machine learning for the learning model TM. Since the data handled by the learning device 100 is time-series data, a recurrent neural network (RNN) is used therein. As shown in FIG. 1, the learning device 100 includes an RNN that performs machine learning, and performs recurrent machine learning using a large amount of actual data LD and teacher data TC related to behavior classifications corresponding to the actual data LD. This learning work is performed before actually comparing the behavior trajectory of the supported worker USR-A with the behavior trajectory of the proficient worker USR-B.

[0018] In the present embodiment, the actual data LD used for learning is time-series data obtained from the above-described work specification sensors 11 and 12, and image data obtained in time series from the wearable camera 15. For a certain behavior performed by the worker USR, the RNN is trained to classify a large number of behaviors by inputting the actual data LD and providing the teacher data TC as a combination of a noun indicating an action target and a verb indicating an action, for example, "bolt (noun)" + "tighten (verb)". The term "behavior" is used because there may be cases where there is no target of an action, such as movement of the worker USR or repetition (naming) of work content, so the term is used to include both actions with and without a work target. When there is no such work target, the behavior is handled in a form such as "Null (noun indicating empty)" + "move (verb)", or "Null (noun)" + "name (verb)".

[0019] When acquiring actual data LD for machine learning, among the illustrated configuration, it is only necessary to have the gloves GVL, GVR and wearable camera 15 equipped with work specification sensors 11, 12, a data input unit 21, an imaging input unit 22, and a memory MEM that stores the actual data LD. As the skilled worker USR-B repeats a specific action, actual data LD is collected, teacher data TC indicating the behavior when each actual data LD is collected, that is, "action target (noun) + action (verb)", is prepared in advance, and this teacher data TC is associated with each actual data LD and stored in the memory MEM in advance.

[0020] Such association between the collection of actual data LD and teacher data TC for the worker's behavior may be performed by one skilled worker USR-B, or may be performed by a plurality of skilled workers. In addition, the gloves GVL, GVR, work specification sensors 11, 12, wearable camera 15 and the like used when collecting actual data LD may be the same as those used by the supported worker USR-A when performing work for proficiency, but different gloves, pressure sensors and wearable cameras may be used as long as their characteristics are the same or similar.

[0021] (A3) Generation of trajectory of supported worker: Next, a process of generating a trajectory during work of the supported worker USR-A who needs support for proficiency will be described with reference to FIG. 2. FIG. 2 is a flowchart showing a process performed by the proficiency support device 20. This process is executed when the supported worker USR-A starts work, first acquires an image from the wearable camera 15 (step S101), and then acquires data related to the action of the worker USR from the work specification sensors 11 and 12 (step S111). That is, the actual data LD of the worker is input in time series.

[0022] Next, using this real data LD, the system references the learned model TM and performs a process to recognize the categories of worker USR's behavior (step S121). If worker USR is performing an action such as tightening a bolt R, the system determines that the entire sequence of actions, from beginning to end, constitutes a single category. This determination can be easily achieved using the learned model TM, which has been pre-trained using an RNN.

[0023] Next, the learning model TM determines whether a behavioral pattern has ended and a new category has been established (step S131). If it is not determined that a new category has been established (step S131: "NO"), the process returns to step S101 and is repeated. On the other hand, if it is determined that the worker USR's behavior has become a new category (step S131: "YES"), the previous category recognized in step 121 is added to the end of the trajectory list (step S141).

[0024] In other words, the actions of the worker USR are analyzed time-series using actual data LD, for example, • Bolt R + tightening • Connector E+ snap-in • Jig X + rotation As shown above, the tasks performed by the worker USR are divided into units of object (noun) + action (verb), and these are listed in chronological order. This list is called the "worker trajectory," or simply the "trajectory list." The output stage of the recurrent neural network RNN ​​is equipped with outputs of this object and action, and by learning the results in accordance with the training data TC, the learning (generalization) of the divisions is completed.

[0025] The worker USR's behavior is categorized according to the actual data LD, and the behavior recognized up to that point is added to the end of the trajectory list. Then, it is determined whether all of the worker USR's work has been completed (step S151). If the work is still ongoing (step S151: "NO"), the process is repeated from step S101. The process up to this point (steps S101 to S151) is called the behavior recognition process (step S100). By repeating steps S101 to S151, the worker's behavior can be recognized as an object (noun) + action (verb), as described above.

[0026] In step S151, if it is determined that all operations have been completed (step S151: "YES"), the trajectory list is closed, the worker trajectory generation process is considered complete, the worker trajectory (trajectory list) is stored in the storage unit 45, and this processing routine is terminated.

[0027] The above process is schematically shown in Figure 3. The actual data LD is sequentially input to the proficiency support device 20 over time, and the proficiency support device 20 sequentially recognizes the behavior of the worker USR and adds it to the trajectory list CS. This process does not change whether the worker is a supported worker USR-A who is not proficient in the work or a proficient worker USR-B who is proficient in the work. Therefore, it is easy to create a proficient worker trajectory CSB in advance for the proficient worker USR-B.

[0028] (A4) Processes to support skill acquisition: Next, we will explain the process of providing proficiency support to the supported worker USR-A. Figure 4 is a flowchart of the proficiency support processing routine for the supported worker USR-A. When this processing routine is started, first, the process of step S100 of the worker trajectory generation processing routine shown using Figure 2, that is, the process of recognizing the worker's behavior, is performed for the supported worker USR-A. As a result, the behavior of the supported worker USR-A is recognized, and then a worker trajectory is generated by arranging the recognized behavior categories in chronological order (step S162). This process corresponds to step S161 in Figure 2, but the difference is that in Figure 2, the worker was not identified, whereas here the worker is identified as the supported worker USR-A who is receiving proficiency support.

[0029] After saving these worker trajectories to the storage unit 45, the system compares the trajectory CSA of worker USR-A with the trajectory CSB of experienced worker USR-B (step S171), and then presents the comparison results (step S181). This completes the work proficiency support process.

[0030] Figure 5 shows the process of comparing trajectories (step S171) and presenting the comparison results (step S181) in the work proficiency support process. In the figure, trajectory CSA is a time-series arrangement of the behavior of worker A. Worker A is the supported worker USR-A. On the other hand, trajectory CSB is a time-series arrangement of the behavior of worker B, i.e., the proficiency worker USR-B. Figure 5 compares the trajectories of these two in chronological order, and the comparison results are shown at the far right. These trajectories and comparison results are displayed on the display unit 50.

[0031] In this example, from time t0 to time t1, both worker A and worker B perform the same action, namely tightening bolt R. Therefore, the comparison result (difference) column is left blank or displays a corresponding evaluation such as "identical". Similarly, from time t1 to t2, both workers A and B perform the action of inserting connector E, and there is no display of difference in the comparison result column.

[0032] However, between times t2 and t3, worker A rotates jig X, while worker B rotates jig Y, showing a difference in their behavior. Therefore, the difference is displayed in the comparison results column as "Incorrect object to work on". For the tasks indicated by hatching in the diagram, it is judged that there is some difference between worker A's work and worker B's work. Here, it is pointed out that worker A's behavior indicates that they are working on the wrong object.

[0033] Furthermore, worker A tightens bolt P from time t3 to t5, but this differs from worker B's work in the following respects. Worker B also tightens bolt P from time t3 to t4, but from time t4 to t5, he performs the next task, namely attaching part G. Therefore, the comparison result for time t4 to t5 is displayed as "Worker A's work is delayed."

[0034] The delay in work caused by worker A between times t4 and t5 is not resolved by time tn, the end of the series of tasks, as shown in the diagram. Therefore, the same warning is displayed in the comparison results column between t5 and tn. Normally, it is difficult for someone receiving work proficiency support to resolve such delays, but if worker A had sped up at some point and caught up with worker B's work, the warning about the delay would be resolved.

[0035] As described above, the proficiency support system 10 of this embodiment can quantitatively show a proficient worker USR-B, who is proficient in a task, exactly where the work of a supported worker USR-A, who is not yet proficient, is lacking, based on specific details of the work, such as differences in behavior or delays. Therefore, by using the proficiency support system 10, supported worker USR-A can easily work on improving their own work. Rather than simply seeing that the work is slow, they can specifically understand what kind of actions are problematic in relation to the work object at each stage, making it easier to correct incorrect actions or delays. Furthermore, it is also easy to provide guidance to supported worker USR-A to increase their proficiency in the work.

[0036] First, the system recognizes the object of the work and the work performed on that object, and displays this as a trajectory. This makes it easier to understand, for example, if the wrong object is selected, whether it was simply a mistake or if there is a high frequency of misidentification of bolts, suggesting a problem with bolt identification. Similarly, it allows users to identify characteristics of behaviors that tend to cause work delays and work towards improvement.

[0037] In each of the above embodiments, some of the configurations implemented by hardware may be replaced with software. At least some of the configurations implemented by software can also be implemented by discrete circuit configurations. Furthermore, if some or all of the functions of this disclosure are implemented by software, that software (computer program) can be provided in the form of being stored on a computer-readable recording medium. "Computer-readable recording medium" is not limited to portable recording media such as flexible disks and CD-ROMs, but also includes various internal storage devices within a computer such as RAM and ROM, and external storage devices fixed to a computer such as hard disks. In other words, "computer-readable recording medium" has a broad meaning that includes any recording medium on which data packets can be fixed rather than temporary.

[0038] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features of the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. For example, some of the configurations implemented by hardware in the above embodiments can be implemented by software. [Explanation of symbols]

[0039] 10…Learning support system, 11,12…Work parameter sensor, 15…Wearable camera, 20…Learning support device, 21…Data input unit, 22…Image input unit, 30…Recognition unit, 40…Identification unit, 45…Storage unit, 50…Display unit, 100…Learning device, MEM…Memory, RNN…Recurrent neural network

Claims

1. A learning support system that assists in mastering a prescribed task, An acquisition unit that acquires the actions of the worker who is the target of training support, A recognition unit recognizes the worker's behavior from the acquired actions according to predetermined categories, A specific unit compares the trajectory of the supported worker, which is the change in the categories of behavior of the worker, with the trajectory of the skilled worker, which is the change in the categories of behavior of a skilled worker who has become proficient in the predetermined task. A display unit that presents the results of the comparison, A learning support system equipped with the following features.

2. The recognition unit recognizes the behavior by associating a predetermined label with the worker's actions. The specified unit generates the trajectory of the supported worker based on the label. The proficiency support system according to claim 1.

3. The label associated with the worker's action includes a noun that identifies the object to which the worker's action is directed, and a verb to which the action belongs. The learning support system according to claim 2, wherein the recognition unit recognizes the worker's behavior based on the combination of the noun and the verb.

Images