Work classification device, work classification method, and program

Abstract

This work classification device comprises: a storage unit that stores image data in which a work area has been imaged; and an arithmetic circuit that classifies the behavior of a worker on the basis of the image data. The classification includes a first class, a second class different from the first class, a first subclass included in the first class, and a second subclass different from the first subclass in the first class. The arithmetic circuit switches the granularity of the first class when assigning the behavior of the worker to the classification, on the basis of information in the work area in an image indicated by the image data.

Description

Work classification device, work classification method, and program

[0001] The present disclosure relates to a task classification device, a task classification method, and a program that use a machine learning model and / or an image recognition algorithm.

[0002] Patent Literature 1 discloses a device for analyzing work performed by a worker on an object. The work analysis device in Patent Literature 1 identifies the positions of the worker's hands and the object, calculates the distance between the worker's hands and the object, and identifies the details of the action performed by the worker based on the calculated distance.

[0003] Patent No. 7010542

[0004] The present disclosure provides a task classification device, a task classification method, and a program that can effectively classify the actions of workers according to the capture conditions of image data.

[0005] A task classification device according to one aspect of the present disclosure includes a memory unit that stores image data of an image of a work area; and an arithmetic circuit that classifies worker actions based on the image data, wherein the classification includes a first class, a second class different from the first class, a first subclass included in the first class, and a second subclass of the first class different from the first subclass, and the arithmetic circuit switches the granularity of the first class when assigning the worker actions to the classification based on information within the work area in the image shown by the image data.

[0006] A task classification method according to one aspect of the present disclosure includes the steps of: an arithmetic circuit acquiring image data of an image of a task area; and the arithmetic circuit classifying the worker's actions based on the image data, wherein the classification includes a first class, a second class different from the first class, a first subclass included in the first class, and a second subclass of the first class different from the first subclass; and the task classification method further includes the step of switching the granularity of the first class when assigning the worker's actions to the classification based on information within the task area in the image shown by the image data.

[0007] A program according to one aspect of the present disclosure is a program for causing a calculation circuit to execute the above task classification method.

[0008] According to the present disclosure, worker actions can be effectively classified according to the capture conditions of image data.

[0009] Schematic diagram showing an overview of a task classification system according to a first embodiment of the present disclosure. Schematic diagram illustrating an image represented by image data generated by the camera of FIG. 1. Block diagram showing an example configuration of the task classification device of FIG. 1. Graph showing an example of a classification result of activity content by the task classification device according to the first embodiment. Flowchart illustrating the operation of the task classification device of FIG. 1. Flowchart illustrating details of the task classification process shown in FIG. 5. Table showing an example of the classification result DB of FIG. 3. Schematic diagram showing an example of a display image showing the classification result by the task classification process shown in FIG. 5. Schematic diagram illustrating an image represented by image data generated by a camera in a second embodiment. Flowchart illustrating the task classification process according to the second embodiment. Table showing an example of a modified classification result DB.

[0010] Hereinafter, embodiments will be described with reference to the drawings as appropriate. However, more detailed explanations than necessary may be omitted. For example, detailed explanations of well-known matters or redundant explanations of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art.

[0011] The inventors provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims.

[0012] 1. First Embodiment 1-1. Overview FIG. 1 is a schematic diagram illustrating an overview of a task classification system 1 according to a first embodiment of the present disclosure.

[0013] The task classification system 1 includes a camera 2 and a task classification device 10. The task classification system 1 is applied to classifying the content of the actions of workers performing tasks such as line work in a workplace 6 such as a factory. The task classification system 1 also includes a display device 4 for presenting the classification results of the worker's task content to a user 3 such as a manager or analyst of the workplace 6.

[0014] The camera 2 is positioned so as to capture a worker performing work in the workplace 6. The camera 2 captures images of the workplace 6, for example, at a predetermined interval, and generates image data representing the captured images. Note that while one camera 2 is illustrated in FIG. 1 , the number of cameras 2 included in the task classification system 1 is not limited to one and may be two or more. The camera 2 may, for example, capture a video of the workplace 6, and generate video data representing the captured video.

[0015] FIG. 2 is a schematic diagram illustrating an image 20 represented by image data generated by the camera 2. The image 20 shows a workplace 6. Eight workers 21 to 28 are working in the workplace 6. Three work areas 31 to 33 are shown in FIG. 2. For example, the work areas 31 to 33 are predetermined as predetermined regions within the image 20. The positions, sizes, etc. of the work areas 31 to 33 can be arbitrarily set by, for example, the user 3. The actual workplace 6 may have areas corresponding to the work areas 31 to 33 on the image.

[0016] The area of ​​responsibility of workers 21 and 22 is work area 31. The area of ​​responsibility is determined in advance as the area where the worker performs work. At least one area of ​​responsibility is determined for each worker. The area of ​​responsibility of workers 23 and 24 is work area 32. The area of ​​responsibility of workers 25 to 28 is work area 33. Unlike the example shown in FIG. 2 where each work area includes multiple workers, there may be a one-to-one correspondence between work areas and workers.

[0017] 3 is a block diagram showing an example configuration of the task classification device 10. The task classification device 10 includes a control unit 11, a storage unit 12, an input interface (I / F) 13, and an output interface 14.

[0018] The control unit 11 includes, for example, a processor and / or an arithmetic circuit that cooperates with software to realize predetermined functions, and controls the overall operation of the task classification device 10. The control unit 11 reads data and programs stored in the memory unit 12, performs various arithmetic processes, and realizes various functions. For example, the control unit 11 operates as a detection unit 111 and a determination unit 112.

[0019] The control unit 11 may be a hardware circuit such as a dedicated electronic circuit or a reconfigurable electronic circuit designed to realize a predetermined function. The control unit 11 may include various semiconductor integrated circuits such as a CPU, MPU, GPU, GPGPU, TPU, microcomputer, DSP, FPGA, ASIC, etc. instead of or as an arithmetic circuit. The task classification method of this embodiment may also be executed by distributed computing.

[0020] The control unit 11 includes an operation detection model 113 that detects an object and an operation of a worker through image recognition processing.

[0021] The task detection model 113 is a trained model that has undergone training using a neural network such as a convolutional neural network. The task detection model 113 performs image recognition processing on the image represented by the image data. The task detection model 113 outputs, as a detection result, an area in the image in which a predetermined object, such as a worker's hand, is captured. In this embodiment, the detection object of the task detection model 113 is set to the worker's hand. The area output as a detection result is defined, for example, by horizontal and vertical positions on the image, and indicates a rectangular area surrounding the detection object.

[0022] In this embodiment, when the area to be detected cannot be recognized in the image (i.e., the worker's hand cannot be detected), the activity detection model 113 outputs, for example, a null value as the detection result. The detection result may include, for example, information indicating the time the image was captured. The activity detection model 113 is obtained by performing supervised learning using training data that associates, for example, images showing the worker's hand with correct answer labels indicating the correct answer.

[0023] The trained model of the activity detection model 113 is not limited to a neural network, but may be another machine learning model related to image recognition. An image recognition algorithm may be adopted as the activity detection model 113, rather than a model generated by machine learning. For example, the activity detection model 113 may be configured to perform activity detection using rule-based image recognition processing.

[0024] The storage unit 12 is a recording medium that records various information, including programs and data necessary to realize the functions of the task classification device 10. The storage unit 12 is realized, for example, by a semiconductor storage unit such as a flash memory or a solid-state drive (SSD), a magnetic storage unit such as a hard disk drive (HDD), or other recording media, either alone or in combination. The storage unit 12 is not limited to a built-in storage unit installed in the same housing as the control unit 11, but may also be, for example, an external storage unit, a NAS (network-attached storage) storage unit, or the like. The storage unit 12 may also include a volatile memory, such as a RAM.

[0025] The storage unit 12 stores image data 121 received from the camera 2, a classification result database (DB) 122 containing the classification results obtained by the task classification device 10, and worker information 123. The worker information 123 is, for example, a database that links identification information for identifying multiple workers with the task area (area of ​​responsibility) in which each worker is scheduled to perform their task.

[0026] The input interface 13 is an example of an input unit that connects the task classification device 10 and the camera 2 in order to input information such as image data from the camera 2 to the task classification device 10. The input interface 13 may be a communication circuit that performs data communication in accordance with an existing wired communication standard or wireless communication standard.

[0027] The output interface 14 is an example of an output unit that connects the task classification device 10 to an external device such as the display device 4 in order to output information such as control signals, video signals, and task classification results from the control unit 11 to the external device. The output interface 14 may be a communication circuit that performs data communication in accordance with an existing wired communication standard or wireless communication standard. The output interface 14 may have a configuration similar to that of the input interface 13.

[0028] The input interface 13 and the output interface 14 may be realized as separate interfaces as shown in Fig. 3, but are not limited to this. For example, the input interface 13 and the output interface 14 may be configured as an integrated interface.

[0029] 1-3. Operation 1-3-1. Overview In the image 20 shown in FIG. 2, workers 21 and 22 in work area 31 and workers 23 and 24 in work area 32 are photographed in positions and orientations that make it easy for their hands to be seen. In this way, the relationship between the camera 2 and the position and orientation of the worker determines whether the area in which the worker is located is an area where their hands are easy to be seen. In this context, work areas 31 and 32 are areas where the workers' hands are easy to be seen. In contrast, work area 33 is an area where it is difficult for the workers' hands to be seen.

[0030] A typical task breakdown technique (hereinafter referred to as a "typical technique") can identify the details of a worker's actions with a relatively high degree of accuracy in a task area where the worker's hands are easily visible. Alternatively, a typical technique can identify the details of a worker's actions for a relatively long period of time during a video recording for a worker in a task area where the worker's hands are easily visible. However, a typical technique cannot identify the details of a worker's actions unless it can identify the position of the worker's hands. Therefore, a typical technique cannot adequately identify the details of a worker's actions in an area where the worker's hands are difficult to see. Therefore, a typical technique cannot evaluate the type of work a worker is performing, or evaluate the efficiency of the work, particularly in an area where the worker's hands are difficult to see.

[0031] To make it possible to carry out the above-mentioned evaluation, it is conceivable to position workers or cameras so that the hands of all workers are easily visible, but this is not easily realized due to constraints such as the structure of the workplace, the conditions for positioning workers, the conditions for where cameras can be installed, and the number of cameras that can be prepared.

[0032] Therefore, as a result of intensive research, the inventors came up with the idea of ​​changing the granularity of classification of the worker's activity content depending on the capture conditions of the image data, and have arrived at the present disclosure. Here, the granularity of classification refers to the degree of detail to which the objects to be classified are divided. The granularity of classification may be the hierarchical depth of the assigned classes or categories with different levels of abstraction.

[0033] For example, if an object such as a worker's hand is captured in the image shown by the image data, the task classification device 10 according to this embodiment classifies the details of the worker's actions in detail based on the detection results of the object.

[0034] In contrast, when an image does not show an object, the inventors came up with the idea of ​​classifying the worker's activity content at the highest possible granularity rather than giving up on classifying the worker's activity content. Therefore, the task classification device 10 according to the present embodiment lowers the classification granularity when an image does not show an object compared to when an image shows an object, and makes it possible to classify the worker's activity content to that extent. As a result, the task classification device 10 according to the present embodiment can obtain more classification information from the same amount of data than a technology that does not classify the worker's activity content when an image does not show an object. Therefore, the task classification device 10 according to the present embodiment can reduce the amount of data required to obtain the same amount of classification information, thereby reducing memory, computational processing, and communication traffic associated with data exchange.

[0035] Fig. 4 is a graph showing an example of the classification result of activity content by the task classification device 10 according to the present embodiment. The bar graph in Fig. 4 is a graph that visualizes the classification result of activity content by the task classification device 10. The task classification device 10 determines whether a worker is present (present) or absent (absent) in the assigned area.

[0036] In this specification, the behavior of a worker includes not only the actions that the worker performs in the area of ​​responsibility, but also the worker's presence in the area of ​​responsibility and the worker's absence (absence) in the area of ​​responsibility.

[0037] When the task classification device 10 detects a worker's hand, which is an example of a target object, in an image, it detects whether the task being performed by the worker in the image is a value-added task or a non-value-added task. A value-added task is a task that has been predetermined as a target for classification processing by the task classification device 10. A value-added task is an example of a target task in the present disclosure.

[0038] Non-value-added work refers to the actions of a worker other than value-added work. Worker actions include both actions and omissions. For example, when a worker is performing non-value-added work, it includes cases where the worker is actively performing an action other than value-added work, cases where the worker is stopped, etc.

[0039] By performing the above-described processing, the task classification device 10 can generate task classification results such as the bar graph on the left side of Figure 4 when an object such as a worker's hand is captured in the image.

[0040] On the other hand, even when an object is not shown in the image, the task classification device 10 can generate a task classification result like the bar graph on the right side of Figure 4. In this way, when an object is not shown in the image, the task classification device 10 lowers the granularity of the classification compared to when the object is shown in the image. When an object is not shown in the image, the task classification device 10 does not not classify the content of the worker's activity, but rather lowers the granularity, but makes it possible to classify the content of the worker's activity to that extent. Here, subclasses with lower granularity of the class "present" correspond to value-added tasks or non-value-added tasks. "Present" is a higher category than value-added tasks or non-value-added tasks, and is a superordinate concept.

[0041] In images taken over a predetermined period, for example, on a specific day (e.g., 24 hours), the task classification results for a specific worker for one day will usually be a mixture of classification results for cases where hands are visible, as in the bar graph on the left side of Figure 4, and classification results for cases where hands are not visible, as in the bar graph on the right side of Figure 4.

[0042] According to the task classification device 10 of this embodiment, even if the worker's assigned area is an area where the worker's hands are not easily visible, it is possible to grasp at least the time the worker spent in the assigned area. If there is a period in which the worker's hands are visible, the task classification device 10 can further grasp the value-added task time and non-value-added task time during that period.

[0043] By analyzing such task classification results or by viewing the task classification results displayed on the display device 4, the user 3 can know the content (behavior) of the tasks of the workers at the highest possible level of granularity depending on the imaging conditions. Based on such knowledge, the user 3 can evaluate the task content of each worker. By feeding back such evaluations to the workers themselves, the labor system, etc., the efficiency of the work performed in the workplace 6 can be improved.

[0044] Furthermore, based on the above-mentioned knowledge, user 3 can improve the efficiency of work performed in workplace 6 by reviewing the system of the placement of objects, people, etc. in workplace 6.

[0045] 5 is a flowchart illustrating the operation of the task classification device 10. Each process shown in this flow is executed by the control unit 11 of the task classification device 10, for example.

[0046] The control unit 11 acquires the worker information 123 (S1). The control unit 11 may acquire the worker information 123 from an external source via the input interface 13, or may acquire the worker information 123 stored in advance in the storage unit 12.

[0047] The control unit 11 acquires image data from the camera 2 via the input interface 13 (S2). The control unit 11 stores the acquired image data 121 in the storage unit 12. Unlike the example of Fig. 5, step S2 may be executed before step S1.

[0048] The control unit 11 selects one worker to be detected (S3). For example, the control unit 11 selects one worker to be detected from among multiple workers in the worker information 123. The control unit 11 may select a worker who is expected to be in the area covered by the detection target, such as a factory, at the time the image is acquired. In this case, the worker information 123 may include information indicating the time when each worker is expected to be in the area covered by the detection target.

[0049] Next, the control unit 11 assigns identification information for identifying the worker selected in step S3 to the worker (S4). As the identification information, identification information associated with each worker in the worker information 123 may be used.

[0050] The control unit 11 executes a task classification process (S5). Details of the task classification process S5 will be described later.

[0051] Next, the control unit 11 determines whether there are other workers to be detected (S6). If there are other workers to be detected (Yes in S6), the control unit 11 executes steps S3 to S5 for one of the other workers. In this case, in step S3, one of the other workers is selected as the worker to be detected.

[0052] If there are no other workers to be detected (No in S6), the control unit 11 calculates the working time of each worker based on the results of the task classification process S5 (S7).

[0053] Next, the control unit 11 causes at least one of the results of the task classification process S5 and the calculation results of step S7 to be displayed on the display device 4 via the output interface 14 (S8).

[0054] The user 3 can learn the details of the worker's actions by looking at the task classification results displayed on the display device 4. For example, the user 3 can evaluate the tasks of each worker based on this knowledge. By feeding back such evaluations to the workers themselves, the labor system, etc., the efficiency of the work performed in the workplace 6 can be improved.

[0055] 1-3-3 Task Classification Processing Fig. 6 is a flowchart illustrating the details of the task classification processing S5 shown in Fig. 5.

[0056] The control unit 11 detects whether the worker selected in step S3 of Fig. 5 is in the area covered by the worker in the image (S11). The area covered by the worker is, for example, predetermined as a predetermined area in the image. The worker is detected, for example, by using an existing technology for detecting people in an image.

[0057] When the control unit 11 detects that the worker is in the area of ​​responsibility (Yes in S11), it determines that the worker is present in the area (presence determination) (S12).

[0058] If the control unit 11 does not detect that the worker is in the area of ​​responsibility (No in S11), it determines that the worker is absent (absence determination) (S13). In this example, the absence of the worker means that the worker is not in the area of ​​responsibility.

[0059] After step S12, the control unit 11 determines whether or not the worker's hand has been detected (S14). For example, the control unit 11 determines whether or not the hand has been detected within the worker's assigned area in the image.

[0060] In the present embodiment, the worker's hand refers to the part of the worker beyond the wrist (distal). If the worker is wearing gloves, the worker's hand in the present embodiment includes the gloves. In other words, if the worker is wearing gloves, the control unit 11 may determine that the worker's hand has been detected when the control unit 11 detects the worker's gloves.

[0061] When the control unit 11 detects the worker's hands (Yes in S14), it detects whether the work being performed by the worker in the image is a value-added work (S15).

[0062] At least one of the processes of steps S14 and S15 is executed by, for example, the detection unit 111 of the control unit 11. The detection unit 111 executes at least one of the processes of steps S14 and S15 using, for example, the task detection model 113.

[0063] When the control unit 11 detects that the work being performed by the worker is value-added work (Yes in S15), it determines that the worker is performing value-added work (S16).

[0064] When the control unit 11 does not detect that the work being performed by the worker is value-added work (No in S15), it determines that the worker is performing non-value-added work (S17).

[0065] If the worker's hands are not detected in step S14 (No in S14), the control unit 11 determines that the worker is performing an unspecified task (S18).

[0066] In this specification, a worker being engaged in unspecified work means that the worker is at least present within the assigned area. A worker being engaged in unspecified work includes a case where the worker is performing value-added work and a case where the worker is performing non-value-added work. A case where the control unit 11 determines that the worker is performing unspecified work means a case where the control unit 11 cannot detect the worker's hands (No in S14) and therefore cannot determine whether the worker is performing value-added work.

[0067] The above steps S12, S13, and S16 to S18 are executed by the determination unit 112 of the control unit 11, for example.

[0068] The presence determination in step S12 is an example of classifying the worker's behavior into a first class, and the absence determination in step S13 is an example of classifying the worker's behavior into a second class.

[0069] The value-added task determination in step S16 is an example of classifying the worker's behavior into a first subclass. The non-value-added task determination in step S17 is an example of classifying the worker's behavior into a second subclass. The first and second subclasses are each a further subclass of the first class. In one example, the first subclass is value-added tasks, and the second subclass is non-value-added tasks.

[0070] After the above steps S13 and S16 to S18, the control unit 11 records the determination result and the time in the classification result DB 122 (S19).

[0071] Fig. 7 is a table showing an example of the classification result DB 122. In the classification result DB 122 of Fig. 7, the identification information of the worker and the action content that is the result of the task classification process S5 are linked to time information. The time indicated by the time information corresponds to the time when the image that is the target of the task classification process S5 was captured.

[0072] The example of the classification result DB 122 in Fig. 7 shows that worker A was engaged in unspecified work at 10:00:01 and 10:00:02 on a specific day, and was absent at 10:00:03. The example of the classification result DB 122 in Fig. 7 shows that worker B was engaged in value-added work from 10:00:01 to 10:00:03 on the same day.

[0073] The control unit 11 can tally up the worker's unspecified task time, value-added task time, non-value-added task time, and / or time the worker was absent (absent time) by recording the worker's task status at a predetermined cycle as shown in Fig. 7. Because the worker is present within his / her assigned area during unspecified task, value-added task, or non-value-added task, in this specification, the unspecified task time, value-added task time, and non-value-added task time may be collectively referred to as "presence time."

[0074] The presence time, absence time, value-added task time, and non-value-added task time are examples of the first to fourth time periods of the present disclosure. The unspecified task time is an example of the fifth time period of the present disclosure.

[0075] An example of displaying the classification results of the worker's presence time, absence time, etc. thus collected will be described with reference to FIG.

[0076] 8 is a schematic diagram showing an example of a display image 40 showing the classification results obtained by the task classification process S5. The display image 40 in FIG. 8 is displayed on the display device 4 in step S8 in FIG.

[0077] The display image 40 shows a bar graph for visualizing the classification results of the task classification process S5. The bar graph in Fig. 8 shows the total values ​​of unspecified task time, absence time, value-added task time, and non-value-added task time for each worker on a specific day. These total values ​​can be calculated using the classification result DB 122 shown in Fig. 7.

[0078] In the example of the bar graph shown in Figure 8, presence time, i.e., value-added work time, unspecified work time, and non-value-added work time, is stacked upward with the reference axis 41 as the base. On the other hand, absence time is shown as a bar graph extending downward with the reference axis 41 as the base. This makes it easy for the user 3 to compare absence time for each worker, and to compare absence time and presence time for each worker.

[0079] The bar graph shown in Fig. 8 shows an example in which the areas covered by workers A and B are areas where hands are likely to be visible, such as areas 31 and 32 in Fig. 2. Therefore, in the bar graphs for workers A and B, the ratio of unspecified task time to value-added task time and non-value-added task time is low.

[0080] On the other hand, the bar graph shown in Fig. 8 shows an example in which the areas covered by workers C and D are areas where their hands are difficult to see, such as area 33 in Fig. 2. Therefore, in the bar graphs for workers C and D, the ratio of unspecified task time to value-added task time and non-value-added task time is higher than in the bar graphs for workers A and B.

[0081] Furthermore, the bar graph shown in FIG. 8 indicates that the absence times of workers A and B are the same, but the value-added work time of worker A is longer than that of worker B.

[0082] The bar graph shown in Figure 8 shows that worker C's absence time is longer than that of the others, and that worker D's absence time is shorter than that of the others. It can also be seen that the ratio of worker C's absence time to the total presence time (unspecified task time, value-added task time, and non-value-added task time) is also larger than that of the others. It can also be seen that the ratio of worker C's absence time to the total presence time of worker D is smaller than that of the others.

[0083] By viewing the display image 40 displayed on the display device 4, the user 3 can analyze the content of the work of the workers as described above. The user 3 can evaluate the work content of each worker based on the results of such analysis. The user 3 can improve the efficiency of the work performed in the workplace 6 by feeding back such evaluations to the workers themselves, the labor system, etc. Furthermore, the user 3 can improve the efficiency of the work performed in the workplace 6 by reviewing the system, such as the placement of objects and people in the workplace 6, based on the results of the analysis described above.

[0084] 1-5. Effects, etc. As described above, the task classification device 10 according to this embodiment includes a memory unit 12 and a control unit 11, which is an example of an arithmetic circuit. The memory unit 12 stores image data of an image of a task area, which is an example of a task region. The control unit 11 classifies the worker's actions based on the image data. The classification includes a first class, a second class different from the first class, a first subclass included in the first class, and a second subclass of the first class different from the first subclass. The control unit 11 switches the granularity of the first class when assigning the worker's actions to the classification, based on information within the task area in the image 20 represented by the image data. For example, the control unit 11 switches the classification of the worker's actions between the first and second classes and the first and second subclasses (S5).

[0085] With this configuration, the task classification device 10 can effectively classify the content of the worker's actions according to the capture conditions of the image data. The task classification device 10 can also achieve at least the same effect in the following aspects.

[0086] The control unit 11 may determine whether or not the worker is in the work area in the image 20 (S11). If the control unit 11 determines that the worker is in the work area in the image 20 (Yes in S11), it classifies the behavior into a first class (S12). If the control unit 11 determines that the worker is not in the work area in the image 20 (No in S11), it classifies the behavior into a second class (S13).

[0087] When the control unit 11 determines that the worker is in the work area in the image 20 (Yes in S11), it may detect the worker's hand, which is an example of a predetermined object related to the work performed by the worker, in the image 20 (S14). When the control unit 11 detects the worker's hand (Yes in S14), it determines whether the work is a value-added work, which is an example of a predetermined target work (S15). When the control unit 11 determines that the work is a value-added work (Yes in S15), it classifies the action into a first subclass (S16). When the control unit 11 determines that the work is not a value-added work (No in S15), it classifies the action into a second subclass (S17).

[0088] According to this configuration, when the worker's hands are detected, the worker's actions can be classified in more detail.

[0089] If the worker's hand is not detected (No in S14), the control unit 11 may classify the action into a class different from any of the second class, the first subclass, and the second subclass. With this configuration, even if the worker's hand is not detected, the action content of the worker can be classified with as high a granularity as possible.

[0090] The control unit 11 may measure a first time period during which the worker performs an action classified into a first class and a second time period during which the worker performs an action classified into a second class in the image 20 (S7). By measuring the time periods corresponding to each class, for example, the user 3 can quantitatively grasp the content of the worker's action.

[0091] The control unit 11 may measure, in the image 20, a third time during which the worker performed an action classified into the first subclass, and a fourth time during which the worker performed an action classified into the second subclass.

[0092] The task classification device 10 may further include an output interface 14, which is an example of an output unit that outputs information to the display device 4. The control unit 11 may cause the display device 4 to display data indicating the action classification results via the output interface 14.

[0093] The control unit 11 may cause the display device 4 to display information indicating the first time and the second time via the output interface 14. The control unit 11 may cause the display device 4 to display information indicating the first to fourth times via the output interface 14.

[0094] By viewing the data displayed on the display device 4, the user 3 can analyze the content of the work performed by the workers. Based on the results of such analysis, the user 3 can evaluate the work content of each worker. By providing feedback of such evaluation to the worker himself, the labor system, etc., the user 3 can improve the efficiency of the work performed in the workplace 6. Furthermore, based on the results of the above analysis, the user 3 can improve the efficiency of the work performed in the workplace 6 by reviewing the system, such as the placement of objects and people in the workplace 6.

[0095] 2. Second Embodiment In the first embodiment, an example in which the target object is the worker's hand has been described. In the second embodiment, an example in which the target object is a lamp will be described.

[0096] 9 is a schematic diagram illustrating an image 20a represented by image data generated by the camera 2 in the second embodiment. The image 20a shows a worker 21a working in a workplace.

[0097] Three boxes 51 to 53 are placed in the work area. Boxes 51 to 53 contain parts X, Y, and Z, respectively. In this embodiment, worker 21a takes out parts X, Y, and Z from boxes 51 to 53 and carries the taken-out parts to a predetermined location for shipping.

[0098] 9 shows three work areas 34 to 36. For example, the work areas 34 to 36 are predetermined regions within the image 20a. In the example of FIG. 9, the work areas 34 to 36 are regions corresponding to boxes 51 to 53, respectively. For example, the positional relationship between the work area 34 and the box 51 is configured such that when the worker 21a stands in front of the box 51, the worker 21a is within the work area 34 in the image 20a. The same applies to the positional relationship between the work area 35 and the box 52, and the positional relationship between the work area 36 and the box 53.

[0099] Lamps 54 to 56 are disposed in front of boxes 51 to 53 (between boxes 51 to 53 and camera 2), respectively. Lamps 54 to 56 are configured to light up before worker 21a begins work. When worker 21a takes component X from box 51, lamp 54 corresponding to box 51 goes out. Similarly, when worker 21a takes component Y from box 52, lamp 55 goes out, and when worker 21a takes component X from box 53, lamp 56 goes out.

[0100] In this embodiment, when the worker 21a is in front of a box whose corresponding lamp is lit, the control unit 11 determines that the worker 21a is performing a value-added task. The value-added task assumed in this embodiment is the worker 21a taking out a part from a box whose corresponding lamp is lit.

[0101] In this embodiment, when the worker 21a is standing in front of a box whose corresponding lamp is off, the control unit 11 determines that the worker 21a is performing a non-value-added task. The non-value-added task assumed in this embodiment is a task other than part removal, such as sorting.

[0102] In this embodiment, the control unit 11 determines that the worker 21a is performing an unspecified task when the corresponding lamp is not shown in the image 20a. The control unit 11 determines that the worker 21a is absent when the worker 21a is not in front of the box.

[0103] 10 is a flowchart illustrating the task classification process S5a according to the present embodiment. In the present embodiment, the control unit 11 executes the task classification process S5a in FIG. 10 instead of the task classification process S5 in the first embodiment.

[0104] Compared with the task classification process S5 of the first embodiment shown in FIG. 6, the task classification process S5a of FIG. 10 includes step S24 instead of step S14, and step S25 instead of step S15.

[0105] 10, the control unit 11 first detects whether the worker 21a is in the work area in the image (S11). In this embodiment, the work areas 34 to 36 are the areas in front of the boxes 51 to 53 (between the boxes 51 to 53 and the camera 2).

[0106] When the control unit 11 detects that the worker 21a is in the work area (Yes in S11), it determines that the worker is present in the work area (presence determination) (S12).When the control unit 11 does not detect that the worker is in the work area (No in S11), it determines that the worker is absent (absence determination) (S13).

[0107] After step S12, the control unit 11 determines whether or not a lamp has been detected (S24). In the example of Fig. 9, the control unit 11 determines whether or not any of the lamps 54 to 56 has been detected.

[0108] If the control unit 11 detects a lamp (Yes in S24), it detects whether the lamp is on or not (S25). In this embodiment, detecting whether the lamp is on or not is an example of detecting whether the work being performed by the worker in the image is value-added work or not.

[0109] If the control unit 11 detects that the lamp is on (Yes in S25), it determines that the worker is performing value-added work (S16). If the control unit 11 does not detect that the lamp is on (No in S25), it determines that the worker is performing non-value-added work (S17).

[0110] If the control unit 11 does not detect the lamp in step S24 (No in S24), the control unit 11 determines that the worker is performing an unspecified task (S18).

[0111] According to this embodiment, the task classification device 10 can also effectively classify the content of the worker's actions according to the imaging conditions of the image data.

[0112] 3. Other Embodiments As described above, the embodiments have been described as examples of the technology in the present disclosure. However, the technology in the present disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. Therefore, other embodiments will be described below as examples.

[0113] In the first embodiment, an example has been described in which the target object is a worker's hand and the target task includes hand movement. However, the present disclosure is not limited to this example, and the target object may be the worker's foot. In this case, the target task may include foot movement. The target object may be any part of the worker's body, or may be a tool or the like used by the worker for the task.

[0114] 7 in the first embodiment, an example of the classification result DB 122 including only unspecified task time, value-added task time, and non-value-added task time as the content of worker actions has been described, but the classification result DB is not limited to this. Fig. 11 is a table illustrating a modified example 122a of the classification result DB.

[0115] 11, the content of a worker's action is classified as a main category into "present" (first class) or "absent" (second class). When the main category is "present" (first class), the content of the worker's action is further classified as a secondary category (subclass) into "value-added work" (first subclass), "non-value-added work" (second subclass), or "unspecified work."

[0116] In the first embodiment, an example has been described in which the control unit 11 executes step S3 of selecting one worker to be detected from the image represented by the image data 121. However, the present disclosure is not limited to this. For example, a specific work area may be associated with a specific worker in advance. In this case, step S3 may be omitted. For example, once a work area is identified, the control unit 11 can identify the worker associated with the work area. In this case, even if the worker associated with the work area is not shown in the work area in the image represented by the image data 121, the worker associated with the work area can be identified.

[0117] When a specific work area and a specific worker are associated in advance as described above, the control unit 11 can identify the worker associated with the work area once the work area is specified. Therefore, in this case, step S4 in FIG. 5 may be omitted.

[0118] 4. Example Aspects The following describes examples of the present disclosure.

[0119] <Aspect 1> A task classification device comprising: a storage unit that stores image data of an image of a work area; and an arithmetic circuit that classifies worker actions based on the image data, wherein the classification includes a first class, a second class different from the first class, a first subclass included in the first class, and a second subclass of the first class different from the first subclass, and the arithmetic circuit switches the granularity of the first classes when assigning the worker actions to the classifications based on information within the work area in an image represented by the image data.

[0120] <Aspect 2> The task classification device according to Aspect 1, wherein the arithmetic circuit determines whether the worker is in the task area in the image, classifies the action into a first class if it is determined that the worker is in the task area in the image, and classifies the action into a second class if it is determined that the worker is not in the task area in the image.

[0121] <Aspect 3> The task classification device according to Aspect 2, wherein the arithmetic circuit: when it is determined in the image that the worker is within the task area, detects in the image a predetermined object related to the task to be performed by the worker; when the object is detected, determines whether the task is a predetermined target task; when it is determined that the task is the target task, classifies the action into the first subclass; and when it is determined that the task is not the target task, classifies the action into the second subclass.

[0122] <Aspect 4> The task classification device according to aspect 3, wherein the arithmetic circuit classifies the activity into a class different from any of the second class, the first subclass, and the second subclass when the target object is not detected.

[0123] <Aspect 5> The task classification device according to any one of Aspects 1 to 4, wherein the arithmetic circuit measures a first time period during which the worker performs an action classified into the first class in the image, and a second time period during which the worker performs an action classified into the second class in the image.

[0124] Aspect 6: The task classification device according to Aspect 5, wherein the arithmetic circuit measures a third time during which the worker performs the action classified into the first subclass in the image, and a fourth time during which the worker performs the action classified into the second subclass in the image.

[0125] <Aspect 7> The task classification device according to any one of Aspects 1 to 6, further comprising an output unit that outputs information to a display device, wherein the arithmetic circuit causes the display device to display data indicating a classification result of the action via the output unit.

[0126] <Aspect 8> The task classification device according to aspect 5 or 6, further comprising an output unit that outputs information to a display device, wherein the arithmetic circuit causes the display device to display information indicating the first time period and the second time period via the output unit.

[0127] <Aspect 9> The task classification device according to aspect 6, further comprising an output unit that outputs information to a display device, wherein the arithmetic circuit causes the display device to display information indicating the first to fourth times via the output unit.

[0128] <Aspect 10> The task classification device according to aspect 3 or 4, wherein the target object is the worker's hand, and the target task includes a movement of the hand.

[0129] <Aspect 11> The task classification device according to aspect 3 or 4, wherein the target object is a foot of the worker, and the target task includes a movement of the foot.

[0130] <Aspect 12> The task classification device according to Aspect 3 or 4, wherein the target object is a lamp installed in the work area, and the arithmetic circuit detects whether the lamp is lit based on the image data, and if it detects that the lamp is lit, determines that the task is the target task.

[0131] <Aspect 13> A task classification method for classifying the actions of a worker, comprising: a step in which an arithmetic circuit acquires image data of an image of a work area; and a step in which the arithmetic circuit classifies the actions of the worker based on the image data, wherein the classification includes a first class, a second class different from the first class, a first subclass included in the first class, and a second subclass of the first class different from the first subclass, and the task classification method further comprises a step in which the arithmetic circuit switches the granularity of the first class when assigning the actions of the worker to the classification based on information within the work area in an image represented by the image data.

[0132] <Aspect 14> A program for causing a calculation circuit to execute the task classification method according to aspect 13.

[0133] The present disclosure is applicable to a device for analyzing the work of a worker in a workplace such as a factory.

[0134] REFERENCE SIGNS LIST 1 Task classification system 2 Camera 3 User 4 Display device 6 Workplace 10 Task classification device 11 Control unit 12 Memory unit 13 Input interface 14 Output interface 20 Image 21-28 Worker 31-36 Work area 40 Display image 41 Reference axis 51-53 Box 54-56 Lamp 111 Detection unit 112 Determination unit 113 Task detection model 121 Image data 122 Classification result database 123 Worker information

Claims

1. A task classification device comprising: a memory unit that stores image data of an image of a work area; and an arithmetic circuit that classifies worker actions based on the image data, wherein the classification includes a first class, a second class different from the first class, a first subclass included in the first class, and a second subclass of the first class different from the first subclass, and the arithmetic circuit switches the granularity of the first class when assigning the worker actions to the classification based on information within the work area in an image represented by the image data.

2. The task classification device of claim 1, wherein the arithmetic circuit determines whether the worker is within the work area in the image, classifies the action into a first class if it is determined that the worker is within the work area in the image, and classifies the action into a second class if it is determined that the worker is not within the work area in the image.

3. The task classification device described in claim 2, wherein the arithmetic circuit: when it is determined that the worker is within the work area in the image, detects a specified object in the image related to the task to be performed by the worker; when the object is detected, determines whether the task is a specified target task; when it is determined that the task is the target task, classifies the action into the first subclass; and when it is determined that the task is not the target task, classifies the action into the second subclass.

4. The task classification device according to claim 3, wherein the calculation circuit classifies the action into a class different from any of the second class, the first subclass, and the second subclass when the object is not detected.

5. The task classification device according to claim 1, wherein the calculation circuit measures a first time when the worker performs an action classified into the first class in the image, and a second time when the worker performs an action classified into the second class in the image.

6. The task classification device according to claim 5, wherein the arithmetic circuit measures a third time during which the worker performs an action classified into the first subclass in the image, and a fourth time during which the worker performs an action classified into the second subclass in the image.

7. The task classification device according to claim 1, further comprising an output unit that outputs information to a display device, wherein the arithmetic circuit causes the display device to display data indicating the classification result of the action via the output unit.

8. The task classification device as described in claim 5, further comprising an output unit that outputs information to a display device, and the arithmetic circuit causes the display device to display information indicating the first time and the second time via the output unit.

9. The task classification device according to claim 6, further comprising an output unit that outputs information to a display device, wherein the arithmetic circuit causes the display device to display information indicating the first to fourth times via the output unit.

10. The task classification device according to claim 3, wherein the target object is the worker's hand, and the target task includes a movement of the hand.

11. The task classification device according to claim 3, wherein the target object is the worker's foot, and the target task includes a movement of the foot.

12. The task classification device described in claim 3, wherein the target object is a lamp installed in the work area, and the arithmetic circuit detects whether the lamp is on or not based on the image data, and if it detects that the lamp is on, determines that the task is the target task.

13. A task classification method for classifying a worker's actions, comprising: a step in which an arithmetic circuit acquires image data of an image of a work area; and a step in which the arithmetic circuit classifies the worker's actions based on the image data, wherein the classification includes a first class, a second class different from the first class, a first subclass included in the first class, and a second subclass of the first class different from the first subclass, and the task classification method further comprises a step in which the arithmetic circuit switches the granularity of the first class when assigning the worker's actions to the classification based on information within the work area in the image represented by the image data.

14. A program for causing a calculation circuit to execute the task classification method according to claim 13.

Images