Work recognition device and work recognition method

Abstract

We provide work recognition technology that can recognize tasks with high precision. [Solution] The work recognition device for recognizing the work of an operator comprises: an acquisition unit that acquires observation information obtained by observing the target work with a sensor; and a work identification unit that identifies the type of target work using a first model that has been pre-trained to know the order in which multiple tasks are performed, the first model's prediction result which predicts the type of target work as the next task based on the type of the previous task, and a second model that has been pre-trained to estimate the type of target work based on the observation information.

Description

Technical Field

[0001] The present disclosure relates to a work recognition device and a work recognition method.

Background Art

[0002] Regarding a work recognition device that recognizes the work of an operator, Patent Document 1 discloses a technique for estimating the type of operation appearing in a video or a skeleton sequence by inputting the video or the skeleton sequence into an action recognition model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, for example, when performing work recognition at a place where a huge variety of work is carried out, such as a manufacturing site, if only information obtained by observing work with sensors, such as a video or a skeleton sequence, is used as the basis for work recognition, there may be cases where the work cannot be recognized accurately.

Means for Solving the Problems

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to one embodiment of the present disclosure, a work recognition device is provided that recognizes the work of an operator. The work recognition device comprises: an acquisition unit that acquires observation information obtained by observing a target work with a sensor; and a work identification unit that identifies the type of the target work using a first model which has been pre-trained to know the order in which a plurality of work is performed, and which predicts the type of the target work as the next work based on the type of the previous work, and an estimation result from a second model which has been pre-trained to estimate the type of the target work based on the observation information. In this configuration, by using the first and second models, it is possible to recognize tasks with high accuracy by considering not only the observation information from sensors but also the order in which the tasks are performed. (2) In the above configuration, the prediction result includes predicted type information representing a plurality of work types predicted by the first model as the type of the target work, and predicted probability information representing the probability that each of the predicted plurality of work types matches the type of the target work, and the estimation result includes estimated type information representing a plurality of work types estimated by the second model as the type of the target work, and estimated probability information representing the probability that each of the estimated plurality of work types matches the type of the target work, and the work identification unit may determine a plurality of candidate target work using one of the two information sets, the predicted type information and the predicted probability information, and the estimated type information and the estimated probability information, and then identify the type of the target work from among the plurality of candidates using the other information set of the two information sets. According to this configuration, the order in which the work is performed and the actual observation results by the sensor can be considered in a balanced manner to recognize the work with higher accuracy. (3) In the above configuration, the task identification unit may use the predicted type information and the predicted probability information to determine the plurality of candidates, and from among the plurality of candidates, use the estimated type information and the estimated probability information to identify the type of task with the highest probability represented in the estimated probability information as the type of target task. According to this configuration, after determining the candidates considering the order in which the tasks are performed, the system can ultimately place more emphasis on the actual observation results from the sensor and identify the type of task that is estimated to have a high probability of being the type of target task from the perspective of the actual observation results as the type of target task. (4) In the above configuration, the first model is configured to predict the type of the next task from the types of two or more consecutive previous tasks, and the task identification unit may input information representing the types of two or more consecutive tasks preceding the target task into the first model, thereby causing the first model to predict the type of the target task. According to this configuration, the type of the target task can be predicted by considering not only the task immediately preceding the target task but also the task even earlier, thereby improving the accuracy of the prediction by the first model. As a result, tasks can be recognized with higher accuracy. This disclosure can be implemented in forms other than the work recognition device described above, such as a work recognition system, a work recognition method, a program for implementing the work recognition method, a non-temporary recording medium on which the program is recorded, or a program product. The program product may be provided, for example, as a recording medium on which the program is recorded, or as a program product that can be distributed via a network. [Brief explanation of the drawing]

[0007] [Figure 1] This is an explanatory diagram showing the schematic configuration of the work recognition system. [Figure 2] This is a diagram illustrating operations in a factory. [Figure 3] This is a conceptual diagram illustrating the process of recognizing tasks. [Figure 4] This is a diagram illustrating the process of identifying a task. [Figure 5] This is a flowchart showing the processing steps for task recognition. [Modes for carrying out the invention]

[0008] A. First Embodiment: Figure 1 is an explanatory diagram showing the schematic configuration of the work recognition system 10 in the first embodiment. The work recognition system 10 is used to recognize work performed by worker WK. The work to be recognized by the work recognition system 10 is also called the "target work". To "recognize the target work" means, more specifically, to recognize the type of target work.

[0009] The work recognition system 10 is used in the work area where worker WK performs work. In this embodiment, the work area is a factory FC for manufacturing vehicles. In this embodiment, the work is a variety of tasks for manufacturing vehicles, including, for example, tasks related to vehicle assembly, tasks related to attaching parts to vehicles, and tasks related to vehicle inspection.

[0010] Figure 2 is a diagram illustrating the operations in a factory fuel chain (FC). Figure 2 shows process information Pi, which represents each work process in the factory FC. Process information Pi is included, for example, in the Bill of Process (BOP) of the factory FC. Figure 2 shows work processes WP1, WP2, WP3, and WP4 as examples of work processes. Work processes WP1, WP2, WP3, and WP4 are performed in this chronological order.

[0011] As shown in Figure 2, in this embodiment, each work process is represented by a combination of a "target part" and a "unit action." The "target part" refers to the part that is the object of work in the work process. The "unit action" refers to an action that does not limit the object. It can also be said that the unit action represents how the target part is handled. Furthermore, the combination of the "target part" and the "unit action" can be said to correspond to the combination of an object and a verb. In this embodiment, there are 10 or more types of unit actions, more specifically, 15 or more types.

[0012] In this embodiment, the work recognition system 10 recognizes the unit operation shown in Figure 2 as the type of target work. Therefore, for example, even if the target parts are different in work process WP1 and work process WP3, the work recognition system 10 recognizes the type of target work as "extraction" regardless of whether the target work corresponds to work process WP1 or work process WP3. In other embodiments, the type of target work recognized by the work recognition system 10 is not limited to unit operations and may be arbitrary. For example, the type of work may be an operation in a higher category than a unit operation, i.e., an operation in a larger unit than a unit operation, or an operation in a lower category than a unit operation, i.e., an operation in a smaller unit than a unit operation. The number of such operation categories may also be arbitrary. Furthermore, the type of target work may correspond to the work process shown in Figure 2, i.e., it may represent a combination of "target part" and "unit operation". Furthermore, the size of the target part category and the number of target part categories may also be arbitrary.

[0013] Furthermore, in this embodiment, the work recognition system 10 is used to recognize the work being performed by worker WK in real time. That is, the "target work" in this embodiment corresponds to the current work. In other embodiments, the work recognition system 10 may be used to recognize the target work retrospectively.

[0014] The work recognition system 10 comprises a sensor group 60 including one or more sensors 50, and a work recognition device 100.

[0015] Sensor 50 observes the work performed by worker WK. "Observing work with sensor 50" means observing at least one of the following regarding the work being observed: the worker WK performing the work, the object being worked on, the equipment EQ used for the work, the tools TL used for the work, and the work environment in which the work is performed. The information obtained by observing the target work with sensor 50 is also called observation information. Sensor 50 transmits the observation information from sensor 50, i.e., the detection result from sensor 50, to the work recognition device 100. The observation information is associated with information indicating the timing at which the observation information was detected.

[0016] Sensor 50 includes various sensors such as a camera 51, a microphone 52, an inertial measurement unit (IMU) 53, a bending sensor 54, a vibration sensor 55, a vital sensor 56, an area sensor, and a pressure sensor. Sensor 50 is installed, for example, in various locations in the factory FC, on equipment EQ used for work, on tools TL used for work, and on equipment worn by workers WK. Equipment includes, for example, goggles, work clothes WW, and gloves WG. Sensors 50 may be installed in various locations in the factory FC, for example, a camera 51 and an area sensor. Goggles worn by workers WK may be equipped with, for example, a camera 51 as a first-person camera and a microphone 52 for detecting sounds around the worker WK. Work clothes WW may be equipped with, for example, an IMU 53 for detecting acceleration and angular velocity occurring in the worker WK. Furthermore, the glove WG may be equipped with, for example, an IMU 53 for detecting acceleration and angular velocity in the worker WK's arm, a bending sensor 54 for detecting bending of the worker WK's fingers and wrist, a vibration sensor 55 for detecting vibrations associated with work, a pressure sensor for detecting pressure on the fingers associated with work, a sound detection sensor for detecting sounds associated with work, and a vital sensor 56 for detecting vital signs such as the worker WK's heart rate, blood pressure, and body temperature. Note that the types and combinations of sensors 50 are not limited to those described above.

[0017] The operation recognition device 100 is configured by a computer including a processor 101, a memory 102 including a ROM and a RAM, an input / output interface 103, and an internal bus 104. The processor 101, the memory 102, and the input / output interface 103 are connected via the internal bus 104 so as to be communicable bidirectionally. A communication device 105 and a display device 106 are connected to the input / output interface 103. The communication device 105 can communicate directly or indirectly with each sensor 50 by wired communication or wireless communication. The display device 106 is configured by, for example, a liquid crystal display or the like, and displays various information such as information regarding the operation recognition result by the operation recognition system 10. Various information such as a program PG1, a first model 210, a second model 220, and history data HD are stored in the memory 102. The processor 101 realizes various functions including functions as an acquisition unit 110 and an operation identification unit 120 by executing the program PG1.

[0018] FIG. 3 is a conceptual diagram for explaining the flow of operation recognition in the present embodiment. As shown in FIG. 3, the acquisition unit 110 acquires observation information OB from each sensor 50. Further, the acquisition unit 110 acquires pre-operation information PW representing the type of the operation before the target operation. In the present embodiment, as shown in FIG. 1, the acquisition unit 110 acquires the pre-operation information PW by referring to the history data HD stored in the memory 102. Details of the history data HD will be described later.

[0019] Returning to the explanation of FIG. 3, the pre-operation information PW includes at least information representing the type of the operation immediately before the target operation. In the present embodiment, the pre-operation information PW includes information representing the type of each of two or more consecutive operations before the target operation. Hereinafter, the number of types of operations included in the pre-operation information PW is also referred to as the number N. For example, when the number N is 2, the pre-operation information PW includes information representing the type of the operation immediately before the target operation and information representing the type of the operation one before the immediately before operation. In the present embodiment, the value of the number N is 2 or more. When there is no immediately before operation or no operation one before the immediately before operation, the pre-operation information PW includes information representing that such operations do not exist.

[0020] The operation specifying unit 120 executes an operation specifying process. The operation specifying process is a process of specifying the type of the target operation using the first model 210 and the second model 220.

[0021] The first model 210 is a machine learning model that has learned order information in advance. The order information is information regarding the order in which a plurality of operations are performed at the work location, that is, the work order of the plurality of operations. As the order information, for example, the above process information Pi or BOP can be used. Multiple types of order information may be used for the learning of the first model 210. In this case, in each order information, the specifications and types of the vehicles to be manufactured may be different. Also, for each order information, the factory FC where the order information is used may be different. By doing so, the generalization performance of the first model 210 can be further improved.

[0022] Further, the first model 210 is configured as a context-based prediction model that predicts the type of the next operation contextually based on the work order. More specifically, the first model 210 is configured to output a prediction result PR regarding the type of the next operation with information representing the type of the previous operation as an input. Here, the "previous operation" means the operation before the "next operation" and at least includes the operation immediately before the next operation. Also, the number of operations included in the "previous operation" is the same as the number N. That is, in the present embodiment, the first model 210 is configured to be able to predict the type of the next operation from the types of two or more consecutive previous operations. Also, in the present embodiment, the type of operation predicted by the first model 210 corresponds to the above "unit operation". As shown in FIG. 3, in the operation specifying process, the operation specifying unit 120 inputs the previous operation information PW to the first model 210 to cause the first model 210 to output a prediction result PR regarding the type of the target operation. At this time, the target operation corresponds to the "next operation".

[0023] In this embodiment, a machine learning model utilizing a neural network is used as the first model 210. The neural network includes convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In this embodiment, the first model 210 is trained by supervised learning using sequential information. In the supervised learning of the first model 210, each operation represented by sequential information is used as the "previous operation" and the "next operation". In the supervised learning of the first model 210, the "previous operation" corresponds to the explanatory variable, and the "next operation" corresponds to the target variable, i.e., the label. Such supervised learning can be easily performed, for example, by scanning the sequential information in a time series using a sliding window. In this case, the window width of the sliding window is set based on the number N.

[0024] In other embodiments, various machine learning models such as random forests or support vector machines (SVMs) may be used as the first model 210. Furthermore, in other embodiments, the learning method for the first model 210 is not limited to supervised learning. For example, the first model 210 may be trained by unsupervised learning or reinforcement learning.

[0025] In this embodiment, the first model 210 is configured to output, as the prediction result PR, first type information representing a predetermined set of work types, and information representing the probability of agreement for each work type represented in the first type information. The probability of agreement for a given work represents the probability that the work type and the target work type match. The probability of agreement may be zero or a probability equivalent to 100%. As a result of the first model 210 being configured as described above, in this embodiment, the prediction result PR includes prediction type information and prediction probability information. The prediction type information represents a set of prediction types. The prediction type represents the work type predicted by the first model 210 as the target work type. In this embodiment, "type predicted as the target work type" means a type among the types represented in the first type information that has a probability of agreement greater than zero. In other embodiments, "type predicted as the target work type" may be, for example, a type among the types represented in the first type information that has a probability of agreement greater than or equal to a predetermined threshold greater than zero. The prediction probability information represents the prediction probability for each prediction type. The prediction probability represents the probability that the prediction type matches the type of target task. This first model 210 is configured, for example, as a logistic regression model having multiple output layer units. The number of output layer units of the first model 210 is set, for example, based on the number of target task types to be recognized.

[0026] The second model 220 is a machine learning model that has been pre-trained to estimate the type of target work from the observation information OB. More specifically, the second model 220 is configured to take the observation information OB as input and output an estimated result ER regarding the type of target work. As shown in Figure 3, in the work identification process, the work identification unit 120 inputs the observation information OB to the second model 220, causing the second model 220 to output the estimated result ER.

[0027] In this embodiment, a machine learning model utilizing a neural network is used as the second model 220, similar to the first model 210. In this embodiment, the second model 220 is trained by supervised learning using sequential information. In the supervised learning of the second model 220, observational information obtained by observing the target work with the sensor 50 is used as explanatory variables, and the type of the target work is used as the target variable, i.e., the label. In other embodiments, various other machine learning models may be used as the second model 220, similar to the first model 210. Furthermore, the learning method for the second model 220 is not limited to supervised learning.

[0028] In this embodiment, the second model 220 is configured to output, as the estimation result ER, second type information representing a predetermined set of work types, and information representing the probability of agreement for each work type represented in the second type information. As a result of the second model 220 being configured in this way, in this embodiment, the estimation result ER includes the estimation type information and the estimation probability information. The estimation type information represents a set of estimated types. The estimated type represents the work type estimated by the second model 220 as the type of target work. In this embodiment, "the type estimated as the type of target work" means the type among the types represented in the second type information that has a probability of agreement greater than zero. In other embodiments, "the type estimated as the type of target work" may be, for example, a type among the types represented in the second information that has a probability of agreement greater than or equal to a predetermined threshold greater than zero. The estimation probability information represents the estimated probability for each estimated type. The estimated probability represents the probability that the estimated type matches the type of target work.

[0029] Figure 4 is a diagram illustrating the work identification process in this embodiment. In the work identification process in this embodiment, the work identification unit 120 uses one of two information sets, one consisting of predicted type information and predicted probability information, and the other consisting of estimated type information and estimated probability information, to determine candidate CDs for multiple target work. From among the candidate CDs, the work identification unit 120 uses the other information set and does not use the other information set to identify the type of target work. More specifically, in the work identification process in this embodiment, the work identification unit 120 uses the predicted type information and predicted probability information included in the prediction result PR from the first model 210 to determine candidate CDs for multiple target work types. The work identification unit 120 then identifies the type of work with the highest estimated probability among the identified candidate CDs as the type of target work. That is, in this embodiment, the predicted type information and predicted probability information correspond to "one set" in this disclosure, and the estimated type information and estimated probability information correspond to "the other set" in this disclosure. Furthermore, in this embodiment, it can be said that the estimated result ER is used only to narrow down the type of target work from candidate CD, without the prediction result PR being used again.

[0030] In the example in Figure 4, the task identification unit 120 uses the predicted type information and predicted probability information included in the prediction result PR to determine "check," "tighten," "register," and "attach" as candidate CDs. When determining candidate CDs, for example, a predetermined number of types may be determined as candidate CDs in order of highest predicted probability, or types whose predicted probability is above a predetermined probability threshold may be determined as candidate CDs, or the number and probability threshold may be used in combination to determine candidate CDs. Next, the task identification unit 120 uses the estimated type information and estimated probability information included in the estimation result ER by the second model 220 to identify "check," which has the highest estimated probability among the candidate CDs, as the type of target task.

[0031] Let's return to the explanation in Figure 3. The task identification unit 120 records the type of target task identified by the task identification process as the task recognition result RR in the memory 102. The recognition result RR, by being recorded in the memory 102 in this way, is used as history data HD, which records the history of task recognition by the task recognition system 10. The task identification unit 120 also outputs the recognition result RR. More specifically, the task identification unit 120 displays the recognition result RR on the display device 106 or transmits the recognition result RR to an external device via the communication device 105.

[0032] Figure 5 is a flowchart showing the processing procedure for work recognition to implement the work recognition method in this embodiment. The work recognition process is executed by the processor 101 of the work recognition device 100 when predetermined execution conditions are met. The execution conditions may be, for example, conditions relating to the observation results of the sensor 50. Conditions relating to observation results may be, for example, conditions in which the sensor values ​​from one or more specific sensors 50 have changed to a predetermined standard level or higher. Alternatively, the execution conditions may be, for example, conditions relating to elapsed time. Conditions relating to elapsed time may be, for example, conditions in which a predetermined time has elapsed since the completion timing of the previous work recognition process.

[0033] In step S100 of Figure 3, the acquisition unit 110 acquires the previous work information PW. In step S110, the acquisition unit 110 acquires the observation information OB from each sensor 50.

[0034] Steps S120 to S150 correspond to the task identification process. In step S120, the task identification unit 120 inputs the previous task information PW acquired in step S100 into the first model 210, causing the first model 210 to output the prediction result PR. In step S130, the task identification unit 120 determines candidate CD according to the prediction type information and prediction probability information included in the prediction result PR output in step S120.

[0035] In step S140, the work identification unit 120 inputs the observation information OB acquired in step S110 into the second model 220, causing the second model 220 to output the estimation result ER.

[0036] In step S150, the task identification unit 120 identifies the type of target task from among the candidate CDs according to the estimated type information and estimated probability information included in the estimated result ER output in step S140. Also in step S150, the task identification unit 120 records the identified type of target task as the recognition result RR in the memory 102. In step S160, the task identification unit 120 outputs the recognition result RR.

[0037] In step S150, if the estimated probability of all types included in candidate CD is zero, the work identification unit 120 may, for example, terminate the work recognition process without identifying the type of target work, or it may, assuming that the previous work is continuing, identify the type identified in the previous step as the type of target work. In this case, the work identification unit 120 may also notify the user of the work recognition system 10 of the error, for example, through a notification device such as a display device 106 or a speaker.

[0038] As described above, the work recognition system 10 in this embodiment identifies the type of target work by using a first model 210 that estimates the type of target work as the type of the next work based on the type of the previous work, and a second model 220 that has been trained to estimate the type of target work based on observation information OB from the sensor 50. Therefore, work can be recognized with high accuracy by considering not only the observation information OB but also the work order. More specifically, compared to a configuration in which work recognition is performed using only the second model 220, for example, the work recognition system 10 can suppress situations in which multiple types of work, including similar operations, are confused, and can recognize work with high accuracy.

[0039] Furthermore, according to this embodiment, compared to a configuration in which work recognition is performed using only the second model 220, for example, it is possible to maintain a relatively high level of accuracy in work recognition even when the types and number of sensors 50 used are reduced. As a result, for example, costs can be reduced. In addition, by reducing the types and amount of observation information OB that is processed, the processing load in work recognition can be reduced and the processing speed can be improved.

[0040] Furthermore, in this embodiment, candidate CDs for multiple target tasks are determined using one of two information sets: predicted type information and predicted probability information, and estimated type information and estimated probability information. From the determined candidate CDs, the type of target task is identified using the other information set, but without using the other set. This prevents excessive disregard for the work order and excessive disregard for the actual observation results from the sensor 50, compared to, for example, a configuration in which the type of task with a higher sum of predicted probability and estimated probability is identified as the type of target task. More specifically, it prevents the identification of a type of task that cannot be predicted from the perspective of work order, or a type of task that cannot be estimated from the perspective of observation information OB, as the type of target task. Thus, according to this embodiment, the work order and the actual observation results from the sensor 50 are considered in a balanced manner, enabling more accurate task recognition.

[0041] Furthermore, in this embodiment, among the multiple candidate CDs determined using the prediction result PR, the type of work with the highest estimated probability is identified as the target work type. In this way, after determining candidate CDs considering the work order, the actual observation results from the sensor 50 are given more weight in the final identification of the work type that is estimated to have a high probability of being the target work type from the perspective of the actual observation results.

[0042] Furthermore, in this embodiment, the first model 210 is configured to estimate the type of the next task from the types of two or more consecutive previous tasks, and the task identification unit 120 inputs previous task information PW, which represents the types of two or more consecutive tasks immediately preceding the target task, to the first model 210, thereby causing the first model 210 to estimate the type of the target task. In this way, the first model 210 can predict the type of the target task by considering not only the task immediately preceding the target task but also the tasks even earlier, thereby improving the accuracy of the prediction by the first model 210. As a result, tasks can be recognized with higher accuracy.

[0043] B. Other embodiments: (B1) In the above embodiment, the work identification unit 120 identifies the type of work with the higher estimated probability among the candidate CD as the type of target work, but is not limited to this. For example, the work identification unit 120 may identify the type of work with the higher sum of the predicted probability and the estimated probability as the type of target work, or it may identify the type of work with the higher value obtained by adding the predicted probability and the estimated probability with different weights as the type of target work. Alternatively, the work identification unit 120 may, for example, in the opposite case to the above embodiment, determine a plurality of candidate types of target work using the estimated result ER, and then identify the type of target work from among the determined candidates using the predicted result PR. In this case, the work identification unit 120 can, for example, identify the type of work with the highest predicted probability from among the candidates determined using the estimated result ER as the type of target work. That is, in this case, the estimated type information and estimated probability information included in the estimated result ER correspond to "one set" in this disclosure, and the predicted type information and predicted probability information included in the predicted result PR correspond to "the other set" in this disclosure. Furthermore, the work identification unit 120 may, for example, identify the type of target work from among the candidates determined using the prediction result PR and the estimation result ER, by using the sum of the predicted probability and the estimated probability, or by adding the predicted probability and the estimated probability with different weights.

[0044] (B2) In the above embodiment, the work identification unit 120 inputs information representing the types of two or more consecutive tasks preceding the target work to the first model 210, thereby causing the first model 210 to predict the type of the target work. In contrast, the work identification unit 120 may, for example, input only information representing the type of the task immediately preceding the target work to the first model 210, thereby causing the first model 210 to predict the type of the target work. That is, the previous work information PW may include only information representing the type of the task immediately preceding the target work. In this case, the first model 210 may also be configured to predict the type of the next work by taking only the type of the task immediately preceding the next work as input.

[0045] (B3) In the above embodiment, factory FC is a vehicle manufacturing plant, but is not limited to this. For example, factory FC may be a vehicle inspection plant, or a manufacturing plant or inspection plant for various items other than vehicles, or any other type of factory. Furthermore, the work recognition system 10 may be used not only in factory FC, but also in various work locations where worker WK performs work.

[0046] (B4) In the above embodiment, some or all of the functional units such as the acquisition unit 110 and the work identification unit 120 may be provided in an external device such as an external computer that is different from the work recognition device 100.

[0047] In each of the above embodiments, some or all of the functions and processes implemented in software may be implemented in hardware. Conversely, some or all of the functions and processes implemented in hardware may be implemented in software. As hardware for implementing the various functions in each of the above embodiments, various circuits such as integrated circuits and discrete circuits may be used.

[0048] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of Symbols]

[0049] 10...Work recognition system, 50...Sensor, 51...Camera, 52...Microphone, 53...Inertial measurement device, 54...Bending sensor, 55...Vibration sensor, 56...Vital sensor, 60...Sensor group, 100...Work recognition device, 101...Processor, 102...Memory, 103...Input / output interface, 104...Internal bus, 105...Communication device, 106...Display device, 110...Acquisition unit, 120...Work identification unit, 210...First model, 220...Second model

Claims

1. A work recognition device that recognizes the work of an operator, An acquisition unit that obtains observation information by observing the target work using sensors, A work recognition device comprising: a first model that has been pre-trained to know the order in which multiple tasks are performed, and a work identification unit that identifies the type of the target task using the prediction result of the first model which predicts the type of the target task as the next task based on the type of the previous task, and the estimation result of a second model which has been pre-trained to estimate the type of the target task based on the observation information.

2. A work recognition device according to claim 1, The prediction result includes prediction type information representing a plurality of work types predicted by the first model as the type of the target work, and prediction probability information representing the probability that each of the plurality of predicted work types matches the type of the target work. The estimation result includes estimated type information representing a plurality of work types estimated as the type of the target work by the second model, and estimated probability information representing the probability that each of the plurality of estimated work types matches the type of the target work. The aforementioned work identification unit is, Using one of the two sets of information, namely the prediction type information and the prediction probability information, and the estimation type information and the estimation probability information, a plurality of candidate target tasks are determined. From among the aforementioned multiple candidates, the type of the target work is identified using the other information set of the two information sets, and without using the other information set. Work recognition device.

3. A work recognition device according to claim 2, The aforementioned work identification unit is, Using the aforementioned prediction type information and prediction probability information, the plurality of candidates are determined. From among the multiple candidates, the type of work with the highest probability represented in the estimated probability information is identified as the type of target work, using the estimated type information and the estimated probability information. Work recognition device.

4. A work recognition device according to any one of claims 1 to 3, The first model is configured to predict the type of the next operation from two or more consecutive types of the previous operations, The work identification unit is a work recognition device that inputs information representing the types of two or more consecutive work preceding the target work into the first model, thereby causing the first model to predict the type of the target work.

5. A method for recognizing the work performed by an employee, Observation information obtained by observing the target work using sensors is acquired. A work recognition method that identifies the type of target work using a first model that has been pre-trained to know the order in which multiple tasks are performed, the prediction result of the first model that predicts the type of the target work as the next task based on the type of the previous task, and the estimation result of a second model that has been pre-trained to estimate the type of the target work based on the observation information.

Images