Work recognition system and work recognition method

Abstract

To provide a work recognition system and a work recognition method capable of appropriately recognizing a work state.SOLUTION: A work recognition system 1 according to the present embodiment is a recognition system that recognizes a work state of a worker 200 who performs work on a workpiece W conveyed on a line 10, and includes a movable state acquisition unit that acquires a running state of the line, a sensor that detects motion information of the worker, a machine learning model that estimates a work content of the worker from the motion information detected by the sensor and outputs a reliability degree with respect to the estimation of the work content, and a determination unit that determines that the worker is in a hand-waiting state when the reliability degree is equal to or less than a threshold value during a stop period in which the line 10 is stopped.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an activity recognition system and an activity recognition method. [Background technology]

[0002] Patent Document 1 discloses an information processing device that identifies the work content of a person based on captured video. The information processing device in Patent Document 1 identifies the work content based on frame images through machine learning using a learning model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-149154 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, a worker performs multiple tasks A, B, and C in this order. Furthermore, consecutive integers such as 1, 2, and 3 are assigned as labels to the task order. The information processing device associates a task label corresponding to the identified task content with each frame. Furthermore, the information processing device corrects the task content based on the task order and the chronological relationship between a first period in which a first task content is identified and a second period in which a second task content is identified.

[0005] In this way, when identifying work content, there may be a waiting state in which the worker is not performing a predetermined work. During work, a waiting state may occur between each work due to a conveyor stop or the like. In a waiting state, the worker is not performing work. For example, in a waiting state, the worker may exhibit various postures, such as rotating their shoulders, stretching, or simply standing. In a waiting state, there are many movement patterns, making it difficult for a learning model to learn them. Therefore, there is a problem in that it is not possible to improve the recognition accuracy of the machine learning model, making it difficult to appropriately recognize the work state.

[0006] Therefore, an object of the present disclosure is to provide an activity recognition system and an activity recognition method that can appropriately recognize activity states. [Means for solving the problem]

[0007] The work recognition system according to the present disclosure is a recognition system that recognizes the work status of a worker working on a workpiece, and includes: an operating status acquisition means for acquiring the operating status of a line that transports the workpiece; a sensor that detects motion information of the worker working around the line; a machine learning model that estimates the work content of the worker from the motion information detected by the sensor and outputs a reliability of the estimated work content; and a determination means that determines that the worker is in a waiting state if the reliability is below a threshold during a stoppage period when the line is stopped.

[0008] The work recognition method according to the present disclosure is a recognition method that uses an information processing device to recognize the work status of a worker performing work on a workpiece, and includes the steps of acquiring the operating status of a line that transports the workpiece, using a sensor to detect motion information of the worker performing work around the line, using a machine learning model to estimate the work content of the worker from the motion information detected by the sensor and outputting a reliability of the estimated work content, and determining that the worker is in a waiting state if the reliability is below a threshold during a stop period when the operation status of the line is stopped. [Effects of the Invention]

[0009] The present disclosure provides an activity recognition system and an activity recognition method that can appropriately recognize the activity state of a worker. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram illustrating an overall configuration of an activity recognition system according to an embodiment; [Figure 2] 10 is a graph showing an example of an operating state, an estimation result, and a recognition result. [Figure 3] 1 is a flowchart illustrating a task recognition method. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential means for solving the problems. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. In each drawing, the same elements are given the same reference numerals, and duplicate explanations are omitted as necessary.

[0012] Let us consider an example where workers are assembling engines and other components in a factory that manufactures automobiles and other products. The workpieces are transported on a line such as a conveyor belt. The workers assemble the workpieces according to a pre-set procedure. For example, the workers attach parts to the workpieces. The parts attached to the workpieces include covers, brackets, guides, plates, studs, and wires.

[0013] Workers perform multiple tasks such as picking parts, placing parts, fixing parts, and flipping workpieces according to work procedures. Workers perform multiple tasks within a specified takt time. Picking parts includes walking to pick up parts from racks, etc. Fixing parts includes actions such as temporarily fastening, tightening, and fastening with bolts, etc. The work content and procedures that workers perform are specified in advance.

[0014] 1 is a schematic diagram showing the overall configuration of an activity recognition system 1. The activity recognition system 1 recognizes the activity state of a worker 200 who performs an activity on a workpiece W transported on a line 10. The activity recognition system 1 includes the line 10, a line control device 12, an image sensor 15, an operating state determination device 18, an estimation device 30, an information processing device 40, and a sensor 210.

[0015] The line 10 is equipped with a belt conveyor, roller conveyor, or the like that transports the workpieces W. The workpieces W flow on the line 10 in the direction of the arrow. The line 10 transports the workpieces W at a constant speed. Alternatively, the line 10 may transport the workpieces W in an intermittent operation that alternates between moving and stopping. A worker 200 performs line work such as assembly on the workpieces W near the line 10.

[0016] The line control device 12 controls the driving of the line 10. For example, the line control device 12 drives the motor of the line 10. The line control device 12 controls the line 10 to transport the workpiece W at a constant transport speed. Furthermore, when the line 10 operates intermittently, the line control device 12 controls the line 10 so that the line 10 operates in a predetermined cycle. The line control device 12 may output operation information indicating whether the line 10 is operating or stopped to the operation status determination device 18.

[0017] The image sensor 15 is a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and is installed in the vicinity of the line 10. The image sensor 15 captures images of the line 10 and its surroundings. The image sensor 15 may also capture images of the worker 200 in the vicinity of the line 10 or the workpiece W on the line 10. Alternatively, the image sensor 15 may capture images of only the worker 200. The image sensor 15 captures moving images or a series of still images. Of course, the task recognition system 1 may be equipped with two or more image sensors 15.

[0018] The image sensor 15 outputs the captured image as operation information to the estimation device 30. The image sensor 15 may also output the captured image to the operating state determination device 18. The operation information is information indicating the operation of the worker. The image sensor 15 may also output a part of the captured image as operation information to the estimation device 30.

[0019] The image sensor 15 may extract information extracted from the captured image as extracted information by image processing. For example, the image sensor 15 may extract feature quantities and the like from the captured image using a built-in processor. For example, the processor built into the image sensor 15 may extract feature quantities using various image filters. The image sensor 15 may also detect the skeleton of the worker 200 from the image and use information related to the skeleton as extracted information. The image sensor 15 may output the extracted information, rather than the captured image, to the estimation device 30 as motion information. Of course, the image sensor 15 may output both the captured image and the extracted information as motion information to the estimation device 30. The image sensor 15 may also output the extracted information to the operating state determination device 18.

[0020] The operating state determination device 18 determines the operating state of the line 10. Specifically, the operating state determination device 18 determines whether the line 10 is operating or stopped. A period when the work W on the line 10 is moving is an operating period, and a period when the work W on the line 10 is stopped is a stopped period. As described above, information indicating the operating state of the line 10 is input from the line control device 12 to the operating state determination device 18. Therefore, based on the operation information of the line 10, the operating state determination device 18 can determine the operating state of the line.

[0021] Alternatively, the operating state determination device 18 may determine the operating state of the line 10 based on the captured images or extracted information of the image sensor 15. For example, the image sensor 15 compares two or more consecutive frame images. If the position of the workpiece W has changed in two or more frame images, the image sensor 15 determines that the line 10 is operating. The information used by the operating state determination device 18 for its determination may be only the operating information from the line control device 12, or only the frame images from the image sensor 15.

[0022] The operating state determination device 18 outputs the determination result to the information processing device 40. For example, the output of the operating state determination device 18 is a binary value indicating whether the line is operating or stopped. An example of the output of the operating state determination device 18 is shown in the upper part of Figure 2. In Figure 2, the horizontal axis represents frames and the vertical axis represents the operating state. Here, the operating state determination device 18 makes a determination for each frame image, so the determination result for each frame is shown. The output of the operating state determination device 18 is time-series data in which 1 indicates that the line 10 is stopped and 0 indicates that the line 10 is operating. It functions as an operating state acquisition means for acquiring the operating state of the line 10. Alternatively, the line control device 12 may function as an operating state acquisition means for acquiring the operating state of the line 10.

[0023] The worker 200 may be wearing a sensor 210. The sensor 210 is an acceleration sensor, a pressure sensor, a GPS sensor, or the like, and detects the movement of the worker 200. The sensor 210 may be mounted on a wearable device such as a smartwatch or smart glasses. The sensor 210 may also be mounted on a smartphone or the like. The sensor 210 outputs the detected information to the estimation device 30 as movement information. For example, the sensor 210 detects the movement of the worker 200 from the acceleration of the arm or the like of the worker 200. Alternatively, the sensor 210 is a satellite positioning sensor or the like, and detects a change in the position of the worker 200.

[0024] Furthermore, the sensor 210 may have a viewpoint camera having an angle of view according to the viewpoint of the worker 200. The sensor 210, which is a viewpoint camera, captures an image of the workpiece W, a component, or the like that the worker 200 is looking at. The image of the viewpoint camera and information extracted from the image become the movement information. The sensor 210 is not limited to being worn by the worker 200, but may also be provided on a tool used by the worker 200. For example, the sensor 210 may be a pressure sensor or an acceleration sensor attached to the tool. The worker 200 may wear two or more sensors 210. For example, the worker 200 may wear an acceleration sensor and a viewpoint camera. The two or more sensors 210 may output movement information to the estimation device 30. Recognition accuracy can be improved by the sensor 210 detecting more movement information.

[0025] The estimation device 30 is a personal computer or the like, and has a processor, memory, and the like for performing arithmetic processing. The estimation device 30 performs the following processing by executing a program. The estimation device 30 estimates the work content of the worker 200 based on movement information indicating the worker's movements. The estimation device 30 estimates the work content using a machine learning model. The machine learning model inputs the worker's movement information and outputs a data value indicating the work content.

[0026] For example, if the work content of worker 200 is set to include transport, fastening, walking, and part picking, the machine learning model becomes a classifier that performs multi-value classification. The machine learning model outputs a data value that indicates the classification result. A different data value is set for each work. The machine learning model outputs consecutive integers such as 1, 2, 3, and 4 according to the work content.

[0027] Here, four work contents are set: transport, fastening, walking, and parts removal. The work contents estimated by the estimation device 30 are expressed as four values: parts removal is "1", walking is "2", fastening is "3", and transport is "4". Of course, the number of work contents is not limited to four, and any number greater than two, or even five or more, may be set. Furthermore, work contents other than the above four may be set. For example, work contents may be set for each work procedure shown in the work manual.

[0028] The estimation device 30 estimates the work content and outputs a data value indicating the estimated work content. The estimation device 30 may perform the estimation using a deep neural network (DNN) constructed by deep learning. Furthermore, the estimation device 30 may perform the estimation using a recurrent neural network (RNN) to which successive frame images, etc. are sequentially input.

[0029] For example, the machine learning model calculates a score for each task. The machine learning model estimates the task based on the score. Specifically, the machine learning model outputs a data value indicating the task with the highest score.

[0030] Here, the movement information is information indicating the movement of the worker, and is, for example, an image captured by the image sensor 15. For example, the image captured by the image sensor 15 includes the worker 200. One or more frames of the captured image are input to the estimation device 30. Alternatively, the movement information may be a part of the captured image, or extracted information extracted from the captured image. The estimation device 30 may estimate the worker's skeleton from the frame image and estimate the work content from the skeleton.

[0031] Furthermore, the estimation device 30 may estimate the work content based on the motion information detected by the sensor 210. In this case, the motion information is an image from a viewpoint camera or detection data from an acceleration sensor. By using the image captured by the image sensor 15 and the detection results from the sensor 210 as the motion information, the estimation accuracy can be improved. In this way, at least one of the sensor 210 and the image sensor 15 functions as a sensor that detects the motion information of the worker.

[0032] Furthermore, the machine learning model outputs the reliability of the estimation result. For example, the machine learning model calculates a score for each task and outputs the task with the highest score. Here, the machine learning model calculates the reliability of the estimation result according to the distribution of scores for all task contents. Of course, the machine learning model may calculate task contents and reliability using likelihoods or the like, instead of scores.

[0033] In this way, the estimation device 30 functions as a machine learning model that estimates the work content of the worker 200 from the motion information and outputs the reliability of the estimated work content. The estimation device 30 outputs the estimation result to the information processing device 40. That is, the estimation device 30 outputs a data value indicating the estimated work content and the reliability of the estimation to the information processing device 40.

[0034] The middle part of Fig. 2 is a graph showing the estimation results of the estimation device 30. The estimated work content is shown by the solid line graph, and the reliability is shown by the dashed line graph. For example, the reliability is shown as a continuous value ranging from 0 to 1. The higher the reliability, the more reliable the work content estimation result. For example, the more uniform the score distribution, the lower the reliability.

[0035] The machine learning model may be constructed by supervised learning. For example, the estimation device 30 or another information processing device performs machine learning as a learner. The learner constructs the machine learning model by performing machine learning in advance based on the work content performed by the worker. The learner may be the same device as the estimation device 30 or a different device. Furthermore, the learner may include two or more processing devices.

[0036] The learning device uses captured images of a worker using the image sensor 15 as training data. The learning device can also assign a ground truth label to the training data for each task. The learning device prepares captured images for each task and assigns a ground truth indicating the task to each captured image. For example, when the image sensor 15 captures multiple images of a worker performing a transport task, the learning device assigns a ground truth label indicating the transport task to each image. Similarly, when the image sensor 15 captures multiple images of a worker performing fastening, walking, and picking up tasks, the learning device assigns a ground truth label indicating each task to each image. When the sensor 210 is used, the learning device assigns a ground truth label to the detection data of the sensor 210 to create training data. The learning device then performs supervised learning using the teacher data to which the ground truth labels have been assigned.

[0037] If the reliability is equal to or less than a threshold during a stop period when the line 10 is stopped, the information processing device 40 determines that the worker 200 is in a waiting state. The bottom part of FIG. 2 shows the determination result of the information processing device 40. The line 10 is stopped during period R. If the reliability is equal to or less than a threshold th during period R, the information processing device 40 recognizes that the worker 200 is in a waiting state. Note that when frames of a captured image are used as a reference, period R is a frame section including one or more frames.

[0038] In other words, during the period when the line 10 is in operation, even if the reliability is greater than the threshold th, the work content estimated by the estimation device 30 is used as the recognition result as is. Also, when the reliability is equal to or greater than the threshold th, even during the period R when the line 10 is stopped, the work content estimated by the estimation device 30 is used as the recognition result as is. In other words, when the reliability is greater than the threshold th, the information processing device 40 does not recognize the line 10 as being in a waiting state. Similarly, during the period when the line 10 is in operation, the information processing device 40 does not recognize the line 10 as being in a waiting state. When the information processing device 40 does not recognize the line 10 as being in a waiting state, the information processing device 40 outputs the work content estimated by the estimation device 30 as the recognition result.

[0039] In this way, the information processing device 40 recognizes the worker 200 as being in a waiting state based on the operating state and reliability. This allows for improved recognition accuracy. In a waiting state, the worker may perform various actions. Using training data on all of these actions, the information processing device 40 can recognize the worker as being in a waiting state without machine learning. Furthermore, the recognition result of the worker can indicate that the worker is in a waiting state, not just the predefined work content. This allows for more detailed recognition of the work.

[0040] Although the operational state determination device 18, the estimation device 30, and the information processing device 40 are described as separate devices, they may be a single device. For example, a single information processing device 40 may have the functions of the operational state determination device 18 and the estimation device 30. Alternatively, the operational state determination device 18, the estimation device 30, or the information processing device 40 may perform distributed processing using multiple physical devices. The estimation device 30 may perform estimation using a machine learning model distributed on the cloud. Furthermore, the operational state determination device 18 may be mounted on the image sensor 15 or the line control device 12.

[0041] The task recognition method will be described with reference to Fig. 3. First, the image sensor 15 or the sensor 210 acquires motion information of the worker (S11). Both the image sensor 15 and the sensor 210 output the motion information to the estimation device 30. Of course, both the image sensor 15 and the sensor 210 may detect motion information and output it to the estimation device 30. The estimation device 30 estimates the task content using a machine learning model (S12). That is, the estimation device 30 inputs the motion information into the machine learning model, thereby outputting the task content of the worker to the information processing device 40.

[0042] Furthermore, the operating status determination device 18 acquires the operating status of the line (S21). The operating status determination device 18 outputs the operating status to the information processing device 40. The information processing device 40 determines whether the line is stopped during period R (S22). If the line is not stopped during period R (NO in S22), the information processing device 40 outputs the work recognition result (S31). That is, the work content indicated by the estimation result is output as the recognition result.

[0043] If the line is stopped during period R (YES in S22), the information processing device 40 determines whether the reliability of the estimation result during period R is equal to or less than threshold th (S23). If the reliability is greater than threshold th (NO in S23), the information processing device 40 outputs the task recognition result (S31). That is, the task content indicated by the estimation result is output as the recognition result.

[0044] If the reliability is equal to or less than the threshold th (YES in S23), the information processing device 40 determines that the worker is in a waiting state during the period R (S24). In this way, it is possible to recognize with high accuracy a waiting state, which is difficult to estimate using a machine learning model.

[0045] In this way, the task recognition system 1 can recognize with high accuracy that a worker is in a waiting state. In a waiting state, the worker's movements and posture are not specified, which can reduce the reliability of estimations made by the machine learning model. For example, in a waiting state, a worker may rotate their shoulders or stretch. Or, the worker may simply be standing upright. It is difficult to prepare training data for all such movements. As in this embodiment, recognition of a waiting state based on the reliability of estimation and line operation information can improve recognition accuracy.

[0046] Furthermore, part or all of the processing in the above-described operating state determination device 18, estimation device 30, information processing device 40, etc. can be realized as a computer program. Such a program can be stored using various types of non-transitory computer-readable media and provided to a computer. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be provided to a computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable media can provide the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0047] The present invention is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the invention. [Explanation of symbols]

[0048] 10 lines 12 Line control device 15 Image Sensor 18 Operational status determination device 30 Estimation device 40 Information processing equipment 200 workers 210 Sensors

Claims

1. A recognition system that recognizes the working state of a worker performing work on a workpiece, an operating status acquisition means for acquiring an operating status of the line that conveys the workpiece; a sensor that detects motion information of the worker performing work around the line; a machine learning model that estimates the work content of the worker from the motion information detected by the sensor and outputs a reliability of the estimation of the work content; and determining means for determining that the worker is in a waiting state when the reliability is equal to or less than a threshold during a stop period when the line is stopped.

2. The task recognition system according to claim 1 , wherein the sensor includes an image sensor that captures images of the line and the worker.

3. The task recognition system according to claim 2 , wherein the machine learning model receives an image captured by the image sensor as an input and outputs the task content and reliability.

4. 4. The task recognition system according to claim 1, wherein the sensor includes an acceleration sensor, a pressure sensor, a viewpoint camera, or a GPS sensor worn by the worker.

5. A recognition method for recognizing a work state of a worker performing work on a workpiece using an information processing device, comprising: acquiring an operating status of a line that transports the workpiece; detecting, using a sensor, motion information of the worker performing work around the line; using a machine learning model to estimate the work content of the worker from the motion information detected by the sensor, and outputting a reliability of the estimation of the work content; and determining that the worker is in a waiting state if the reliability is equal to or less than a threshold during a stop period when the operation of the line is stopped.

Images