Monitoring device and monitoring method

Abstract

To provide a monitoring device and a monitoring method that improve the accuracy of determining whether an operator's behavior is appropriate or not.SOLUTION: A monitoring device 14 includes an image acquisition unit 21 that acquires an image including a part or all of an operator's hand and an operation target operated by the operator using an imaging device 11 attached to the arm of the operator who performs the operation, an operation determination unit 23 that determines whether the operation is a predetermined operation using the image acquired by the image acquisition unit 21, and a determination result output unit 25 that outputs a determination result from the operation determination unit 23.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an apparatus and method for monitoring an operator performing a predetermined operation. [Background technology]

[0002] A method is known in which the operation of an operating tool placed on a control panel is detected by correlating the coordinates of the operating tool on the control panel with the time when the operating tool is operated by an operator (hereinafter also referred to as "operator"), the coordinates of the operator's line of sight on the control panel are detected by correlating them with the time when the line of sight was detected, an image showing the coordinates of the operator's operating position on the control panel is acquired by correlating it with the time when the image was acquired, and when the operator operated the operating tool, it is determined whether the operator's line of sight and the coordinates of the operating position were aligned with the coordinates of the operating tool, thereby determining whether the operator acted appropriately without looking away (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-13958 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above-mentioned conventional technology, if the operator's body or operating tools become obstacles and the camera cannot detect the operator's operating position, it is not possible to determine whether the operator's actions were appropriate, resulting in a problem of reduced accuracy in judgment.

[0005] The problem to be solved by the present invention is to provide a monitoring device and a monitoring method that improve the accuracy of determining whether the behavior of an operator is appropriate. [Means for solving the problem]

[0006] The present invention provides a mounting portion for mounting on the palm side of an operator's forearm; The present invention solves the above problem by using an imaging device attached to the arm of an operator performing an operation to acquire an image including an operation target, which is an object on which the operator performs an input operation, and a part or all of the operator's hand performing the input operation on the object, and determining whether or not an operation identified from the positional relationship between the operator's hand and the operation target in the acquired image is a predetermined operation. a mounting portion for mounting on the palm side of an operator's forearm; The above problem is solved by using an imaging device attached to the arm of the operator performing the operation to acquire an image including the operation target, which is the object on which the operator performs the input operation, and part or all of the operator's hand that is in contact with the object, and using the acquired image to determine whether the operator's operation is a specified operation. [Effects of the Invention]

[0007] According to the present invention, an image suitable for determining the behavior of an operator can be acquired, and the accuracy of determining the behavior of an operator can be improved. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing one embodiment of a monitoring system including a monitoring device according to the present invention. [Figure 2] 2 is a side view of an operator's arm showing an example of the positional relationship between the imaging device 11, the detection device 12, and the display device 13 shown in FIG. [Figure 3] FIG. 2 is an explanatory diagram illustrating an example of a trained model used in the operation identification unit of FIG. 1. [Figure 4] FIG. 10 is an explanatory diagram showing an example of a predetermined procedure. [Figure 5] FIG. 2 is an explanatory diagram showing an example of a trained model used in the operation procedure determination unit of FIG. 1; [Figure 6] 2 is a flowchart showing an example of an information processing procedure in the monitoring system of FIG. 1. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a monitoring device and a monitoring method according to embodiments of the present invention will be described with reference to the accompanying drawings.

[0010] [Monitoring system] 1 is a block diagram showing a monitoring system 1 according to the present invention. The monitoring system 1 is a device that monitors the operation of an operator who performs an operation on a certain operation object, and can be used, for example, in a vehicle assembly factory to monitor whether a worker engaged in assembly work has operated a tool in accordance with a predetermined procedure, or in a vehicle dealership (hereinafter also referred to as a "dealer") to monitor whether a mechanic engaged in vehicle maintenance has performed maintenance in accordance with a procedure defined in a manual. There are no particular limitations on the operators whose operations are monitored by the monitoring system 1, and examples include factory workers and dealer mechanics.

[0011] As shown in FIG. 1, the monitoring system 1 includes an imaging device 11, a detection device 12, a display device 13, and a monitoring device 14. The devices constituting the monitoring system 1 are connected to each other via known means such as a wired or wireless LAN so that they can exchange data with each other. The number of imaging devices 11, detection devices 12, and display devices 13 is not particularly limited as long as there is at least one of each. Furthermore, the imaging device 11, detection device 12, and display device 13 do not need to be installed together with the monitoring device 14, and may be installed in locations remote from the monitoring device 14. For example, the imaging device 11, detection device 12, and display device 13 may be installed near an assembly line in an assembly factory, and the monitoring device 14 may be installed in a central control room away from the assembly line or in a server in a remote location away from the assembly factory.

[0012] The imaging device 11 is a device for acquiring image data of surrounding objects, such as a camera equipped with an imaging element such as a CCD, an ultrasonic camera, or an infrared camera. The objects include not only the operator but also objects present around the operator. Examples of the objects include switches and touch panels arranged around the worker, parts and tools being assembled by the worker, and vehicles being serviced by dealership mechanics. The imaging device 11 is attached to the operator's arm, thereby enabling reliable acquisition of images of the operator operating the object. The imaging device 11 has an attachment portion for attachment to the operator's arm, and can be attached to the palm side of the operator's upper arm using, for example, a hook-and-loop fastener or an adhesive pad.

[0013] The detection device 12 is a device for detecting the status of the operator and the conditions around the operator, and includes any type of sensor. Examples of sensors include an electromyograph that measures the operator's myoelectric potential, a microphone that captures sounds around the operator as audio data, an accelerometer attached to the operator to detect the operator's physical movements, and a thermometer that measures the temperature around the operator. The electrodes included in the electromyograph may be needle electrodes, surface electrodes, or wire electrodes. The electromyograph is attached to the operator's body, particularly the arm including the upper arm and / or forearm, using, for example, an adhesive pad or hook-and-loop fastener, and measures the myoelectric potential of the part of the body that comes into contact with the attached electromyograph.

[0014] Microphones include stand microphones, close-talking microphones, and gun microphones, and may be omnidirectional or directional. The communication method may be either wired or wireless. The microphone captures the operator's voice and the voices of people around the operator, as well as sounds resulting from the operator's actions. Examples of sounds resulting from the user's actions include the sound generated from a switch operated by the worker, the sound generated from a speaker when the worker touches a touch panel, the sound of parts engaging when the worker connects multiple parts, and the operating sounds of tools used by dealership mechanics. The microphone is installed in a position where it can detect sounds around the operator, such as the assembly line in an assembly plant, the workbench, and the tools used by the worker.

[0015] The accelerometer is not particularly limited, but is preferably of a size that can be attached to the operator's body, particularly the arm including the upper arm and / or forearm. For example, a small accelerometer using MEMS (Micro Electro Mechanical Systems) can be used. The thermometer is also not particularly limited, and may be a contact thermometer or a non-contact thermometer using infrared rays. The monitoring device 14 obtains detection results from these sensors at predetermined time intervals.

[0016] The display device 13 is a device for notifying the operator and / or the supervisor of the operations monitored by the monitoring device 14. The supervisor does not need to be near the operator and may be located in a central control room of the factory or in a remote location away from the factory. The display device 13 is, for example, an LCD display, a projector, or the like, and may be equipped with a speaker. The display device 13 is installed, for example, at a location near the operator, such as the worker's work area in an assembly factory, where it can notify the operator of necessary information, or at a location where it can notify a supervisor located away from the operator of necessary information. The display device 13 may also be attached to the operator as a wearable terminal. Furthermore, the imaging device 11, the detection device 12, the display device 13, and the monitoring device 14 may be integrated into a single wearable terminal attached to the operator.

[0017] FIG. 2 shows an example of the positional relationship between the imaging device 11, the detection device 12, and the display device 13. In FIG. 2, the imaging device 11 is attached to the operator's forearm A1 using an attachment portion 11a, and the detection device 12 is attached to the operator's upper arm A2 using an adhesive pad. The operator holds an operation target O in his hand H. If the viewing angle of the imaging device 11 is within the range indicated by VA, the imaging device 11 can capture an image including a part of the operator's hand H and the operation target O. The image data captured by the imaging device 11 and the detection result of the detection device 12 are transmitted to the monitoring device 14 via wireless communication. In FIG. 2, the display device 13 is provided in front of the operator, and the operator can check the display on the display device 13.

[0018] Returning to FIG. 1, the monitoring device 14 is a device for monitoring whether an operator has properly performed a certain operation. Image data used for the monitoring and the detection results of the sensor are acquired at predetermined time intervals from the imaging device 11 and the detection device 12, respectively. The monitoring device 14 uses a processor 15 to realize functions such as processing the acquired data, determining whether the operation has been properly performed, and outputting the results of the determination. The processor 15 includes a ROM (Read Only Memory) 152 in which a program is stored, a CPU (Central Processing Unit) 151 which is an operating circuit for functioning as the monitoring device 14 by executing the program stored in the ROM 152, and a RAM (Random Access Memory) 153 which functions as an accessible storage device.

[0019] [Operation monitoring section] The program used in the monitoring device 14 of this embodiment includes an operation monitoring unit 2, which is a functional block for enabling the monitoring device 14 to perform functions such as processing acquired data, determining whether an operation has been performed appropriately, and outputting the results of the determination. The operation monitoring unit 2 has the functions of acquiring image data from the imaging device 11, acquiring detection results from the detection device 12, identifying an operation performed by a user based on the acquired data, determining whether the identified operation has been performed appropriately in a predetermined procedure, and outputting the determination result. As shown in FIG. 1, the operation monitoring unit 2 includes an image acquisition unit 21, an operation identification unit 22, an operation determination unit 23, an operation procedure determination unit 24, and a determination result output unit 25. FIG. 1 illustrates each unit extracted for convenience.

[0020] 1 includes all of the above-mentioned functional blocks, it is not necessary for a single monitoring device 14 to include all of the functional blocks, and some of the above-mentioned functional blocks may be provided in other devices included in the monitoring system 1 or in another information processing device (not shown). For example, in the monitoring system 1 of FIG. 1, the determination result output unit 25 may be provided in the display device 13. In this case, the functions of the determination result output unit 25 are executed using the CPU, ROM, and RAM of the display device 13.

[0021] Furthermore, it is not necessary for all of the processes of each functional block to be executed by a single device, and the functions of each functional block may be realized across multiple devices that are connected in a state where data can be exchanged. For example, in the monitoring system 1 of FIG. 1, part of the processes executed by the operation identification unit 22 may be executed by the imaging device 11, and the remaining processes may be executed by the monitoring device 14. In this case, part of the processes for realizing the function of the operation identification unit 22 is performed using the CPU, ROM, and RAM of the imaging device 11. As another example, part of the processes executed by the operation determination unit 23 and / or the operation procedure determination unit 24 may be executed by the imaging device 11, and the remaining processes may be executed by the monitoring device 14.

[0022] The functions performed by each functional block of the operation monitoring unit 2 will be described below.

[0023] The image acquisition unit 21 has a function of acquiring an image including an operator performing an operation and an operation object operated by the operator. In particular, the image acquisition unit 21 of this embodiment acquires image data from an imaging device 11 attached to the operator's arm, and the acquired image data includes a part or all of the operator's hand H and an operation object O operated by the operator. The operator's hand H may be a right hand or a left hand, and it is sufficient that at least a part of the hand H is included. However, in particular, if a part or all of the operator's palm and the operation object O are included in the image, the operation can be determined more accurately.

[0024] The operation performed by the operator is not particularly limited, and includes any operation in which the operator inputs some kind of information to an operation target. Specific examples include an operation in which an assembly plant worker touches a touch panel to change the display of an assembly line device, an operation in which an assembly plant worker fits a coupler connected to a sensor with a coupler connected to an electronic control unit (ECU), an operation in which an assembly plant worker tightens bolts using a tool to attach an exhaust manifold to an engine block, an operation in which a dealership mechanic fits a spark plug into an engine, and an operation in which a dealership mechanic tightens a bolt using a torque wrench.

[0025] The operation target is not particularly limited, and includes any object that the operator can use to perform some kind of input operation, such as an ON / OFF switch, a touch panel user interface, parts such as couplers and exhaust manifolds, and tools such as torque wrenches and electric screwdrivers.

[0026] Furthermore, the image acquisition unit 21 can change the quality of the images acquired from the imaging device 11, and may reduce the frame rate of the acquired time-series images (video) while the operation determination unit 23 (described later) is not processing the image data, such as while the operator is not performing an operation. For example, if the operation determination unit 23 (described later) determines that the operator's hand movement does not correspond to a predetermined movement, the frame rate of the acquired time-series images may be reduced below a predetermined frame rate (e.g., 60 fps). The predetermined frame rate can be set to an appropriate value within a range in which the monitoring device 14 can appropriately monitor the operation. On the other hand, while the operator is actually performing an operation, the frame rate of the acquired time-series images may be increased above the predetermined frame rate.

[0027] The operation identification unit 22 has a function of identifying an operation performed by an operator from the image data acquired by the image acquisition unit 21. The operation identification unit 22 performs analysis such as pattern matching on the image data acquired from the imaging device 11 and classifies the objects included in the image data. Next, the operation identification unit 22 selects an operator from the classified objects and acquires data related to the operator's hand extracted from the image data. Similarly, the operation identification unit 22 selects an operation target from the classified objects and acquires data related to the operation target extracted from the image data. Then, from this data, the positional relationship between the operator's hand and the operation target is recognized, and the user's operation is identified from the recognized positional relationship.

[0028] A correspondence between the positional relationship between the operator's hand H and the operation target O, and the operation the operator is about to perform is determined in advance for each operation and is stored in a database such as the database 16. The operation identification unit 22 can acquire the correspondence between the positional relationship and the operation from the database 16 as necessary. The operation identification unit 22 identifies the operation the operator is about to perform from the image data acquired by the image acquisition unit 21, based on the correspondence acquired from the database 16.

[0029] As an example, a case will be described in which the image data acquired by the image acquisition unit 21 includes a state in which a worker working on a factory assembly line holds a coupler in his left hand and extends his right hand to another coupler. In this case, the operation identification unit 22 searches for an operation corresponding to the positional relationship between the right hand extended toward the coupler and the coupler from the correspondence relationship between positional relationships and operations acquired from the database 16. If the operations corresponding to the relevant action are an operation of removing one coupler from a pair of already engaged couplers and an operation of engaging a coupler connected to a sensor with a coupler connected to an electronic control device, the operation identification unit 22 determines that the positional relationship corresponds to an operation of engaging a coupler connected to a sensor with a coupler connected to an electronic control device because the worker is holding the coupler in his left hand.

[0030] As another example, a case will be described in which the image data acquired by the image acquisition unit 21 includes a state in which a dealer's mechanic holds engine oil in his left hand and extends his right hand into the engine compartment. In this case, the operation identification unit 22 searches for an operation corresponding to the positional relationship between the right hand extended into the engine compartment and the right hand inside the engine compartment from the correspondence relationship between positional relationships and operations acquired from the database 16. If the operations corresponding to the action are an operation to refill engine coolant and an operation to refill engine oil, the operation identification unit 22 determines that the positional relationship corresponds to the operation of refilling engine oil because the mechanic is holding engine oil in his left hand.

[0031] Furthermore, the operation identification unit 22 may identify an operation using time-series images, such as a video, acquired by the image acquisition unit 21. The operation identification unit 22 acquires image data from the image acquisition unit 21 at predetermined time intervals and analyzes the image data. In this way, by using time-series images having predetermined time intervals, an operation can be identified more accurately. The time intervals can be set appropriately depending on the calculation capacity of the CPU 151. Alternatively or in addition to this, the operation identification unit 22 may identify an operation using the detection result of the detection device 12. For example, the operation of the operator may be detected from the value of the operator's myoelectric potential detected by an electromyograph attached to the operator's upper arm and the value of acceleration detected by an accelerometer attached to the operator's forearm, and the detected operation may be used to identify the operation.

[0032] Furthermore, the operation identification unit 22 may identify an operation using a first trained model that has been trained to identify an operation from image data acquired by the image acquisition unit 21. The trained model is a model that has been trained in advance by machine learning so as to obtain appropriate output data for certain input data, and includes at least a program that performs calculations from input data to obtain output data, and weighting coefficients (parameters) used in the calculations. When image data acquired by the image acquisition unit 21 is input as input data, the first trained model of this embodiment causes a computer (particularly, the CPU 151 of the processor 15) to function so as to identify an operation that the operator is attempting to perform on the image based on the input data and output output data. By using such a first trained model, an operation can be identified even for image data showing a positional relationship for which no correspondence has been previously established.

[0033] The first trained model of this embodiment is not particularly limited, and may be, for example, a first neural network 3 as shown in FIG. 3 . The first neural network 3 includes a first input layer 31, a first hidden layer 32, and a first output layer 33, each of which includes at least one neuron. The first input layer 31 receives first input data 34 including at least one of image data acquired by the image acquisition unit 21 and the detection result of the detection device 12, and outputs the input data to the first hidden layer 32. The first hidden layer 32 extracts the positional relationship between the operator's hand and the operation target from the data input from the first input layer 31. Then, it identifies an operation corresponding to the extracted positional relationship. The first output layer 33 outputs the data input from the first hidden layer 32 as first output data 35 including the identified operation.

[0034] In the first intermediate layer 32, a correspondence between the positional relationship between the operator's hand and the operation object and the operation the operator is attempting to perform is established through machine learning to identify the operation the operator is attempting to perform. This allows the operation the operator is attempting to perform to be identified from the positional relationships between the operator's hand and the operation object, such as the positional relationship between the operator's hand and the switch, the positional relationship between the operator's hand and the touch panel user interface, the positional relationship between the operator's hand and the part, and the positional relationship between the mechanic's hand and the tool. This correspondence is associated with multiple factors, such as the positional relationship between the operator's hand and the operation object, as well as the operator's surroundings detected by the detection device 12. Parameters are set so that the appropriate operation is output for the input data. For example, the fact that the operator is holding a coupler in each hand and the fact that the mechanic is holding engine oil in his left hand are taken into account as parameters when identifying the operation.

[0035] The correspondences in the first hidden layer 32 may be learned in advance by machine learning. Alternatively, the correspondences may be newly learned using first teacher data 36 including first input data 34 previously input to the first neural network 3 and first output data 35 previously output from the first neural network 3. Alternatively, a trained model previously trained by machine learning may be further trained. This learning is performed by the first machine learning unit 22a included in the operation identification unit 22. The first teacher data 36 is stored in a database such as the database 16 shown in FIG. 1 and can be acquired as needed. By learning using a combination of past first input data 34 and first output data 35, the operation identification unit 22 can more accurately identify an operation based on the positional relationship between the operator's hand and the operation target.

[0036] Returning to FIG. 1 , the operation determination unit 23 has a function of determining whether or not the operation identified by the operation determination unit 23 is a predetermined operation, using image data acquired by the image acquisition unit 21. The predetermined operation is an operation that is predetermined as an operation to be performed by an operator. For example, for a factory worker, operations such as fitting a coupler connected to a sensor with a coupler connected to an electronic control unit (ECU) and tightening a bolt using a tool to attach an exhaust manifold to an engine block are predetermined as predetermined operations. The operation determination unit 23 determines whether or not the operation identified by the operation determination unit 23 is a predetermined operation, using, for example, a series of images acquired by the image acquisition unit 21, and outputs the determination result to the operation procedure determination unit 24 and the determination result output unit 25.

[0037] The operation determination unit 23 can also acquire the movement of the operator's hand H from the image acquired by the image acquisition unit 21 and determine whether the acquired movement corresponds to a predetermined movement. If the movement of the hand H is determined to correspond to the predetermined movement, the operation determination unit 23 determines whether the identified operation corresponds to a predetermined operation only during the period from a predetermined time before the movement of the hand H was determined to be the predetermined movement to a predetermined time after the movement of the hand was determined to be the predetermined movement. In this way, the operation determination can be performed only during the period before and after the movement of the hand H was determined to be the predetermined movement, which is when the determination of the operation is necessary. The movement of the operator's hand H can be acquired by analyzing the image data acquired by the image acquisition unit 21, the value of the myoelectric potential detected by an electromyograph attached to the operator's upper arm A2, and the value of acceleration detected by an accelerometer attached to the operator's forearm A1. The predetermined movement is preset as the initial movement of the hand H for the predetermined operation and stored in the database 16, etc. The operation determination unit 23 acquires data of the predetermined movement from the database 16 as needed.

[0038] The operation procedure determination unit 24 has a function of determining whether or not the predetermined operation has been performed in a predetermined procedure when the operation determination unit 23 determines that the operation identified by the operation identification unit 22 corresponds to the predetermined operation. In particular, the operation procedure determination unit 24 determines whether or not the predetermined operation has been performed in a predetermined procedure by using the time-series image data acquired by the image acquisition unit 21. The predetermined procedure is set in advance for each predetermined operation and is stored in a database such as the database 16.

[0039] The operation procedure determination unit 24 acquires the identified operations from the operation identification unit 22 at predetermined time intervals and grasps the positional relationship between the operator's hand and the operation target contained in the image data. Then, it grasps the operator's operations in chronological order and, for each operation, determines whether the identified operation matches the predetermined procedure described above. If there is no deviation from the predetermined procedure until the identified operation is completed, the operation procedure determination unit 24 determines that the predetermined operation was performed according to the predetermined procedure. On the other hand, if the identified operation differs from the predetermined procedure or if the identified operation deviates from the predetermined procedure, it determines that the predetermined operation was not performed according to the predetermined procedure. These determination results are output to the determination result output unit 25.

[0040] As an example, a case where the predetermined operation is an operation of mating two couplers will be described using FIG. 4. In FIG. 4, a coupler C1 held in a worker's left hand is mated with a coupler C2 held in the left hand. When the mating portions Z of the couplers C1 and C2 are properly mated, an operating sound S is generated. In this case, the predetermined procedure is, for example, as shown in FIG. 4, a procedure consisting of four steps: step 1) the worker holds couplers C1 and C2 with his left and right hands, respectively; step 2) combines coupler C1 with coupler C2; step 3) pushes couplers C1 and C2 together until the mating portions Z of the couplers engage and generate an operating sound S that clicks; and step 4) releases the hands from couplers C1 and C2 when the operating sound S is generated. The operation procedure determination unit 24 acquires the identified operation from the operation identification unit 22 at predetermined time intervals and determines the positional relationship between the operator's hands mating the couplers and the couplers. The system then chronologically tracks how the operator who engages the coupler operates the coupler, and each time determines whether the four steps described above match the identified operation.

[0041] If the coupler engagement is completed according to the above steps, it is determined that the coupler has been properly engaged according to the predetermined procedure. On the other hand, if the coupler was not pushed in properly in step 3) and therefore not properly engaged, it is determined that the coupler has not been properly engaged. Here, the operation procedure determination unit 24 may use the detection results of the detection device 12 when determining whether the predetermined operation has been performed according to the predetermined procedure. In this case, for example, the operator's action of engaging the coupler may be detected from a myoelectric potential value acquired by an electromyograph attached to the operator's upper arm and an acceleration value acquired by an accelerometer attached to the operator's forearm accelerometer, and the detected values ​​may be used for the determination by the operation procedure determination unit 24. Furthermore, a microphone installed near the operator engaging the coupler may detect a clicking sound S generated when the coupler engagement portion Z engages, and the detected sound may be used for the determination by the operation procedure determination unit 24.

[0042] As another example, a case where the predetermined operation is an operation for refilling engine oil will be described. In this case, the predetermined procedure consists of four steps: 1) removing the oil filler cap, 2) pouring engine oil into the oil inlet, 3) closing the oil filler cap, and 4) checking the engine oil level with the oil level gauge. The operation procedure determination unit 24 acquires the identified operations from the operation identification unit 22 at predetermined time intervals and determines the positional relationship between the operator's hands when refilling oil and the engine oil and oil level gauge. The operation procedure determination unit 24 then chronologically determines the operations performed by the operator when refilling oil and determines each time whether the four steps described above match the identified operations.

[0043] If the oil refilling is completed according to the above steps, it is determined that the engine oil has been properly refilled according to the predetermined procedure. On the other hand, if the engine oil level is not confirmed in step 4), it is determined that the engine oil has not been properly refilled. As in the above example, the operation procedure determination unit 24 may use the detection result of the detection device 12 when determining whether the predetermined operation has been performed according to the predetermined procedure. In this case, for example, the operation of gripping the oil level gauge may be detected from the myoelectric potential value acquired by an electromyograph attached to the operator's upper arm and the acceleration value acquired by an accelerometer attached to the operator's forearm accelerometer, and the detected values ​​may be used for the determination by the operation procedure determination unit 24.

[0044] Furthermore, the operation procedure determination unit 24 may determine whether a predetermined operation has been performed in a predetermined procedure using a second trained model trained to determine whether a predetermined operation has been performed in a predetermined procedure using time-series image data acquired by the image acquisition unit 21 and / or the detection result of the detection device 12 that detects the state around the operator. The second trained model of the present embodiment, when time-series image data acquired by the image acquisition unit 21 and / or the detection result of the detection device 12 that detects the state around the operator are input as input data, identifies the operation that the operator is attempting to perform in the image based on the input data, and causes the computer (particularly, the CPU 151 of the processor 15) to function so as to output output data including a determination result as to whether the operator performed the identified operation in a predetermined procedure. By using such a second trained model, it is possible to determine whether an operation has been performed in a predetermined procedure even for image data indicating a positional relationship for which a correspondence has not been previously established.

[0045] The second trained model is not particularly limited, but may be, for example, a second neural network 4 as shown in FIG. 5 . The second neural network 4 includes a second input layer 41, a second hidden layer 42, and a second output layer 43, each of which includes at least one neuron. The second input layer 41 receives second input data 44, which includes at least one of the time-series image data acquired by the image acquisition unit 21 and the detection result of the detection device 12, and outputs the input data to the second hidden layer 42. The second hidden layer 42 extracts the positional relationship between the operator's hand and the operation target from the data input from the second input layer 41. Next, it identifies an operation corresponding to the extracted positional relationship. It then determines whether the identified operation corresponds to a predetermined operation, and if the identified operation corresponds to the predetermined operation, it determines whether the predetermined operation was performed according to a predetermined procedure. The second output layer 43 outputs the data input from the second hidden layer 42 as second output data 45, which includes a determination process for whether the predetermined operation was performed according to the predetermined procedure.

[0046] In the second intermediate layer 42, similar to the first intermediate layer 32 described above, a correspondence between the positional relationship between the operator's hand and the operation target and the operation the operator is about to perform is established by machine learning in order to identify the operation the operator is about to perform. This correspondence is associated with multiple factors, such as the positional relationship between the operator's hand and the operation target, as well as the state of the operator's surroundings detected by the detection device 12, and parameters are set so that an appropriate operation is output for the input data. In addition, in the second intermediate layer 42, a correspondence between the identified operation and a predetermined procedure is established in order to determine whether the identified operation was performed in a predetermined procedure when the identified operation corresponds to a predetermined operation. Specifically, the correspondence between data in which the identified operations are arranged in chronological order and each step of the predetermined procedure is learned by machine learning, and necessary parameters are set so that the identified operation can be appropriately compared with the predetermined procedure.

[0047] The correspondence relationship in the second hidden layer 42 may be learned in advance by machine learning, or may be newly learned using second teacher data 46 including second input data 44 previously input to the second neural network 4 and second output data 45 previously output from the second neural network 4, or a trained model previously trained by machine learning may be further trained. The learning is performed by a second machine learning unit 24a included in the operation procedure determination unit 24. The second teacher data 46 is stored in a database such as the database 16 shown in FIG. 1 and can be acquired as needed. Learning using a combination of past second input data 44 and second output data 45 allows the operation procedure determination unit 24 to more accurately determine whether a predetermined operation has been performed according to the predetermined procedure.

[0048] Returning to Fig. 1, the judgment result output unit 25 has a function of acquiring the judgment results of the operation judgment unit 23 and the operation procedure judgment unit 24 and outputting them to an external device, and in particular, when the operation judgment unit 23 judges that the operation performed by the operator is not a predetermined operation, it outputs to the display device 13 a message that the operation is inappropriate. Also, when the operation procedure judgment unit 24 judges that the predetermined operation has not been performed according to the predetermined procedure, it outputs to the display device 13 a message that the operation is inappropriate. By outputting these judgment results to the display device 13 and displaying them on the display device 13, it is possible to notify the operator and / or a supervisor that the predetermined operation has not been performed appropriately.

[0049] The power supply of the monitoring device 14 may be turned on after it is confirmed from the detection result of the detection device 12 that the operator has started using the monitoring system 1. Furthermore, the determination results of the operation determination unit 23 and the operation procedure determination unit 24 may be output to a server (not shown).

[0050] [Processing in the operation detection system] The procedure for processing information by the monitoring device 14 will be described with reference to Fig. 6. Fig. 6 is an example of a flowchart showing information processing in the monitoring system 1 of this embodiment. The processing described below is executed by the processor 15 of the monitoring device 14 at predetermined time intervals.

[0051] First, in step S1, an image including all or part of the hand of the operator performing the operation and the operation object operated by the operator is acquired from the imaging device 11 by the function of the image acquisition unit 21. For example, when a worker engaged in assembly work in a factory tightens a bolt using a tool, an image including part of the worker's hand, the bolt, and the tool is acquired.

[0052] In the next step S2, the detection device 12 detects the state of the operator's surroundings through the function of the operation identification unit 22. For example, when a worker engaged in assembly work in a factory tightens a bolt using a tool, the value of the worker's myoelectric potential is obtained using an electromyograph attached to the worker's arm.

[0053] In the next step S3, the operation identification unit 22 identifies the operation that the operator is about to perform from the image data acquired by the image acquisition unit 21 and the detection result of the detection device 12. For example, when a worker engaged in assembly work in a factory tightens a bolt using a tool, it is identified that the worker is about to tighten a bolt and which bolt the worker is about to tighten.

[0054] In the following step S4, the function of the operation determination unit 23 is used to determine whether the identified operation corresponds to a predetermined operation. If the identified operation corresponds to a predetermined operation, the process proceeds to step S5. On the other hand, if the identified operation does not correspond to a predetermined operation, the process proceeds to step S6. For example, in the case where a worker engaged in assembly work in a factory tightens a bolt using a tool, if tightening the bolt by the worker is a predetermined operation, the process proceeds to step S5. On the other hand, if the work of the worker does not include tightening a bolt, the process proceeds to step S6.

[0055] In step S5, the function of the operation procedure determination unit 24 determines whether the predetermined operation has been performed in the predetermined procedure. If it is determined that the predetermined operation has been performed in the predetermined procedure, execution of the routine is stopped and the monitoring process is terminated. On the other hand, if it is determined that the predetermined operation has not been performed in the predetermined procedure, the process proceeds to step S6. For example, in the case where a worker engaged in assembly work in a factory tightens a bolt using a tool, if the worker tightens the bolt in accordance with the predetermined procedure, execution of the routine is stopped and the monitoring process is terminated. On the other hand, if the worker leaves the bolt loose, the process proceeds to step S6.

[0056] In step S6, the function of the determination result output unit 25 outputs to the display device 13 that the identified operation was not a predetermined operation and that the predetermined operation was not performed properly according to a predetermined procedure. Then, the display device 13 receives the output of the determination result output unit 25 and notifies the operator and / or the monitor that the operation was not performed properly. For example, in the case where a worker engaged in assembly work in a factory tightens a bolt using a tool, the monitor is notified that the worker did not tighten the bolt properly. Then, after this display, the execution of the routine is stopped and the monitoring process is terminated.

[0057] As another example, when a worker engaged in assembly work in a factory is mating two couplers, for example, in step S1, image data is acquired by the imaging device 11 attached to the worker as a wearable terminal, and in step S2, sounds around the worker are detected by the operation identification unit 22. In the following step S3, based on the image data acquired by the image acquisition unit 21 and the detection result of the detection device 12, it is identified that the worker's operation is mating of the couplers. In the following step S4, the operation determination unit 23 determines whether the identified operation corresponds to a predetermined operation. If the identified operation, which is the mating operation of the couplers, is the predetermined operation, the process proceeds to step S5. On the other hand, if the work of the worker does not include mating of the couplers, the process proceeds to step S6. In step S5, the operation procedure determination unit 24 determines whether the predetermined operation was performed in the predetermined procedure. If it is determined that the coupler mating operation was performed in the predetermined procedure, the process stops execution of the routine and ends the monitoring process. On the other hand, if it is determined that the coupler fitting operation was not performed according to the predetermined procedure, the process proceeds to step S6. Then, in step S6, the function of the determination result output unit 25 outputs to the display device 13 that the specified coupler fitting operation was not the predetermined operation or that the coupler fitting operation was not performed appropriately according to the predetermined procedure.

[0058] [Embodiments of the present invention] As described above, the monitoring device 14 of this embodiment provides a monitoring device including: an image acquisition unit 21 that acquires an image including part or all of the operator's hand H and an operation object O operated by the operator using an imaging device 11 attached to the arm of the operator performing the operation; an operation determination unit 23 that determines whether the operation is a predetermined operation using the image acquired by the image acquisition unit 21; and a determination result output unit 25 that outputs the determination result of the operation determination unit 23. This makes it possible to reliably acquire an image of the operator operating the operation object O. Furthermore, it is possible to acquire an image suitable for determining the operator's behavior, thereby improving the accuracy of determining the operator's behavior.

[0059] Furthermore, according to the monitoring device 14 of this embodiment, the image acquisition unit 21 acquires an image including a part or all of the operator's palm and the operation object, thereby making it possible to more reliably acquire an image of the operator operating the operation object.

[0060] Furthermore, according to the monitoring device 14 of this embodiment, the imaging device 11 has an attachment portion for attachment to the palm side of the operator's upper arm A2, which makes it possible to more reliably obtain an image of the operator operating an operation object.

[0061] Furthermore, the monitoring device 14 of this embodiment includes an operation identification unit 22 that identifies the operation from the images acquired by the image acquisition unit 21, and the operation determination unit 23 determines whether the operation identified by the operation identification unit 22 is the predetermined operation from the series of images acquired by the image acquisition unit 21. This makes it possible to identify the operation more accurately.

[0062] Furthermore, according to the monitoring device 14 of this embodiment, the operation identification unit 22 identifies the operation using the time-series images acquired by the image acquisition unit 21 and the detection result of the detection device 12 that detects the state around the operator, thereby enabling more accurate identification of the operation.

[0063] Furthermore, according to the monitoring device 14 of this embodiment, when the operation determination unit 23 determines that the operation is not the predetermined operation, the determination result output unit 25 outputs to the display device 13 that the operation is inappropriate. This makes it possible to notify the operator and / or the monitor that the operation is inappropriate.

[0064] Furthermore, according to the monitoring device 14 of this embodiment, the operation identification unit 22 identifies the operation using a first trained model that has been trained to identify the operation from the image acquired by the image acquisition unit 21. This makes it possible to identify an operation for a positional relationship that has not been set in advance.

[0065] Furthermore, according to the monitoring device 14 of this embodiment, the first trained model is a first neural network 3 that, when first input data 34 including at least one of the image acquired by the image acquisition unit 21 and the detection result of the detection device 12 that detects the state around the operator is input to a first input layer 31, outputs first output data 35 including the identified operation from a first output layer 33, and the operation identification unit 22 includes a first machine learning unit 22a that trains the first neural network 3 using the first input data 34 previously input to the first neural network 3 and the first output data 35 previously output from the first neural network 3 as first teacher data 36, ​​and the first machine learning unit 22a trains the first neural network 3. This makes it possible to more accurately identify operations for positional relationships that were not previously set.

[0066] Furthermore, the monitoring device 14 of this embodiment includes an operation procedure determination unit 24 that, when the operation identification unit 22 determines that the operation is the predetermined operation, determines whether the predetermined operation has been performed in a predetermined procedure, and the determination result output unit 25 outputs the determination result of the operation procedure determination unit 24. This makes it possible to determine the appropriateness of the procedure for performing the operation.

[0067] Furthermore, according to the monitoring device 14 of this embodiment, the operation procedure determination unit 24 determines whether or not the predetermined operation has been performed in accordance with the predetermined procedure, using the time-series images acquired by the image acquisition unit 21. This makes it possible to more accurately determine whether or not the predetermined operation has been performed in accordance with the predetermined procedure.

[0068] Furthermore, according to the monitoring device 14 of this embodiment, when the operation procedure determination unit 24 determines that the predetermined operation has not been performed in accordance with the predetermined procedure, the determination result output unit 25 outputs to the display device 13 that the operation is inappropriate. This makes it possible to notify the operator and / or the monitor that the operation is inappropriate.

[0069] Furthermore, according to the monitoring device 14 of this embodiment, the operation procedure determination unit 24 determines whether the predetermined operation has been performed in the predetermined procedure using the time-series images acquired by the image acquisition unit 21 and the detection result of the detection device 12 that detects the state around the operator, using a second trained model that has been trained to determine whether the predetermined operation has been performed in the predetermined procedure. This makes it possible to determine the operation for a positional relationship that has not been set in advance and the appropriateness of the procedure.

[0070] Furthermore, according to the monitoring device 14 of this embodiment, the second trained model is a second neural network 4 that, when second input data 44 including the image acquired by the image acquisition unit 21 and the detection result of the detection device 12 that detects the state around the operator is input to the second input layer 41, outputs second output data 45 including a determination result as to whether the predetermined operation was performed according to the predetermined procedure from the second output layer 43. The operation procedure determination unit 24 includes a second machine learning unit 24a that trains the second neural network 4 using the second input data 44 previously input to the second neural network 4 and the second output data 45 previously output from the second neural network 4 as second teacher data 46, and the second machine learning unit 24a trains the second neural network 4. This makes it possible to more accurately determine the appropriateness of an operation for a positional relationship that has not been set in advance and the appropriateness of the procedure.

[0071] Furthermore, according to the monitoring device 14 of this embodiment, the operation determination unit 23 acquires the movement of the hand H of the operator from the image acquired by the image acquisition unit 21, determines whether the acquired movement corresponds to a predetermined movement, and if it is determined that the movement corresponds to the predetermined movement, determines whether the operation is the predetermined operation only from a time before a predetermined time from when it was determined that the movement of the hand H was the predetermined movement to a time after the predetermined time from when it was determined that the movement of the hand H was the predetermined movement. This makes it possible to reduce the power consumption of the monitoring device 14.

[0072] Furthermore, according to the monitoring device 14 of this embodiment, when it is determined that the movement does not correspond to a predetermined movement, the image acquisition unit 21 reduces the frame rate of the acquired time-series images, thereby reducing the amount of image data to be stored.

[0073] Furthermore, according to the monitoring method of this embodiment, in a monitoring method for monitoring an operator performing an operation using a processor, the processor acquires an image including all or part of the operator's hand H and an operation object O operated by the operator from an imaging device 11 attached to the operator's arm, determines whether the operation is a predetermined operation using the acquired image, and outputs a determination result as to whether the operation is the predetermined operation. This makes it possible to reliably acquire an image of the operator operating the operation object O. Furthermore, it is possible to acquire an image suitable for determining the operator's behavior, thereby improving the accuracy of determining the operator's behavior. [Explanation of symbols]

[0074] 1. Surveillance system 11...imaging device 11a...Mounting part 12...Detection device 13…Display device 14...Monitoring device 15...Processor 151...CPU 152...ROM 153...RAM 16...Database 2...Operation monitoring section 21...Image acquisition unit 22...Operation specification part 22a…1st Machine Learning Department 23...Operation judgment section 24...Operation procedure judgment section 24a…Second Machine Learning Department 25...Judgment result output unit 3...First neural network 31...First input layer 32...First middle class 33...First output layer 34...First input data 35...First output data 36...First teacher data 4...Second neural network 41...Second input layer 42...Second middle class 43...Second output layer 44...Second input data 45...Second output data 46...Second training data A1...Forearm A2: Upper arm C1, C2...Coupler H...hand O...Operation target S...Operating sound VA…Viewing angle Z...Mating part

Claims

1. an image acquisition unit that acquires an image including an operation target, which is an object on which the operator performs an input operation, and a part or all of the operator's hand that performs the input operation on the object, using an imaging device attached to the arm of the operator performing the operation; an operation determination unit that determines whether the operation identified from the positional relationship between the hand and the operation target in the image acquired by the image acquisition unit is a predetermined operation; a determination result output unit that outputs a determination result of the operation determination unit, The imaging device is a monitoring device having an attachment portion for attachment to the palm side of the operator's forearm.

2. an image acquisition unit that acquires an image including an operation target, which is an object on which the operator performs an input operation, and a part or all of the operator's hand that is in contact with the object, using an imaging device attached to the arm of the operator performing the operation; an operation determination unit that determines whether the operation is a predetermined operation by using the image acquired by the image acquisition unit; a determination result output unit that outputs a determination result of the operation determination unit, The imaging device is a monitoring device having an attachment portion for attachment to the palm side of the operator's forearm.

3. The monitoring device according to claim 1 , wherein the image acquisition unit acquires an image including a part or all of the palm of the operator and the operation target.

4. an operation identification unit that identifies the operation based on a positional relationship between the hand and the operation target in the image acquired by the image acquisition unit; The monitoring device according to any one of claims 1 to 3, wherein the operation determination unit determines whether the operation identified by the operation identification unit is the predetermined operation from the series of images acquired by the image acquisition unit.

5. The monitoring device according to claim 4 , wherein the operation identification unit identifies the operation using the time-series images acquired by the image acquisition unit and a detection result of a detection device that detects a state around the operator.

6. The monitoring device according to claim 4 , wherein the determination result output unit outputs, to a display device, a message indicating that the operation is inappropriate when the operation determination unit determines that the operation is not the predetermined operation.

7. The operation identification unit identifies the operation using a first trained model trained to identify the operation from the image acquired by the image acquisition unit. The monitoring device according to any one of claims 4 to 6.

8. the first trained model is a first neural network in which, when first input data including at least one of the image acquired by the image acquisition unit and a detection result of a detection device that detects a state around the operator is input to a first input layer, first output data including the identified operation is output from a first output layer; 8. The monitoring device of claim 7, wherein the operation identification unit includes a first machine learning unit that trains the first neural network using first input data that was previously input to the first neural network and first output data that was previously output from the first neural network as first teacher data, and trains the first neural network using the first machine learning unit.

9. an operation procedure determination unit that determines, when the operation identification unit determines that the operation is the predetermined operation, whether the predetermined operation has been performed in a predetermined procedure; The monitoring device according to any one of claims 4 to 8, wherein the determination result output unit outputs the determination result of the operating procedure determination unit.

10. The monitoring device according to claim 9 , wherein the operation procedure determination unit determines whether the predetermined operation has been performed in accordance with the predetermined procedure by using the time-series images acquired by the image acquisition unit.

11. The monitoring device according to claim 9 or 10, wherein the determination result output unit outputs, to a display device, a message indicating that the operation is inappropriate when the operation procedure determination unit determines that the specified operation has not been performed in accordance with the specified procedure.

12. The monitoring device according to any one of claims 9 to 11, wherein the operation procedure determination unit determines whether the predetermined operation has been performed in the predetermined procedure using a second trained model trained to determine whether the predetermined operation has been performed in the predetermined procedure by using the time-series images acquired by the image acquisition unit and the detection results of a detection device that detects the state around the operator.

13. the second trained model is a second neural network in which, when second input data including the image acquired by the image acquisition unit and a detection result of a detection device that detects a state around the operator is input to a second input layer, second output data including a determination result as to whether or not the predetermined operation has been performed in the predetermined procedure is output from a second output layer; The monitoring device described in claim 12, wherein the operation procedure determination unit includes a second machine learning unit that trains the second neural network using second input data that was previously input to the second neural network and second output data that was previously output from the second neural network as second teacher data, and trains the second neural network using the second machine learning unit.

14. The operation determination unit acquiring the hand movement of the operator from the image acquired by the image acquisition unit; determining whether the acquired movement corresponds to a predetermined movement; A monitoring device as described in any one of claims 1 to 13, wherein if it is determined that the movement corresponds to a predetermined movement, it determines whether the operation is the predetermined operation only from a time before a predetermined time from when the movement of the hand was determined to be a predetermined movement to a time after a predetermined time from when the movement of the hand was determined to be a predetermined movement.

15. The monitoring device according to claim 14 , wherein when it is determined that the movement does not correspond to a predetermined movement, the image acquisition unit reduces a frame rate of the acquired time-series images.

16. A monitoring method for monitoring an operator performing an operation using a processor, comprising: The processor: acquiring, from an imaging device having an attachment portion to be attached to the palm side of the forearm of the operator and attached to the arm of the operator, an image including an operation target that is an object on which the operator performs an input operation, and a part or all of the hand of the operator performing the input operation on the object; Identifying the operation based on a positional relationship between the hand and the operation target in the acquired image; determining whether the operation is a predetermined operation; and outputting a determination result as to whether the operation is the predetermined operation.

17. A monitoring method for monitoring an operator performing an operation using a processor, comprising: The processor: acquiring, from an imaging device having an attachment portion to be attached to the palm side of the forearm of the operator and attached to the arm of the operator, an image including an operation target that is an object on which the operator performs an input operation and a part or all of the operator's hand that is in contact with the object; Using the acquired image, it is determined whether the operation is a predetermined operation; and outputting a determination result as to whether the operation is the predetermined operation.

Images