Work recognition system and work recognition method
The work recognition system enhances accuracy by using separate models to identify and exclude non-work actions, improving the evaluation of work performance and instruction provision.
Patent Information
- Application Number
- JP2025078343
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2026-11-19
AI Technical Summary
Conventional work recognition systems incorrectly assign trained motion types to non-work actions, leading to reduced recognition accuracy.
A work recognition system using two machine learning models: one for estimating work-related actions and another for detecting non-work actions, with an exclusion mechanism to correct the output based on action type and duration.
Improves recognition accuracy by distinguishing between work and non-work actions, enabling better evaluation of work performance and providing appropriate instructions.
Smart Images

Figure 2026182258000001_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a work recognition system and a work recognition method.
Background Art
[0002] Conventionally, a technique is known that uses a trained machine learning model, which is trained by attaching correct labels representing the types of worker motions to moving images capturing workers, to estimate the type of worker motion from the moving images (Patent Document 1).
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] In conventional techniques, the type of worker motion is assigned from among the pre-trained types. Therefore, even when a worker performs motions unrelated to work, such as wiping sweat or picking up a fallen object, a trained motion type similar to these motions may be assigned.
Means for Solving the Problems
[0005] The present disclosure can be implemented in the following embodiments.
[0006] (1) According to one embodiment of the present disclosure, a work recognition system is provided. The work recognition system includes: an acquisition unit that acquires a video consisting of a plurality of images arranged in chronological order from a camera that captures work performed by a worker; an estimation unit that estimates the type of the worker's action in each image using a trained first machine learning model that, when the video is input, estimates and outputs the type of the worker's action represented by each image constituting the video from among the types of actions related to the work; a detection unit that detects the non-work actions using a trained second machine learning model that, when the video is input, detects and outputs the non-work actions that are not related to the work; and an exclusion unit that excludes the type assigned to the detected non-work actions from the output result of the first machine learning model. According to this embodiment, the work recognition system can detect non-work actions using a machine learning model different from the machine learning model used to estimate the type of work, and exclude the type assigned to the non-work actions from the estimation result of the type of action. This makes it possible to improve the recognition accuracy of the entire work. (2) In the above configuration, the output result of the second machine learning model includes the execution period of the non-work actions in the video, and the exclusion unit may exclude the type assigned to the non-work actions from the output result of the first machine learning model based on the execution period. According to this configuration, the work recognition system can easily exclude the type assigned to the non-work actions from the estimation result of the type of action based on the execution period of the non-work actions in the video. (3) In the above configuration, the camera may be a fixed-point camera installed at the work location where the worker performs the work and capable of outputting a video from a third-person perspective, which is the viewpoint of a third party. In this configuration, the work recognition system can detect actions outside of work by objectively capturing actions outside of work using the position and posture of the worker acquired by the fixed-point camera. Furthermore, since the worker can acquire video from the fixed-point camera without wearing a wearable camera, the work recognition system can reduce the burden on the worker. (4) In the above configuration, the camera may be a wearable camera attached to the worker and capable of outputting the video from a first-person perspective, which is the worker's viewpoint. In this configuration, the work recognition system can detect actions outside of work by subjectively capturing actions outside of work using the worker's viewpoint transitions acquired by the wearable camera. Furthermore, the work recognition system can estimate the type of action using video of the work target area, such as the worker's hands, captured in detail by the wearable camera. This improves the accuracy of estimating the type of action. Thus, the recognition accuracy for the entire work can be further improved. (5) Another embodiment of the present disclosure provides a work recognition method. The work recognition method comprises: an acquisition step of acquiring a video consisting of a plurality of images arranged in chronological order from a camera that captures work performed by a worker; an estimation step of estimating the type of the worker's action in each image using a trained first machine learning model that, when the video is input, estimates and outputs the type of the worker's action represented by each image constituting the video from among the types of actions related to the work; a detection step of detecting the non-work actions using a trained second machine learning model that, when the video is input, detects and outputs the non-work actions that are not related to the work; and an exclusion step of excluding the type assigned to the detected non-work actions from the output result of the first machine learning model. According to this embodiment, non-work actions can be detected using a machine learning model different from the machine learning model used to estimate the type of work, and the type assigned to the non-work actions can be excluded from the estimation result of the type of action. This makes it possible to improve the recognition accuracy of the entire work. This disclosure can be implemented in various forms other than the work recognition system and work recognition method described above. For example, it can be implemented in the form of a computer program that implements the work recognition method, a non-temporary recording medium on which the computer program is stored, etc. [Brief explanation of the drawing]
[0007] [Figure 1] An explanatory diagram showing the general configuration of the work recognition system. [Figure 2] A flowchart illustrating the method for recognizing tasks. [Figure 3] A diagram illustrating a concrete example of task recognition. [Modes for carrying out the invention]
[0008] A. First Embodiment: Figure 1 is an explanatory diagram showing the schematic configuration of the work recognition system 1. The work recognition system 1 recognizes the work performed by worker WK. By recognizing the work, it is possible to, for example, evaluate whether the work is being performed according to predetermined procedures, and to provide appropriate instructions to worker WK.
[0009] The work recognition system 1 is used in the work area WS where worker WK performs work. In this embodiment, the work area WS is a factory FC for manufacturing vehicles. In this embodiment, the work is a variety of tasks for manufacturing vehicles, including, for example, tasks related to vehicle assembly, tasks related to the installation of parts PT into the vehicle, and tasks related to vehicle inspection. These tasks include one or more actions to accomplish them.
[0010] In this disclosure, "recognizing work" means estimating the type of actions performed by worker WK during work and identifying the content of work performed by worker WK. "Type of action" is a classification of various actions performed by worker WK according to their purpose and characteristics. "Action" includes actions related to work and actions unrelated to work. "Actions related to work" include actions directly involved in the manufacture of vehicles, such as those classified by types such as "tightening," "fitting," and "inserting," and actions indirectly involved in the manufacture of vehicles, such as those classified by types such as "removal," "return," "movement," and "walking." "Tightening" means the action of attaching parts PT, such as speakers, to a vehicle by tightening fastening members FM, such as screws, using a tool TL. "Fitting" means the action of fitting fastening members FM, such as connectors, together. "Inserting" means the action of inserting fastening members FM, such as snap-fits, into fastened members. "Removal" means the action of taking various components, such as tools TL, parts PT, and fastening members FM, from a predetermined storage location. "Return" refers to the action of returning the tool TL to a predetermined storage location. "Move" refers to the action of moving a completed product to a subsequent process. "Walking" refers to the action of walking to another location, such as the place where the next product will be worked on. "Non-work actions" include actions classified by type, such as "wiping sweat," "picking up," and "adjusting." "Wiping sweat" refers to the action of wiping away sweat. "Picking up" refers to the action of picking up various parts that have fallen on the floor, etc. "Adjusting" refers to the action of adjusting the position of worn items such as helmets, work caps, gloves, and glasses. Note that the types of actions are not limited to those listed above.
[0011] The work recognition system 1 comprises one or more cameras 5 and a work recognition device 10. The work recognition device 10 recognizes work using video output from the cameras 5.
[0012] Camera 5 captures images of the worker WK performing their work and outputs a video consisting of multiple images arranged in chronological order. Camera 5 outputs, for example, two or more frames per second. Camera 5 may be a fixed-point camera 51 or a wearable camera 52. The fixed-point camera 51 is a camera 5 installed at the work area WS and captures images of the work area WS from an overhead perspective. By capturing images of the worker WK from the outside, the fixed-point camera 51 can output a video from a third-person perspective, which is the viewpoint of a third party viewing the worker WK. The wearable camera 52 is a camera 5 attached to the worker WK and is attached, for example, to the worker WK's head so that at least a portion of the imaging range overlaps with the worker WK's field of view. The wearable camera 52 may be attached to a head covering such as a helmet or work cap, or it may be integrated with eyewear such as goggles or glasses. The wearable camera 52 can capture images of an area corresponding to at least a portion of the worker WK's field of view, thereby outputting a first-person perspective video from the worker WK's point of view. The wearable camera 52 is also called an FPV (First Person View) camera or a first-person camera.
[0013] The work recognition device 10 is composed of a computer comprising a processor 110, a memory 120, an input / output interface 130, an internal bus 140, and a communication device 150. The processor 110, the memory 120, and the input / output interface 130 are connected via the internal bus 140 to enable bidirectional communication. The communication device 150 is connected to the input / output interface 130. The communication device 150 can communicate with the camera 5 via wired or wireless communication. The processor 110 executes a program PG stored in the memory 120 to realize various functions, including those of an acquisition unit 111, an estimation unit 112, a detection unit 113, and an exclusion unit 114.
[0014] The acquisition unit 111 acquires various types of information. For example, the acquisition unit 111 acquires video from camera 5.
[0015] The estimation unit 112 uses the recognition model RM to estimate the type of worker WK's action in each image that makes up the video. The recognition model RM is a pre-trained first machine learning model that, when a video is input, estimates and outputs the type of worker WK's action represented by each image that makes up the video from among the types of actions related to the work. The recognition model RM is prepared, for example, within or outside the work recognition system 1 and is pre-stored in the memory 120 of the work recognition device 10. As the recognition model RM, a machine learning model can be used that has been trained to select the type of worker WK's action represented by each image that makes up the video from among the types of actions related to the work. Specifically, as the recognition model RM, for example, a neural network such as a convolutional neural network (CNN) or a recurrent neural network (RNN) trained by supervised learning using a first training dataset can be used. In this first training dataset, training images of work performed by worker WK are associated with correct labels that represent the type of worker WK's action represented by the training image that is related to the work. In other embodiments, other machine learning models such as random forests or support vector machines (SVMs) may be used as the recognition model RM.
[0016] The detection unit 113 uses the detection model DM to detect non-work actions included in the video. The detection model DM is a pre-trained second machine learning model that detects and outputs non-work actions when a video is input. The detection model DM is prepared, for example, within or outside the work recognition system 1 and stored in the memory 120 of the work recognition device 10. As the detection model DM, a machine learning model can be used that has been trained to classify the actions of worker WK represented by each image constituting the video into work-related actions and non-work actions. Specifically, as the detection model DM, for example, a visual language model (VLM) trained using a second training dataset can be used. This second training dataset includes training images of various actions performed in daily life, such as wiping sweat, as non-work actions. In other embodiments, other machine learning models such as neural networks may be used as the detection model DM. Furthermore, the detection model DM only needs to be able to classify the actions of worker WK in each image that makes up the video into actions related to the work and actions unrelated to the work; it is not necessary to classify unrelated actions such as "wiping sweat," "picking up," and "fixing" by type.
[0017] The detection model DM may also be trained to identify and output the duration of the non-work action in the video when it detects such action. In other words, the output of the detection model DM may include the duration of the non-work action in the video. In this case, the duration of the non-work action may be expressed as a time relative to an arbitrary reference time, such as the elapsed time from the start of the video, or as a standard time (absolute time) defined for each country or region.
[0018] The exclusion unit 114 excludes the type assigned to the motion detected as a non-work motion by the detection unit 113 from the output result of the recognition model RM. For example, the exclusion unit 114 excludes the type assigned to the non-work motion from the output result of the detection model DM based on the execution period of the non-work motion in the moving image. The exclusion unit 114 may further cause a display device (not shown) to display a result obtained by excluding the type assigned to the non-work motion from the output result of the recognition model RM as the final recognition result of the work.
[0019] Figure 2 is a flowchart illustrating the work recognition method. The flowchart shown in Figure 2 is repeatedly executed at a predetermined time interval, for example, during a period in which the work by the worker WK is being performed. In the acquisition step of step S1, the acquisition unit 111 acquires a moving image from the camera 5. In the estimation step of step S2, the estimation unit 112 inputs the moving image acquired by the acquisition unit 111 into the recognition model RM, thereby estimating the type of motion of the worker WK in each image constituting the moving image. In the detection step of step S3, the detection unit 113 inputs the moving image acquired by the acquisition unit 111 into the detection model DM, thereby detecting a non-work motion included in the moving image. In the exclusion step of step S4, the exclusion unit 114 excludes the type assigned to the motion detected as a non-work motion by the detection unit 113 from the output result of the recognition model RM. Note that either the estimation step or the detection step may be executed first, or they may be executed in parallel at the same time.
[0020] Figure 3 is a diagram illustrating a specific example of work recognition. In Figure 3, the type of motion corresponding to the elapsed time from the start point of the moving image is illustrated for each step shown in Figure 2. Further, Figure 3 exemplifies a case where, in the acquisition step of step S1, the acquisition unit 111 acquires a moving image captured to show that a non-work motion of wiping sweat is performed during a series of operations from taking out the tool TL to tightening the fastening member FM. Note that the moving image acquired by the acquisition unit 111 in the acquisition step represents an actual motion that is a motion actually performed by the worker WK. Therefore, in Figure 3, the type of motion in each period of the acquisition step is illustrated as an actual motion.
[0021] In the example shown in FIG. 3, as indicated by the first graph G1 representing actual operations, in the first period P1 from 0 seconds, which is the start time of the moving image, to the time point at which 20 seconds have elapsed, "taking out", which is an operation related to the work, is performed. Further, in the third period P3 from the time point at which 30 seconds have elapsed from the start time of the moving image to the time point at which 70 seconds have elapsed, "tightening", which is an operation related to the work, is performed. The recognition model RM has learned these work-related operations in advance. Therefore, as shown in the second graph G2 representing the estimation result of the operation type, in the estimation step of step S2, the estimation unit 112 can use the recognition model RM to estimate the operation type in each period of the moving image from the feature amount of each image constituting the moving image as follows. The estimation unit 112 can estimate the operation type in the first period P1 as "taking out", and can estimate the operation type in the third period P3 as "tightening". In this way, among the operations of the worker WK, for the operations related to the work, the operation type can be appropriately estimated from the pre-learned types.
[0022] On the other hand, in the example shown in FIG. 3, as indicated by the first graph G1, in the second period P2 from the time point at which 20 seconds have elapsed from the start time of the moving image to the time point at which 30 seconds have elapsed, "wiping sweat", which is an operation unrelated to the work, is performed. The recognition model RM has not learned these non-work operations in advance. Therefore, as shown in the second graph G2, in the estimation step, the estimation unit 112 uses the recognition model RM to pre-estimate the operation type in the second period P2 as "fitting", which is a similar operation among the pre-learned types, from the feature amount of each image constituting the moving image. As described above, among the operations of the worker WK, since a similar operation type is assigned from the pre-learned types for operations unrelated to the work, the operation type may not be appropriately estimated in some cases.
[0023] Therefore, as shown in the fourth graph G4, which represents the final recognition result of the work, in the detection step of step S3, the detection unit 113 detects non-work actions included in the video in order to exclude the type assigned to non-work actions from the output result of the recognition model RM. As mentioned above, in the example shown in Figure 3, non-work actions are performed in the second period P2. The detection model DM has been trained to classify the actions of worker WK in each image constituting the video into actions related to work and actions unrelated to work. Therefore, as shown in the third graph G3, which represents the detection result of non-work actions, in the detection step, the detection unit 113 can use the detection model DM to detect non-work actions with the execution period EP set to the second period P2.
[0024] As a result, as shown in the fourth graph G4, in the exclusion step of step S4, the exclusion unit 114 can generate a series of recognition results for operations by excluding "insertion," which is the type assigned to the operation performed in the second period P2, from the output result of the recognition model RM.
[0025] According to the above embodiment, the work recognition system 1 can detect non-work actions using a machine learning model different from the machine learning model used to estimate the type of work, and exclude the type assigned to the non-work action from the estimation result of the type of action. With this configuration, the recognition accuracy of the entire work can be improved. As a result, for example, it is possible to more accurately evaluate whether the work is being performed according to a predetermined procedure, or to provide more appropriate instructions to the worker WK.
[0026] Furthermore, according to the above embodiment, the work recognition system 1 can easily exclude the type assigned to non-working actions from the estimation result of the type of action, based on the execution period EP of non-working actions in the video. In other embodiments, the work recognition system 1 may exclude the type assigned to non-working actions from the estimation result of the type of action by other methods.
[0027] Furthermore, according to the above embodiment, the work recognition system 1 can use video acquired from the fixed-point camera 51. In this configuration, the position and posture of the worker WK acquired by the fixed-point camera 51 can be used to objectively capture non-work movements, thereby detecting non-work movements. Moreover, in this configuration, the worker WK can acquire video from the fixed-point camera 51 without wearing a wearable camera 52. This reduces the burden on the worker WK.
[0028] Furthermore, according to the above embodiment, the work recognition system 1 can use video acquired from the wearable camera 52. In this configuration, by utilizing the transition of the worker WK's viewpoint acquired by the wearable camera 52, it is possible to subjectively capture actions outside of work and detect those actions. Moreover, in this configuration, the type of action can be estimated using video footage of the work target area, such as the worker WK's hands, captured in detail by the wearable camera 52. This improves the accuracy of estimating the type of action.
[0029] B. Other embodiments: (B1) The work recognition system 1 may be used in a work location WS other than the factory FC for manufacturing vehicles. When the work recognition system 1 recognizes work other than work for manufacturing vehicles, the term "vehicle" in this disclosure may be replaced with "product" or "work object" as appropriate.
[0030] (B2) In the above embodiments, some or all of the functions and processes implemented in software may be implemented in hardware. Also, some or all of the functions and processes implemented in hardware may be implemented in software. As hardware for implementing the various functions in each of the above embodiments, various circuits such as integrated circuits and discrete circuits may be used.
[0031] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features of the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of Symbols]
[0032] 1…Work recognition system, 5…Camera, 10…Work recognition device, 51…Fixed-point camera, 52…Wearable camera, 110…Processor, 111…Acquisition unit, 112…Estimation unit, 113…Detection unit, 114…Exclusion unit, 120…Memory, 130…Input / Output interface, 140…Internal bus, 150…Communication device, DM…Detection model, EP…Execution period, FC…Factory, FM…Fastening member, G1…First graph, G2…Second graph, G3…Third graph, G4…Fourth graph, P1…First period, P2…Second period, P3…Third period, PG…Program, PT…Part, RM…Recognition model, TL…Tool, WK…Worker, WS…Workplace
Claims
1. A work recognition system, An acquisition unit that acquires a video consisting of multiple images arranged in chronological order from a camera that films the work performed by the worker, An estimation unit that estimates the type of the worker's action in each image, using a trained first machine learning model that, when the aforementioned video is input, estimates and outputs the type of the worker's action represented by each image constituting the video from among the types of actions related to the aforementioned work, A detection unit for detecting the aforementioned unrelated actions, using a trained second machine learning model that detects and outputs the aforementioned actions unrelated to the work when the aforementioned video is input, A work recognition system comprising: an exclusion unit that excludes the type assigned to the operation detected as an operation outside of the work from the output result of the first machine learning model.
2. A work recognition system according to claim 1, The output result of the second machine learning model includes the execution period of the actions outside of the work in the video, The exclusion unit is a work recognition system that, based on the execution period, excludes the type assigned to the operation outside of the work from the output result of the first machine learning model.
3. A work recognition system according to claim 1, The camera is a fixed-point camera installed at the work location where the worker performs the work, and is capable of outputting video from a third-person perspective, which is the viewpoint of a third party observing the worker, in the work recognition system.
4. A work recognition system according to claim 1, The camera is a wearable camera attached to the worker and capable of outputting a video from a first-person perspective, which is the worker's viewpoint, in the work recognition system.
5. A method for recognizing work, The acquisition process involves obtaining a video consisting of multiple images arranged in chronological order from a camera that captures the work performed by the worker, An estimation step is performed to estimate the type of the worker's action in each image, using a trained first machine learning model that, when the aforementioned video is input, estimates and outputs the type of the worker's action represented by each image constituting the video from among the types of actions related to the aforementioned work, A detection step for detecting the aforementioned unrelated actions when the aforementioned video is input, using a trained second machine learning model that detects and outputs the aforementioned actions unrelated to the work, A work recognition method comprising: an exclusion step of excluding the type assigned to the operation detected as an operation outside of the work from the output result of the first machine learning model.
Citation Information
Patent Citations
Behavior recognition learning device, behavior recognition device, behavior recognition learning method, behavior recognition method, and behavior recognition learning program
JP2022072444A
