Action sequence abnormality detection device, method, and program

Abstract

A video input means 81 is for inputting a work video in which a work process by a worker has been captured. A behavior identification means 82 identifies behaviors of the worker from the inputted work video. A behavior order identification means 83 traces the identified behaviors of the worker in a time series and, in a case where a change in behavior of the worker has been detected, adds a behavior before the change to a behavior order list. A behavior order comparison means 84 makes a comparison between the behavior order list and an ideal work procedure that defines the order of behaviors in accordance with the work, so as to detect abnormal work.

Description

Technical Field

[0001] The present invention relates to an action sequence abnormality detection device, an action sequence abnormality detection method, and an action sequence abnormality detection program for detecting an abnormality in the order of actions performed by an operator.

Background Art

[0002] In the manufacturing industry, improving productivity in factories and warehouses where products are assembled, sorted, and stored is an issue. As an effort to improve productivity, the work procedure is confirmed based on the judgment by a supervisor in real time or by visually checking the recorded work scene of an operator. However, the judgment by visual confirmation by a supervisor not only incurs costs such as time and labor costs, but there is also a possibility of overlooking something. Therefore, there is a desire to automatically identify whether the work procedure by an operator is being performed in the correct order.

[0003] As a technology related to such a desire, Patent Document 1 describes a work management device that outputs the implementation status of work by an operator based on an image of the operator. The device described in Patent Document 1 acquires a first image showing the operator at the start of work and a second image showing the operator at the end of work. Then, the device measures the required work time by performing image analysis on the first image and the second image, and recognizes the work item, thereby displaying information regarding the implementation status for each work item.

[0004] Also, Non-Patent Document 1 describes the MS-TCN (Multi-Stage Temporal Convolutional Network), which is a technology for action segmentation, as a technology for identifying work content from an image. For example, in order to classify action segments in a long video that has not been trimmed, the MS-TCN described in Non-Patent Document 1 directly classifies video frames by generating a probability for each frame and providing it to a high-level time model.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Document

[0006]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] As described above, the device described in Patent Document 1 measures the start time and end time of work related to a target worker using the video captured by an installed camera, and identifies the work content. However, while the device described in Patent Document 1 can automatically identify the work content and the required time of the worker from the video, it does not have a function to determine whether the work procedure was performed in the correct order. Therefore, there is a problem that the quality of the work performed cannot be grasped.

[0008] Also, the method described in Non-Patent Document 1 compares the result output by the machine learning model for each frame with the correct action content of that frame and outputs the accuracy. Therefore, for example, there is a problem that the correctness of the work content of the worker described above cannot be evaluated simply by applying this technology to a video captured in a factory.

[0009] Therefore, an object of the present invention is to apply an action sequence abnormality detection device, an action sequence abnormality detection method, and an action sequence abnormality detection program that can detect an abnormality in the action sequence by a worker.

Means for Solving the Problem

[0010] The action sequence abnormality detection device according to the present invention includes a video input means for inputting a work video obtained by photographing the work process of an operator, an action identification means for identifying the actions of the operator from the input work video, and tracing the identified actions of the operator in time series. When a change in the actions of the operator is detected, an action sequence identification means for adding the actions before the change to an action sequence list, and comparing an ideal work procedure that defines the order of actions according to the work with the action sequence list , calculate the edit distance indicating the minimum number of times to edit the order of actions included in the action order list until it matches the actions included in the ideal work procedure , Based on the calculated edit distance and an action sequence comparison means for detecting an abnormal work, and is characterized by this.

[0011] The action sequence abnormality detection method according to the present invention the computer inputs a work video obtained by photographing the work process of an operator, the computer identifies the actions of the operator from the input work video, the computer traces the identified actions of the operator in time series. When a change in the actions of the operator is detected, the actions before the change are added to an action sequence list, the computer compares an ideal work procedure that defines the order of actions according to the work with the action sequence list , calculate the edit distance indicating the minimum number of times to edit the order of actions included in the action order list until it matches the actions included in the ideal work procedure , Based on the calculated edit distance and is characterized by detecting an abnormal work.

[0012] The action sequence abnormality detection program according to the present invention causes a computer to execute a video input process for inputting a work video obtained by photographing the work process of an operator, an action identification process for identifying the actions of the operator from the input work video, an action sequence identification process for tracing the identified actions of the operator in time series and adding the actions before the change to an action sequence list when a change in the actions of the operator is detected, and an action sequence comparison process for comparing an ideal work procedure that defines the order of actions according to the work with the action sequence list , calculate the edit distance indicating the minimum number of times to edit the order of actions included in the action order list until it matches the actions included in the ideal work procedure , Based on the calculated edit distance and detecting an abnormal work, and is characterized by this.

Advantages of the Invention

[0013] According to the present invention, it is possible to detect an abnormality in the action sequence by an operator.

Brief Explanation of Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Modes for Carrying Out the Invention

[0015] Embodiments of the present invention will be described in detail below with reference to the drawings. In the following description, a series of operations by an operator are referred to as work, and individual operations included in the work are referred to as actions.

[0016] [First Embodiment] [Explanation of Configuration] FIG. 1 is a block diagram showing a configuration example of an action sequence abnormality detection system according to a first embodiment of the present invention. As shown in FIG. 1, the action sequence abnormality detection system 100 according to the first embodiment of the present invention includes a video input means 1, an action sequence abnormality detection device 21, an ideal work procedure storage device 22, and an output device 3. The action sequence abnormality detection device 21 and the ideal work procedure storage device 22 are communicably connected.

[0017] As shown in FIG. 1, the action sequence abnormality detection device 21 includes an action identification unit 211, an action sequence identification unit 212, and an action sequence comparison unit 213. The action sequence identification unit 212 is connected to the action identification unit 211 and the action sequence comparison unit 213, respectively. Note that the one-way arrows shown in FIG. 1 simply indicate the direction of information flow and do not exclude bidirectionality.

[0018] The video input unit 1 inputs a video (hereinafter referred to as a work video) of the work process performed by the operator. The content of the work video is arbitrary. In order to detect an abnormality from a series of actions of the operator, the work video is preferably a video that captures the entire work in time series. Note that the video input unit 1 may be included in the action sequence abnormality detection device 21.

[0019] The action identification unit 211 identifies the actions of the operator from the input work video. Specifically, the action identification unit 211 identifies the actions of the operator in frame units from the work video. The method by which the action identification unit 211 identifies actions from the work video is arbitrary, and for example, the method described in Patent Document 1 may be used.

[0020] The action sequence identification unit 212 receives the identification result from the action identification unit 211, traces the identified actions of the operator in time series, and when detecting a change in the actions of the operator, adds the action before the change to the action sequence list. Specifically, the action sequence identification unit 212 traces the identified actions of the operator in time series from the first frame of the work video, and when detecting a change in actions before and after the frame, adds the action before the change to the action sequence list.

[0021] Note that when identifying in frame units, there may be noise or incorrect identification results. Therefore, the action sequence identification means 212 may detect a change in the action when the identification result of the operator's action continues with the same content for a predetermined period (number of frames). In this way, while the identification result by the action identification means 211 is data indicating the identification result in frame units, it can be said that the action sequence list is data in which the identification results are compressed in action units.

[0022] By using the action sequence list generated in this way, it becomes possible to facilitate the process in which the action sequence comparison means 213 described later compares with the ideal work procedure.

[0023] The ideal work procedure storage device 22 is a device that records the ideal action sequence (hereinafter referred to as the ideal work procedure) assumed in the work performed by the operator. Specifically, the ideal work procedure is information that defines the order of actions according to the type of work. In the target assembly work or transportation work, it is an example of work when the work is performed in the correct order as assumed by the supervisor without any extra work intervening. The ideal work procedure may be represented as information in frame units, or may be represented in the same format as the action sequence list described later. The ideal work procedure storage device 22 stores the ideal work procedure for each type of work. The ideal work procedure storage device 22 is realized by, for example, a magnetic disk or the like.

[0024] The action sequence comparison means 213 receives the action sequence list from the action sequence identification means 212, compares it with the ideal work procedure stored in the ideal work procedure storage device 22, and detects abnormal work. Further, the action sequence comparison means 213 may output the result of detecting the abnormal work to the output device 3 based on the edit distance between the action sequence list and the ideal work procedure. In this embodiment, it is assumed that the type of work is known.

[0025] The evaluation index called edit distance will be described below. In the evaluation using the edit distance, for the identification results of a series of actions, it is measured how many times replacements, deletions, and insertions of the element contents are used for each element (here, actions) included in the identification results to match the correct action contents.

[0026] Therefore, the action order comparison means 213 may output the result of detecting an abnormal operation by calculating the edit distance from a series of actions included in the action order list to a series of actions included in the ideal operation procedure until they are transformed.

[0027] Specifically, the action order comparison means 213 may output the edit distance indicating the number of action replacements as the number of action swaps. Also, the action order comparison means 213 may output the edit distance indicating the number of action additions as the number of action omissions. Also, the action order comparison means 213 may output the edit distance indicating the number of action deletions as the number of additional extra actions.

[0028] The action identification means 211, the action order identification means 212, and the action order comparison means 213 are realized by a processor (for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit)) of a computer that operates according to a program (action order abnormality detection program).

[0029] For example, the program is stored in a storage unit (not shown) provided in the action order abnormality detection device 21, and the processor may read the program and operate as the action identification means 211, the action order identification means 212, and the action order comparison means 213 according to the program. Also, the functions of the action order abnormality detection device 21 may be provided in the form of SaaS (Software as a Service).

[0030] Further, the action identification means 211, the action sequence identification means 212, and the action sequence comparison means 213 may each be realized by dedicated hardware. Also, part or all of each component of each device may be realized by general-purpose or dedicated circuitry, a processor, etc., or a combination thereof. These may be constituted by a single chip, or may be constituted by a plurality of chips connected via a bus. Part or all of each component of each device may be realized by a combination of the above-described circuitry, etc. and a program.

[0031] Also, when part or all of each component of the action sequence abnormality detection device 21 is realized by a plurality of information processing devices, circuitry, etc., the plurality of information processing devices, circuitry, etc. may be centrally arranged or may be distributed. For example, the information processing devices, circuitry, etc. may be realized in a form in which each is connected via a communication network, such as a client-server system, a cloud computing system, etc.

[0032] [Description of Operations] Next, the operation of the action sequence abnormality detection system 100 of the present embodiment will be described. FIG. 2 is a flowchart showing an operation example of the action sequence abnormality detection system 100 of the first embodiment.

[0033] First, the video input means 1 inputs a work video (step S1). Next, the action identification means 211 identifies the actions of the worker from the input work video (step S2). Specifically, the action identification means 211 identifies the action for each frame of the input work video. Next, when the action sequence identification means 212 traces the identified actions of the worker in time series and detects a change in the actions of the worker, the action before the change is added to the action sequence list (step S3). In other words, the action sequence identification means 212 converts the content of the action for each obtained frame by deleting the frame information into an action sequence list representing which actions were performed in which order.

[0034] FIG. 3 is an explanatory diagram showing an example of a process for generating an action sequence list from work images. Hereinafter, a specific example of a method (conversion method) for generating an action sequence list will be described. First, the action sequence identification means 212 compares the work content of the first frame of the work image (here, action A) with the action of the next frame. When the actions shown are the same, the action sequence identification means 212 performs a process of comparing that action with the action shown in the next further frame. Thereafter, this process is repeated until a frame showing a different action (here, action B) is reached.

[0035] When a frame showing a different action is reached, the action sequence identification means 212 adds the action before the change to the action sequence list. For subsequent frames, the same process is repeated until a frame showing a further different action is reached, and each time a frame showing a different action is reached, the actions up to that point are added to the action sequence list. In the process, if it is determined that the action shown in a frame is unidentifiable, the action identification means 211 may simply ignore that frame and continue the process.

[0036] For example, in the example shown in FIG. 3, it shows that the action sequence list L4 is generated from the identification result L2. Note that an action sequence list L3 representing an ideal work procedure may be generated from the identification result L1 of an image showing an ideal work in a similar manner.

[0037] Next, in FIG. 2, the action sequence comparison means 213 compares the ideal work procedure and the action sequence list to detect an abnormal work (step S4). Specifically, the action sequence comparison means 213 compares the action sequence list generated in step S3 with the ideal work procedure stored in the ideal work procedure storage device 22 by calculating an edit distance, and detects differences such as a swap of procedures.

[0038] Hereinafter, a specific example of the comparison process will be described with reference to FIG. 3. The edit distance is used to compare two action sequence lists, namely, the action sequence list L4 generated from the work video and the ideal action sequence list L3. The edit distance is obtained by calculating how many times "element replacement", "element deletion", and "element insertion" need to be performed on the content of the action sequence list L4 to obtain the same list as the action sequence list L3.

[0039] For example, assume that the action sequence list L4 generated from the work video is [1: Action A, 2: Action B, 3: Action C, 4: Action B, 5: Action D], and the ideal action sequence list L3 is [1: Action A, 2: Action B, 3: Action D, 4: Action C]. At this time, by deleting the fourth element, replacing the third element, and replacing the fifth element in the action sequence list L4, an action sequence list that matches the action sequence list L3 can be obtained.

[0040] In this case, since the two action sequence lists match after three operations, the edit distance is 3. When calculating the edit distance, the action sequence comparison means 213 regards the number of "element deletions" as the number of "additional actions added", the number of "element insertions" as the number of "missing actions", and the number of "element replacements" (for every two items) as the number of "action swaps" (for one item). Then, if an abnormality in the regarded content is detected, the action sequence comparison means 213 outputs the result of detecting the abnormal operation.

[0041] For example, in the example shown in FIG. 3, deletion is performed once and replacement is performed twice. Therefore, the action sequence comparison means 213 outputs a result assuming that one "additional action added" and one "action swap" have occurred.

[0042] After that, in FIG. 2, the action sequence comparison means 213 outputs the result of detecting the abnormal operation to the output device 3 (step S5).

[0043] As described above, in this embodiment, the video input means 1 inputs the work video, and the action identification means 211 identifies the actions of the worker from the input work video. Then, when the action sequence identification means 212 traces the identified actions of the worker in time series and detects a change in the actions of the worker, the action before the change is added to the action sequence list, and the action sequence comparison means 213 compares the ideal work procedure with the action sequence list to detect an abnormal work. Therefore, it is possible to detect an abnormality in the action sequence by the worker.

[0044] That is, by using the action sequence abnormality detection system 100 of this embodiment, it is possible to identify whether the actions of the worker in the work video are performed in the assumed correct procedure. The reason is that the action sequence identification means 212 generates an action sequence list indicating which actions are performed in which order from the identification results of the actions for each frame of the input work video.

[0045] For example, in the example shown in FIG. 3, the action sequence identification means 212 converts the identification result L2 of the work video identified by the action identification means 211 into the action sequence list L4. Similarly, the action sequence identification means 212 converts the identification result L1 into the action sequence list L3. Thereby, the action sequence abnormality detection system 100 can detect the replacement of actions and the omission of actions by comparing the edit distances of the two action sequence lists L3 and L4.

[0046] [Second Embodiment] [Description of Configuration] Next, a second embodiment of the action sequence abnormality detection system according to the present invention will be described. FIG. 4 is a block diagram showing a configuration example of the action sequence abnormality detection system according to the second embodiment of the present invention. The action sequence abnormality detection system 200 according to the second embodiment of the present invention includes, as shown in FIG. 4, a video input means 1, an action sequence abnormality detection device 21, an ideal work procedure storage device 22, a work type search means 23, and an output device 3.

[0047] That is, the action sequence abnormality detection system 200 of the second embodiment differs in that it further includes a work type search means 23 as compared with the action sequence abnormality detection system 100 of the first embodiment. Other configurations are the same as those of the first embodiment. Note that the work type search means 23 may be included in the action sequence abnormality detection device 21.

[0048] Based on the actions of the worker identified from the work video, the work type search means 23 identifies the type of work that the worker is performing. Then, the work type search means 23 searches for and acquires an ideal work procedure corresponding to the identified work type from the ideal work procedure storage device 22.

[0049] The method by which the work type search means 23 identifies the work type based on the actions of the worker is arbitrary. For example, the work type search means 23 may prepare, as a master, typical work types corresponding to the actions of the worker, and identify the work type according to the degree of match with the master. Also, the work type search means 23 may identify the work type using a model generated by learning the work type based on the actions of the worker.

[0050] In this embodiment, since the action sequence abnormality detection system 200 includes the work type search means 23, it is possible to detect an abnormality in the action sequence using the same device for a plurality of types of work.

[0051] Hereinafter, the operation of the work type search means 23 will be described using a specific example. For example, assume that there are three types of work: "product assembly", "product packaging", and "product defect inspection". In the first embodiment, it was assumed that the type of work was known. That is, in advance, it was clear which type of work the work video belonged to among the three types, and the ideal work procedure corresponding to that work type was acquired.

[0052] On the other hand, in the second embodiment, the work type search means 23 acquires an appropriate ideal work procedure from the above three types of ideal work procedures stored in the ideal work procedure storage device 22 according to the type of work specified. Thereby, it is possible to perform abnormality detection regarding a plurality of types of work with the same device.

[0053] Note that the action identification means 211, the action order identification means 212, the action order comparison means 213, and the work type search means 23 are realized by a processor of a computer that operates according to a program (action order abnormality detection program).

[0054] [Explanation of Operations] Next, the operation of the action order abnormality detection system 200 of the present embodiment will be described. FIG. 5 is a flowchart showing an operation example of the action order abnormality detection system 200 of the second embodiment. The processing from step S1 to step S2 illustrated in FIG. 2 is the same as the processing until the action is identified by inputting the work video.

[0055] Next, the work type search means 23 specifies the type of work that the worker is performing based on the actions of the worker identified from the work video (step S11). Then, the work type search means 23 searches for and acquires an ideal work procedure corresponding to the specified work type from the ideal work procedure storage device 22 (step S12). Using this acquired ideal work procedure, a comparison with the action order list is performed.

[0056] Thereafter, the processing from step S3 to step S5 illustrated in FIG. 2 is performed. Note that the processing of step S3 for generating the action order list may be performed before the processing of step S11 or step S12.

[0057] As described above, in the present embodiment, the work type search means 23 specifies the type of work based on the actions of the worker identified from the work video, and acquires an ideal work procedure corresponding to the specified work type from the ideal work procedure storage device 22. Therefore, in addition to the effects of the first embodiment, it is possible to perform abnormality detection regarding a plurality of types of work with the same device.

[0058] [Embodiment] Next, specific examples will be described for the scenarios where the action sequence anomaly detection system of each embodiment is applied. The action sequence anomaly detection system of the above embodiment can be used, for example, in an actual factory. When a fixed-point camera installed in the factory captures the working process of the operator, the video input means 1 inputs the captured working video.

[0059] Next, the action identification means 211 identifies which action among the pre-determined actions, such as "taking a specific part" or "assembling a part", each frame of the captured working video corresponds to. Next, the action sequence identification means 212 converts the identification result into an action sequence list.

[0060] Finally, the action sequence comparison means 213 compares the action sequence list with the ideal working procedure, and outputs how many occurrences of action swapping, action omission, or addition of extra work have occurred in the work to be identified.

[0061] Next, the outline of the present invention will be described. FIG. 6 is a block diagram showing the outline of the action sequence anomaly detection device according to the present invention. The action sequence anomaly detection device 80 (for example, the action sequence anomaly detection device 21) according to the present invention includes a video input means 81 (for example, the video input means 1) that inputs a working video capturing the working process by the operator, an action identification means 82 (for example, the action identification means 211) that identifies the actions of the operator from the input working video, an action sequence identification means 83 (for example, the action sequence identification means 212) that traces the identified actions of the operator in time series and adds the actions before the detected change to the action sequence list when detecting a change in the actions of the operator, and an action sequence comparison means 84 (for example, the action sequence comparison means 213) that compares the ideal working procedure defining the order of actions according to the work with the action sequence list to detect abnormal work.

[0062] With such a configuration, it is possible to detect anomalies in the action sequence by the operator.

[0063] Specifically, the action identification means 82 identifies the actions of the worker in frame units from the work video, and the action sequence identification means 83 traces the identified actions of the worker in time series from the first frame of the work video. When a change in the action is detected before and after the frame, the action before the change may be added to the action sequence list.

[0064] Further, the action sequence comparison means 84 may output the result of detecting an abnormal operation based on the edit distance between the action sequence list and the ideal work procedure.

[0065] Specifically, the action sequence comparison means 84 may output the result of detecting an abnormal operation by calculating the edit distance from a series of actions included in the action sequence list to a series of actions included in the ideal work procedure.

[0066] At that time, the action sequence comparison means 84 may output the edit distance indicating the number of action replacements as the number of action swaps, output the edit distance indicating the number of action additions as the number of action omissions, and output the edit distance indicating the number of action deletions as the number of additional extra actions.

[0067] Further, the action sequence abnormality detection device 80 may include a work type search means (for example, the work type search means 23) that identifies the type of work being performed by the worker based on the actions of the worker identified from the work video and acquires the ideal work procedure corresponding to the identified work type from a storage device (for example, the ideal work procedure storage device 22).

[0068] FIG. 7 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. The computer 1000 includes a processor 1001, a main storage device 1002, an auxiliary storage device 1003, and an interface 1004.

[0069] The above-described action sequence abnormality detection device 80 is implemented in the computer 1000, respectively. The operations of the above-described respective processing units are stored in the auxiliary storage device 1003 in the form of a program. The processor 1001 reads the program from the auxiliary storage device 1003, expands it in the main storage device 1002, and executes the above processing according to the program.

[0070] In at least one embodiment, the auxiliary storage device 1003 is an example of a non-transitory tangible medium. Other examples of non-transitory tangible media include magnetic disks, magneto-optical disks, CD-ROMs (Compact Disc Read-only memories), DVD-ROMs (Read-only memories), semiconductor memories, etc. connected via the interface 1004. Further, when this program is distributed to the computer 1000 via a communication line, the computer 1000 that has received the distribution may expand the program in the main storage device 1002 and execute the above processing.

[0071] Also, the program may be for realizing a part of the above-described functions. Furthermore, the program may be a so-called difference file (difference program) that realizes the above-described functions in combination with other programs already stored in the auxiliary storage device 1003.

[0072] Some or all of the above embodiments may be described as follows in the following supplementary notes, but are not limited thereto.

[0073] (Supplementary Note 1) An image input means for inputting a work video obtained by photographing the work process of a worker, An action identification means for identifying the actions of the worker from the input work video, An action sequence identification means for tracing the identified actions of the worker in time series and adding the actions before the detected change in the actions of the worker to an action sequence list when a change in the actions of the worker is detected, An action sequence comparison means for comparing an ideal work procedure that defines the order of actions according to the work with the action sequence list to detect abnormal work, and An action sequence abnormality detection device characterized by the following.

[0074] (Appendix 2) The action identification means identifies the actions of the worker in frame units from the work video, The action sequence identification means traces the identified actions of the worker in time series from the first frame of the work video, and when detecting a change in actions before and after the frame, adds the action before the change to the action sequence list. The action sequence abnormality detection device according to Appendix 1.

[0075] (Appendix 3) The action sequence comparison means outputs the result of detecting an abnormal operation based on the edit distance between the action sequence list and the ideal work procedure. The action sequence abnormality detection device according to Appendix 1 or Appendix 2.

[0076] (Appendix 4) The action sequence comparison means outputs the result of detecting an abnormal operation by calculating the edit distance from a series of actions included in the action sequence list to a series of actions included in the ideal work procedure. The action sequence abnormality detection device according to Appendix 3.

[0077] (Appendix 5) The action sequence comparison means outputs the edit distance indicating the number of action replacements as the number of action swaps, outputs the edit distance indicating the number of action additions as the number of action omissions, and outputs the edit distance indicating the number of action deletions as the number of additional extra actions. The action sequence abnormality detection device according to Appendix 3 or Appendix 4.

[0078] (Appendix 6) It is provided with work type search means for specifying the type of work being performed by the worker based on the actions of the worker identified from the work video, and acquiring the ideal work procedure corresponding to the specified work type from the storage device. The action sequence abnormality detection device according to any one of Appendices 1 to 5.

[0079] (Appendix 7) Input a work video that captures the work process of the worker, Identify the actions of the worker from the input work video, When tracing the identified actions of the operator in chronological order and detecting a change in the actions of the operator, add the actions before the change to the action sequence list. Compare the ideal work procedure that defines the order of actions according to the work with the action sequence list to detect abnormal work. An action sequence abnormality detection method characterized by the above.

[0080] (Appendix 8) Identify the actions of the operator in units of frames from the work video. When tracing the identified actions of the operator in chronological order from the first frame of the work video and detecting a change in the actions before and after the frame, add the actions before the change to the action sequence list. The action sequence abnormality detection method described in Appendix 7.

[0081] (Appendix 9) On the computer, A video input process for inputting a work video that captures the work process by the operator. An action identification process for identifying the actions of the operator from the input work video. An action sequence identification process for tracing the identified actions of the operator in chronological order and adding the actions before the change to the action sequence list when detecting a change in the actions of the operator, and An action sequence comparison process for comparing the ideal work procedure that defines the order of actions according to the work with the action sequence list to detect abnormal work. A program storage medium that stores an action sequence abnormality detection program for executing the above.

[0082] (Appendix 10) On the computer, In the action identification process, identify the actions of the operator in units of frames from the work video. In the action sequence identification process, make the identified actions of the operator be traced in chronological order from the first frame of the work video, and when detecting a change in the actions before and after the frame, add the actions before the change to the action sequence list. The program storage medium described in Appendix 9 that stores an action sequence abnormality detection program for the above.

[0083] (Appendix 11) Have the computer perform video input processing to input work videos that capture the work process of the worker, perform action identification processing to identify the actions of the worker from the input work videos, perform action sequence identification processing to trace the identified actions of the worker in chronological order and add the actions before the change to the action sequence list when a change in the actions of the worker is detected, and perform action sequence comparison processing to compare the ideal work procedure that defines the order of actions according to the work with the action sequence list and detect abnormal work An action sequence abnormality detection program for causing the above to be executed.

[0084] (Appendix 12) Have the computer perform action identification processing to identify the actions of the worker frame by frame from the work videos, perform action sequence identification processing to trace the identified actions of the worker in chronological order from the first frame of the work videos and add the actions before the change to the action sequence list when a change in the actions is detected before and after the frame The action sequence abnormality detection program according to Appendix 11.

[0085] The present invention has been described with reference to the embodiments and examples, but the present invention is not limited to the above embodiments and examples. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

Industrial Applicability

[0086] The present invention is preferably applied to an action sequence abnormality detection device that detects abnormalities in the order of actions by a worker. Specifically, regarding the assembly work process in the manufacturing industry, the present invention can be used for analyzing the work content, such as which work process tends to take a long time and which mistakes are likely to occur between work processes. In addition, the present invention can also be applied to uses such as confirming whether important processes leading to the quality of products are carried out without excess or deficiency in the work management of the construction industry.

Explanation of Signs

[0087] 1 Image input means 3 Output device 21 Action sequence abnormality detection device 22 Ideal operation procedure storage device 100, 200 Action sequence abnormality detection system 211 Action identification means 212 Action sequence identification means 213 Action sequence comparison means 23 Work type search means

Claims

1. video input means for inputting work videos taken by an operator; action identification means for identifying the actions of the operator from the input work videos; action sequence identification means for tracing the identified actions of the operator in chronological order and adding the actions before the change to the action sequence list when a change in the actions of the operator is detected; action sequence comparison means for comparing the ideal work procedure that defines the order of actions according to the work with the action sequence list, calculating the edit distance indicating the minimum number of times to edit the appearance order of the actions included in the action sequence list until they match the actions included in the ideal work procedure, and detecting abnormal work based on the calculated edit distance; An action sequence abnormality detection device characterized by the above.

2. The action identification means identifies the actions of the operator in frame units from the work videos, The action sequence identification means traces the identified actions of the operator in chronological order from the first frame of the work video, and adds the actions before the change to the action sequence list when a change in the actions is detected before and after the frame. The action sequence abnormality detection device according to Claim 1.

3. The action sequence comparison means outputs the result of detecting abnormal work based on the edit distance. The action sequence abnormality detection device according to Claim 1 or Claim 2.

4. The action sequence comparison means outputs the result of detecting abnormal work by calculating the edit distance from a series of actions included in the action sequence list to a series of actions included in the ideal work procedure. The action sequence abnormality detection device according to Claim 3.

5. The action sequence comparison means outputs the edit distance indicating the number of replacements of actions as the number of swaps of actions, outputs the edit distance indicating the number of additions of actions as the number of omissions of actions, and outputs the edit distance indicating the number of deletions of actions as the number of additions of extra actions. The action sequence abnormality detection device according to claim 3 or claim 4.

6. Comprising a work type search means for identifying the type of work being performed by the worker based on the actions of the worker identified from the work video, and acquiring from the storage device an ideal work procedure corresponding to the identified work type The action sequence abnormality detection device according to any one of claims 1 to 5.

7. A computer inputs a work video that captures the work process of a worker, The computer identifies the actions of the worker from the input work video, When the computer traces the identified actions of the worker in chronological order and detects a change in the actions of the worker, the actions before the change are added to the action sequence list, The computer compares the ideal work procedure that defines the order of actions according to the work with the action sequence list, calculates an edit distance indicating the minimum number of times until the order of appearance of the actions included in the action sequence list is edited to match the actions included in the ideal work procedure, and detects an abnormal work based on the calculated edit distance A method for detecting an abnormal action sequence, characterized in that.

8. A computer identifies the actions of a worker in units of frames from a work video, When the computer traces the identified actions of the worker in chronological order from the first frame of the work video and detects a change in the actions before and after the frame, the actions before the change are added to the action sequence list The method for detecting an abnormal action sequence according to claim 7.

9. To the computer, A video input process for inputting a work video that captures the work process of a worker, An action identification process for identifying the actions of the worker from the input work video, An action sequence identification process for tracing the identified actions of the worker in chronological order and adding the actions before the change to the action sequence list when a change in the actions of the worker is detected, and An action sequence comparison process that calculates an edit distance indicating the minimum number of times to edit the appearance order of actions included in the action sequence list to match the actions included in the ideal work procedure by comparing the ideal work procedure that defines the order of actions according to the work with the action sequence list, and detects abnormal work based on the calculated edit distance An action sequence abnormality detection program for causing the above to be executed.

10. Causing a computer to In an action identification process, cause the actions of the worker to be identified in units of frames from the work video, In an action sequence identification process, cause the identified actions of the worker to be traced in time series from the first frame of the work video, and when a change in actions is detected before and after the frame, add the action before the change to the action sequence list The action sequence abnormality detection program according to claim 9.

Images