Action analysis device, method and program

Abstract

An action analysis device comprising an analysis unit and a visualization unit,　wherein the analysis unit comprising: a detection unit for detecting a person or a part of the person from video frames in which work of the person is captured; and a tracking unit for tracking the person or the part between the video frames; and the visualization unit acquires a tracking result from a predetermined time before a frame time to the frame time, arranges predetermined primitives at the positions of the person or the part included in the acquired tracking result, superimposes the primitives on a video frame at the frame time, and presents the video frame.

Description

ACTION ANALYSIS DEVICE, METHOD AND PROGRAM

[0001] The present disclosure relates to an action analysis device, method, and program.

[0002] A technique for analyzing the action of a part such as a human hand is known.

[0003] By the way, there is a need for a technology that presents useful information obtained by analyzing the action of a worker.

[0004] In view of the above circumstances, an object of the present disclosure is to provide an action analysis device, method, and program that presents useful information obtained by analyzing the action of a worker.

[0005] This disclosure provides an action analysis device comprising an analysis unit and a visualization unit, 　　wherein the analysis unit comprising: 　　a detection unit for detecting a person or a part of the person from video frames in which work of the person is captured; and 　　a tracking unit for tracking the person or the part between the video frames; and 　　the visualization unit acquires a tracking result from a predetermined time before a frame time to the frame time, arranges primitives at the positions of the person or the part included in the acquired tracking result, and superimposes the primitives on a video frame at the frame time, and presents the video frame.

[0006] This disclosure provides an action analysis method including: 　　detecting a person or a part of the person from video frames in which work of the person is captured; 　　tracking the person or the part between the video frames; 　　acquiring a tracking result from a predetermined time before a frame time to the frame time, arranges primitives at the positions of the person or the part included in the acquired tracking result, and superimposing the primitives on a video frame at the frame time, and presents the video frame. 　　wherein the analysis unit including: 　　a detection unit for detecting a person or a part of the person from video frames in which work of the person is captured; and 　　a tracking unit for tracking the person or the part between the video frames; and 　　the visualization unit acquires a tracking result from a predetermined time before a frame time to the frame time, arranges primitives at the positions of the person or the part included in the acquired tracking result, and superimposes the primitives on a video frame at the frame time, and presents the video frame.

[0007] This disclosure provides a program including: 　　detecting a person or a part of the person from video frames in which work of the person is captured; 　　tracking the person or the part between the video frames; 　　acquiring a tracking result from a predetermined time before a frame time to the frame time, arranges primitives at the positions of the person or the part included in the acquired tracking result, and superimposing the primitives on a video frame at the frame time, and presents the video frame.

[0008] This disclosure provides an action analysis method comprising an analysis unit and a visualization unit,

[0009] The action analysis device, method, and program according to the present disclosure presents useful information obtained by analyzing the action of a worker.

[0010] FIG. 1 is a block diagram illustrating the configuration of an action analysis device.FIG. 2 is a block diagram illustrating the configuration of an analysis unit.FIG. 3 is a block diagram of a data preparation unit.FIG. 4 is a schematic diagram.FIG. 5 is a schematic diagram.FIG. 6 is a schematic diagram.FIG. 7 is a schematic diagram.FIG. 8 is a schematic diagram.FIG. 9 is a schematic diagram.FIG. 10 is a schematic diagram.FIG. 11 is a schematic diagram.FIG. 12 is a schematic diagram.FIG. 13 is a schematic diagram.FIG. 14 is a schematic diagram.FIG. 15 is a schematic diagram.FIG. 16 is a schematic diagram.FIG. 17 is a block diagram of the action analysis device.FIG. 18 is a schematic diagram.FIG. 19 is a schematic diagram.FIG. 20 is a flow chart of the operation of the action analysis device.FIG. 21 is a flow chart of the operation of the action analysis device.FIG. 22 is a flow chart of the operation of the action analysis device.FIG. 23 is a flow chart of the operation of the action analysis device.FIG. 24 is a flow chart of the operation of the action analysis device.FIG. 25 is a schematic diagram.FIG. 26 is a schematic diagram.FIG. 27 is a flow chart of the operation of the action analysis device.FIG. 28 is a schematic diagram.FIG. 29 is a schematic diagram.FIG. 30 is a a flow chart of the operation of the action analysis device.FIG. 31 is a schematic diagram.FIG. 32 is a block diagram of the action analysis device.FIG. 33 is a schematic diagram.FIG. 34 is a schematic diagram.FIG. 35 is a schematic diagram.FIG. 36 is a block diagram of the action analysis device.FIG. 37 is a schematic diagram.FIG. 38 is a schematic diagram.FIG. 39 is a schematic diagram.FIG. 40 is a flow chart of the operation of the action analysis device.FIG. 41 is a flow chart of the operation of the action analysis device.FIG. 42 is a schematic diagram.FIG. 43 is a flow chart of the operation of the action analysis device.FIG. 44 is a schematic diagram.FIG. 45 is a schematic diagram.FIG. 46 is a schematic diagram.FIG. 47 is a schematic diagram.FIG. 48 is a schematic diagram.FIG. 49 is a schematic diagram.FIG. 50 is a flow chart of the operation of the action analysis device.FIG. 51 is a block diagram of the action analysis device.FIG. 52 is a schematic diagram.FIG. 53 is a schematic diagram.FIG. 54 is a schematic diagram.FIG. 55 is a block diagram of the action analysis device.FIG. 56 is a block diagram of the analysis unit.FIG. 57 is a schematic diagram.FIG. 58 is a schematic diagram.FIG. 59 is a schematic diagram.FIG. 60 is a schematic diagram.FIG. 61 is a schematic diagram.FIG. 62 is a schematic diagram.FIG. 63 is a schematic diagram.

[0011] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and duplicate descriptions will be omitted as necessary for clarity.

[0012] Embodiment 1 　　Embodiment 1 relates to a UI for visualizing the trajectory of a part (e.g., a hand) of a person (e.g., a worker). The trajectory of the person may be visualized. First, an outline of Embodiment 1 will be described. When a worker has a long working time, it is necessary to find the reason for the long working time in order to make the work more efficient. However, it is difficult to find the cause by simply comparing two videos. The action analysis device according to the first embodiment identifies various actions and visualizes the actions as events. The various actions include, for example, unnecessary, inefficient, and dangerous actions. This makes it easier to find the reason for the long working time of the worker. For example, when fast and long hand movements are visualized, it becomes possible to avoid such movements.

[0013] Fig. 1 illustrates a system including an action analysis device 40. The action analysis device 40 receives a video from a camera 20. The camera may include a Video Management System (VMS). The action analysis device 40 analyzes the video of a target worker and obtains information useful for improving the movement of the target worker. The action analysis device 40 can analyze not only the video of the target worker but also the video of a skilled worker in order to obtain useful information. The action analysis device 40 outputs useful information to a display 30.

[0014] Fig. 2 is a block diagram of an analysis unit 10 included in the action analysis device 40. The analysis unit 10 includes a data preparation unit 11, a condition generation unit 12, and an event detection unit 13.

[0015] The data preparation unit 11 generates time series data for each job cycle.

[0016] The data preparation unit 11 analyzes input video data and generates time series data for each cycle. The data preparation unit 11 specifies the position of a hand in each video frame and extracts features. The features include the hand trajectory, the direction of the movement, the degree of the movement, the speed of the movement, and the like. From these features, the data preparation unit 11 detects the start and end actions of each cycle and actions other than those actions. The data preparation unit 11 outputs the time series data of each specified cycle. The time series data includes extracted feature values and detected actions. The time series data may be divided into multiple tasks that make up a cycle. 　　The feature value may include the position, direction, velocity, acceleration, degree of directional change of the trajectory of the action, the orientation of the person or the part, and the size of the person or the part.

[0017] The condition generation unit 12 generates a condition by performing statistical processing on the information of the multiple cycles of the time series data.

[0018] The condition generation unit 12 receives the time series data and calculates various statistical information. The statistical information may include the position of the person or the part, the speed of movement of the person or the part, time positions of occurrence (appearance timing), duration, appearance interval, appearance probability, an appearance order, etc.

[0019] The event detection unit 13 detects an abnormality by comparing the statistical information with time series data.

[0020] The event detection unit 13 receives the time series data for each cycle, then detects the event by comparing with the statistics. The useful information for work improvement is output as the event. When useful information is obtained by using a video of a target worker and a video of a skilled worker, statistical information based on the time series data of the skilled worker is calculated, and the event is detected by comparing the time series data of the target worker with the statistical information.

[0021] Fig. 3 is a block diagram of the data preparation unit. The data preparation unit 11 includes a hand detection unit 111, an object tracking unit 112, an action detection unit 113, and a cycle detection unit 114.

[0022] The hand detection unit 111 may include an object detection model. The hand detection unit 111 detects a hand from each video frame.

[0023] Referring to Fig. 4, the object tracking unit 112 may, for example, repeat the process of selecting a hand that is the nearest to the hand detected in the previous frame from among 1 or more hands detected in the current frame and adding the selected hand to the corresponding tracking result. Referring to Fig. 5, the tracking result may include a time variation of the X coordinate and the Y coordinate of the hand.

[0024] The definition of the action will be described with reference to Fig. 6. The action represents the trajectory of the hand over a certain period of time. The action detection unit 113 may detect the action by, for example, pattern matching with a predetermined sample of the trajectory of the hand. For example, as shown in the figure on the left, actions A1 to A4 are defined. Action A1 is the action at the start of the cycle and is represented by trajectory T1. Action A2 is the action at the end of the cycle and is represented by trajectory T2. Action A3 is the action to pick up parts X and is represented by trajectory T3. Action A4 is the action to pick up parts Y and Z and is represented by trajectory T4. The action detection unit 113 detects actions A1 to A4 from the trajectory of the worker's hand.

[0025] Referring to Fig. 7, the operation of the action detection unit 113 will be described. As shown in the upper figure, a query (e.g., a hand trajectory) and a threshold are predetermined for each action. The action detection unit 113 calculates a distance between a trajectory in each time slot (…, S22, …, S100, …, S552, …) and each query (Q1, Q2). The action detection unit 113 may calculate the distance by using, for example, DTW (Dynamic Time Warping). The action detection unit 113 generates action detection information if the distance is equal to or less than the threshold value of the action. Or instead of the distance, the action detection unit 113 may calculate a similarity score between a trajectory in each time slot (…, S22, …, S100, …, S552, …) and each query (Q1, Q2). In this case, the action detection unit 113 generates action detection information if the similarity score is equal or more than the threshold value of the action.

[0026] Referring to Fig. 8, the operation of the cycle detection unit 114 will be described. The cycle detection unit 114 detects a cycle by detecting an action at the start of the cycle (e.g., action 1) and an action at the end of the cycle (e.g., action 6). Referring to Fig. 9, the cycle detection unit 114 may further detect a task included in the cycle. As a first method, the cycle detection unit 114 can detect a task by detecting a start action (e.g., action 1) of the task and an end action (e.g., action 3) of the task. As a second method, the cycle detection unit 114 can detect a task by detecting an action (e.g., Action 3, Action 5) as a boundary of the task and dividing the cycle at the boundary.

[0027] Referring to Fig. 10, a first example of the operation of the condition generation unit 12 and the event detection unit 13 will be described. The condition generation unit 12 receives time series data of 200 cycles. Then, the condition generation unit 12 calculates the probability that each action is included in a cycle. For example, when the action 8 is included in 199 cycles, the probability is 199 / 200=0.995. When the action 9 is included in 8 cycles, the probability is 8 / 200=0.04.

[0028] The condition generation unit 12 determines the usual action and the unusual action based on the probability. For example, an action with a probability of 0.95 or more (e.g., action 8) is a usual action. An action with a probability of less than 0.05 (e.g., action 9) is an unusual action. Referring to Fig. 11, if an unusual action exists, the event detection unit 13 issues an unusual action event for an action that is an unusual action. When a usual action is not included in a cycle, the event detection unit 13 issues a missed action event for the cycle.

[0029] Referring to Fig. 12, a second example of the operation of the condition generation 12 unit and the event detection unit 13 will be described. The condition generation unit 12 receives time series data of a predetermined number of cycles (e.g., 200). Then, the first appearance probability of each action is calculated. Then, the condition generation unit 12 outputs the calculated first appearance probability as a condition for detecting the unusual appearance probability.

[0030] The event detection unit 13 receives time series data and stored it in a storage. When the number of cycles stored in storage exceeds a limit (e.g., 100 cycles), data of the oldest cycle may be deleted. The event detection unit 13 calculates the second appearance probability of each action using the received time series data. The event detection unit 13 compares the first appearance probability of each action with the second appearance probability. When the difference between the first appearance probability and the second appearance probability exceeds a predetermined value (e.g., ±0.2), the event detection unit 13 issues an "unusual action appearance probability" event and outputs it.

[0031] Reffering to Fig. 13, a third example of the operation of the condition generation unit 12 and the event detection unit 13 will be described. (1) The condition generation unit 12 receives time series data of a predetermined number of cycles (e.g., 200). Then, the probability of each action is obtained. (2) The condition generation unit 12 specifies the first order of actions for each cycle. An action whose probability is less than a predetermined threshold (e.g., 0.85) is not included in the first order (e.g., Action 2 and Action 5 in Fig. 13). (3) A condition generation unit 12 obtains the number of cycles in which each first order exists. Then, a condition generation unit 12 calculates the percentge of the cycles containing the first order in all cycles. (4) If there is a first order of actions that exists in more than pre-determined percentage of the cycles (e.g. 95%), the condition generation unit 12 selects this order (e.g., (Action 1, Action 3, Action 6) in Fig. 13) as the condition to detect an "unusual order of actions" event.

[0032] (1) When the event detection unit 13 receives the time series data of one cycle, it specifies the second order of the actions. The second order does not include actions not included in the condition. (2) The event detection unit 13 checks whether the second order matches the condition. For example, the second order in Cycle 2, (Action 1, Action 6), does not match the condition (Action 1, Action 3, Action 6). When it does not match, the event detection unit 13 creates and outputs the "unusual order of actions" event.

[0033] Referring to Fig. 14, a fourth example of the operation of the condition generation unit 12 and the event detection unit 13 will be described. (1) The condition generation unit 12 receives time series data of a predetermined number of cycles (e.g., 200) and calculates the probability of an actions. (2) The condition generation unit 12 determines a usual action based on the probability. For example, the probability of the usual action is 0.95 or more. (3) The relations between the work process and the action is input by a user. For example, the GUI showing the work procedure is shown to the user, and the action is selected from the drop-down menu to obtain the relation. (4) The condition generation unit 12 sets the work process for the usual action. For example, the number of the work process is set. In Fig. 14, work process numbers 1, 5, 10, 12 are set to Action 1, Action 3, Action 4, and Action 6, respectively.

[0034] When the event detection unit 13 receives the time sequence data, it checks if any action without corresponding work process exists in a cycle. When such an action exists, the event detection unit 13 generates and outputs an "action not specified in procedure manual" event.

[0035] Referring to Fig. 15, a fifth example of the operation of the condition generation unit 12 and the event detection unit 13 will be described. The condition generation unit 12 calculates the number of detected actions in each cycle. Then, the condition generation unit 12 calculates the minimum N1, the maximum N2, and the mode of each action. For example, N1 is the α percentile, and N2 is the (1-α) percentile. If all cycles are known to be usual, N1 may be minimum and N2 may be maximum. Other common techniques for excluding outliers may be used. When the event detection unit 13 receives the time series data of the cycle, it checks whether the number N of each action satisfies N1<N<N2. When the condition is not satisfied, the event detection unit 13 issues an "unusual number of action occurrence" event.

[0036] The condition generation unit 12 may use a usual appearance interval of actions as a condition. When a cycle whose appearace interval is different from the condition, the event detection unit 13 issues an unusual appearance interval event. 　　From time series data containing hand positions for each cycle, the condition generation unit 12 may determine usual work area. If position of a hand deviates from the usual work area, a duration when hand is outside the usual work area exceeds a given value, a total distance hand moves outside the usual work area exceeds a given distance, and so on, the event detection unit 13 fires an "out of work area" event. 　　From time series data containing hand positions for each cycle, the condition generation unit 12 may determine usual work area for each worker. If position of a hand of one worker is in another worker's usual work area, the event detection unit 13 fires an "interfering in other work area" event. 　　From time series data containing hand movement speed for each cycle, the condition genration unit 12 may determine usual range of hand movement speed. If hand movement speed deviates from the usual range of hand movement speed, the event detection unit 13 fires a "faster / slower hand movement speed" event. 　　The condition generation unit 12 may determine usual range of task duration for each task from task durations identified in the time series data for each cycle. If task duration identified in the time series data deviates from the usual range of task duration, the event detection unit fires an "unusual task duration" event or a "long / short task duration" event. 　　A specific area where hand should not stay in may be registered. From time series data containing hand positions for each cycle, the condition generation unit 12 may determine maximum duration of stay in the registered area. If a duration that a hand is staying in the registered area exceeds the maximum duration of stay in the registered area, the event detection unit 13 fires a "staying in the specific area" event. 　　The condition generation unit 12 determines maximum duration of stop from durations that hand is stopped identified in time series data containing hand movement speed. If duration when hand is stopped exceeds the maximum duration of stop, the event detection unit 13 fires an "idle time" event.

[0037] Referring to Fig. 16, the definition of an event will be descrived. The event is detected by the event detection unit 13. The event is information attached to an action, cycle, or task having a range along the time axis. Thus, the event is displayed as a range along the time axis in the GUI (e.g., by using a seek bar). Since the time of the video is synchronized with the time of the time series data, the video clip to play back is also determined when the event is specified.

[0038] Example 1 　　In Example 1, a video of a target worker is compared with a video of a skilled worker. Fig. 17 is a block diagram of the action analysis device 40. The action analysis device 40 includes a video input unit 50 , an analysis unit 10, and a visualization unit 60. 　　The action analysis device 40 may be a computer comprising a processor and memory. Each function of the action analysis device 40 may be realized when the processor executes the program loaded into the memory.

[0039] The video input unit 50 receives and decodes video file or video stream, and outputs frame images for generating time series data.

[0040] The data preparation unit 11 includes an object detection unit, an object tracking unit, an action detection unit, and a cycle & task detection unit. The object detection unit detects a worker's hand from each frame image (e.g., by using deep learning models). The object tracking unit generates a trajectory from the position of the detected hand. The action detection unit detects an action (e.g., by comparing a part of the trajectory with the sample trajectory of the target action). The cycle & task detection unit detects a cycle and a task in the cycle. For example, the cycle & task detection unit may detect actions corresponding to the start and end of the cycle. Similarly, the cycle & task detection unit may detect actions corresponding to the start and end of the task. The cycle & task detection unit outputs time series data. The time series data includes trajectories, detected actions, and identified cycles and tasks.

[0041] The condition generation unit 12 calculates the probability of actions within each cycle performed by a skilled worker. The condition generation unit 12 determines actions with probability less than a certain probability (e.g., 1%) to be unusual actions and actions with probability greater than a certain probability (e.g., 99%) to be a usual action. The condition generation unit 12 determines the task duration of each task in the time series data of each cycle performed by the skilled worker. The condition generation unit 12 determines the usual work area from the time series data including the hand position of each cycle performed by the skilled worker.

[0042] The event detection unit 13 receives the time series data of each cycle performed by the target worker. When an unusual event is detected in the time series data, the event detection unit 13 issues an unusual action event. When usual action is not detected in the time series data of the cycle, the event detection unit 13 issues a missed action event. When a task period specified in the time series data is out of the usual range of the task duration, the event detection unit 13 issues an unusual task duration event. When a hand position is out of the usual work area, the event detection unit 13 issues an "out of work area" event. The event detection unit 13 detects the duration that hand positions deviate from the usual work area as an action, and can associate the detected action with the "out of work area" event.

[0043] The visualization unit 60 provides a GUI for visualizing the event as well as trajectory information of a person or a part of a person.

[0044] Fig. 18 shows an example of the GUI. Events such as "Unusual Action", "Long or Short Task Duration", and "Out of Work Area" are displayed on the seek bar. The seek bar controls the playback time of the video. When the seek bar is tapped or clicked, the video is played from the start position of the video section corresponding to the event. Hereinafter, a click operation using a pointing device can be replaced with a tap operation using a touch panel device or a selection operation using other input devices. The events related to the problematic hand movement are indicated by circles. Users can tap or click a circle. The color of the circles may be changed depending on the type or importance of the event.

[0045] The visualization unit 60 may obtain the polygon surrounding the trajectory of the person or the part in time segment in which the event is issued, overlaps the polygon as an "out of work" area on the screen, and moves, if the "out of work" area is selected, playback position of the video to time position corresponding to the time segment. Fig. 19 shows an example of the GUI. In addition to displaying the events on the seek bar, the usual work area by a skilled worker is shown. An area where the hand of the target worker is out of the usual work area is also shown. These areas may be displayed regardless of the playback position. Furthermore, when the above area is selected, the playback position may jump to the start position of the corresponding video section. When the hand of the target worker leaves the usual work area, it is detected as an action.

[0046] Fig. 20 is a flowchart showing an operation of the action analysis device 40. It is assumed that the user selects a video of a skilled worker and a video of a target worker on the same work bench. First, the action analysis device 40 analyzes the video of the skilled worker (step S11). Next, the action analysis device 40 detects an event from the video of the target worker (step S12). Next, the action analysis device 40 visualizes the detected event.

[0047] Fig. 21 is a flowchart showing a flow of analyzing the action of a skilled worker. First, the video input device 50 receives and decodes a certain period of the video of the skilled worker, and outputs frame images with time (step S21).

[0048] At step S22, the data preparation unit receives the frame images, detects a hand from each frame image (object detection), generates a trajectory of each hand from the position of the hand (object tracking), detects a certain period of the trajectory of the hand as an action (action detection), detects a cycle and a task based on actions corresponding to the start and end (cycle & task detection), and generates time series data including the trajectory of the hand, the action, the cycle, and the task.

[0049] At step S23, the condition generation unit 12 receives the time series data and generates an event detection condition.

[0050] An example of condition generation will be described below. (1) The probability of an action in a cycle performed by a skilled worker is calculated, and a usual action list and an unusual action list are generated based on the probability. (2) The minimum and maximum of the task duration of each task performed by a skilled worker are obtained, and the usual range of the task duration is obtained. (3) All hand positions of a skilled worker are obtained, and a usual work area is created as a polygon including hand positions.

[0051] At step S24, the event detection unit 13 receives the event detection condition and stores it in an internal storage.

[0052] Fig. 22 is a flowchart showing a flow of detecting an event from a video of a target worker.

[0053] At step S31, the video input unit receives and decodes a certain period of the video of the target worker, and outputs a frame images with time.

[0054] At step S32, the data preparation unit receives the frame images, detects the position of the hand in each frame image (object detection), generates a trajectory of each hand from the position of the hand (object tracking), detects a certain period of the trajectory of the hand as an action (action detection), detects a cycle and task by detecting actions corresponding to the start and end (cycle & task detection), and generates time series data including the trajectory of the hand, actions, cycles, and tasks.

[0055] At step S33, the event detection unit receives the time series data and checks whether any of the event detection conditions stored in the memory are satisfied. When the event detection conditions are satisfied, the event detection unit creates an event corresponding to the event detection condition. The event detection unit 13 outputs the time series data and the created event.

[0056] An example of checking whether the event detection condition is satisfied will be described below. (1) If an action of a cycle of time series data is included in the unusual action list, an "unusual action" event is issued. If any action in the usual action list does not exist in a cycle of time series data, a "missed action" event is issued. (2) If the task duration in the time series data is out of the usual range of the task duration, an "unusual task duration" event is issued. (3) If the hand position is out of the usual work area, an "out of work area" event is issued. The duration when the hand is out of the usual work is identified as an action, and the event may be related to the detected action.

[0057] Fig. 23 is a flowchart showing a flow for visualizing the detected event. At step S41, the action analysis device 40 reads the event, time series data, and frame images.

[0058] At step S42, the action analysis device 40 sets the playback position to the start position of the video.

[0059] At step S43, the action analysis device 40 requests an update of the seek bar and the playback.

[0060] At step S44, the action analysis device 40 determines whether an update of the seek bar is requested. If an update of the seek bar is requested (YES at step S44), the seek bar having the playback position is drawn (step S45). Next, the action analysis device 40 draws an event on the seek bar (step S46).

[0061] If an update of the seek bar is not requested (NO at step S 44), or after step S46, the action analysis device 40 determines whether the playback update is requested (step S47). If an update of the playback is requested (YES at step S 47), the action analysis device 40 draws the frame image at the playback position in the playback area (step S48). Next, the action analysis device 40 draws the trajectory of the hand superimposed on the frame image (step S49). Next, the action analysis device 40 displays the event superimposed on the frame image (step S410). Note that the superimposing processes from S48, S49 and S410 may be done on a buffer storing the frame image and the resultant frame image may be displayed in the playback area.

[0062] If the playback update is not requested (NO at step S47), or after step S410, the action analysis device 40 determines whether the seek bar is clicked (step S411). If the seek bar is clicked (YES at step S411), the action analysis device 40 gets time position and changes the playback position to the time position (step S412). Next, the action analysis device 40 requests an update of the seek bar and the playback (step S413).

[0063] If the seek bar is not clicked (NO at step S411), or after step S413, the action analysis device 40 determines whether the control button is clicked (step S414). If the control button is clicked, the action analysis device 40 sets the playback position to the start position of the video, or changes the playback speed to the speed according to the button (step S415).

[0064] If the control button is not clicked (NO at step S414), or after step 415, the action analysis device 40 determines whether the screen is closed (step S416). If the screen is closed (YES at step S416), the action analysis device 40 ends the processing, and if the screen is not closed (NO at step S416), the action analysis device 40 performs step S44.

[0065] Referring to Fig. 24, a flow of drawing an event on the seek bar will be described. First, the operation analysis unit gets an event e (step S51).

[0066] Next, the action analysis device 40 gets the event type and event time duration (t_begin, t_end) of the event e (step S52).

[0067] Next, the action analysis device 40 searches a lookup table 1 of drawing parameters using the event type (step S53). Fig. 25 shows lookup table 1 of drawing parameters. The event type is associated with primitives, colors, and sizes.

[0068] Next, the action analysis device 40 determines whether a drawing parameter is found (step S54).

[0069] If the drawing parameter is found (YES at step S54), the action analysis device 40 draws the event according to the drawing parameter (step S55). Fig. 26 shows an event displayed on a seek bar.

[0070] If the drawing parameter is not found (NO at step S54), or after step S55, the action analysis device 40 determines whether all events were processed (step S56). If all events were processed (YES at step S56), the action analysis device 40 ends the processing, and if any event was not processed (NO at step S56), the action analysis device 40 executes step S51.

[0071] Fig. 27 is a flowchart showing a flow of drawing the hand trajectory overlapping to the frame image.

[0072] First, the action analysis device 40 gets hand trajectories (H1, H2, …, Hn) for X seconds before the playback position from the time series data (step S61).

[0073] Next, the action analysis device 40 gets a hand trajectory Hi (step S62).

[0074] Next, the action analysis device 40 gets a hand position h in the hand trajectory Hi (step S63).

[0075] Next, the action analysis device 40 gets time t of the hand position h, and searches for the event e that t is within the event's period (step S64).

[0076] Next, the action analysis device 40 searches the drawing parameter lookup table 2 using the event e (if any), the time position t, and the hand ID (step S65). Fig. 28 shows a lookup table 2. Events, time positions, and hand IDs are associated with primitives, colors, and sizes.

[0077] Next, the action analysis device 40 draws the hand position according to the drawing parameter (step S66).

[0078] Next, the action analysis device 40 determines whether all the hand positions were processed (step S67). If any hand position was not processed, the action analysis device 40 performs step S63.

[0079] If all hand positions were processed (YES at step S67), the action analysis device 40 determines whether all the parts of hand trajectories were processed (step S68). If all the parts were processed (YES at step S68), the action ananlysis device 440 ends the processing, and if any part was not processed (NO at step S68), the action analysis device 40 performs step S62.

[0080] If the primitive can present a direction of motion, such as a triangle, the motion vector is determined from the change in the detected positions of the hand, and if the magnitude of the vector is greater than or equal to a certain value, the angle at which the primitive is rotated to match the orientation of the vector is also determined. If the magnitude of the vector is less than or equal to a predetermined value, a dot is selected as a primitive instead. Fig. 29 shows lookup table including a traiangle as a primitive.

[0081] Fig. 30 is a flowchart showing a flow for drawing an event ovelapping to the frame image.

[0082] First, the action analysis device 40 gets an event e (step S71).

[0083] Next, the action analysis device 40 gets an event type and an event time duration (t_begin, t_end) of the event e (step S72).

[0084] Next, the action analysis device 40 searches a lookup table 3 of drawing parameters by using the playback position, event type, and event time duration (step S73). Fig. 31 shows a lookup table 3. Event types and playback positions are associated with primitives, colors, sizes, and text. For highlighting with a time direction parameter (e.g., flashing), its period may be included in the parameter.

[0085] Next, the action analysis device 40 determines whether the drawing parameter is found (step S74).

[0086] If the drawing parameter is found (YES at step S74), the action analysis device 40 draws the event e according to the drawing parameter (step S75).

[0087] If the drawing parameter is not found (NO at step S74), or after step S75, the action analysis device 40 determines whether all the events were processed (step S76). If all the events were processed (YES at step S76), the action analysis device 40 ends the processing, and if any event was not processed, the action analysis device 40 performs step S71.

[0088] Referring to Fig. 31, the work area polygon includes a usual work area and a deviated area. The deviated area is a polygon that surrounds positions of a hand which deviates from the usual work area. The usual work area is drawn by the usual work area the event is detected with. Text corresponding to the event is displayed in the playback area according to the lookup table 3.

[0089] Example 2 　　Example 2 displays inefficient hand movements. Fig. 32 is a block diagram of the action analysis device 40. The action analysis device 40 includes a video input unit 50, an analysis unit 10, and a visualization unit 60.

[0090] The data preparation unit 11 includes an object detection unit, an object tracking unit, and an action speed calculation unit. The object detection unit detects a worker's hand from each frame image (e.g., by using a deep learning model). The object tracking unit generates a trajectory from the positions of the detected hand. The motion speed calculation unit calculates the motion speed of each data point at the hand trajectory.

[0091] The condition generation unit 12 includes an upper / lower limit of usual motion speed determination unit. The upper / lower limit of usual motion speed determination unit calculates the upper limit and the lower limit of the usual motion speed in the time series data for each cycle.

[0092] The event detection unit 13 receives the time series data including the motion speed of the hand and the upper limit and the lower limit of the usual motion speed. When the motion speed of the hand is faster / slower than the upper / lower limit, the event detection unit 13 issues a "faster / slower hand movement speed" event.

[0093] The visualization unit 60 provides a GUI for visualizing the event.

[0094] Fig. 33 shows the GUI. When the user taps or clicks on a circle, the playback position is adjusted to a scene identified by the system as having a hand movement problem.

[0095] An example of a problematic hand movement is described below. When a hand moves quickly or slowly, there is room for improvement in the efficiency of the hand movement. When the hand stays in the same position for a while, it is related to an unfamiliar task and there is room for improvement in efficiency. Or there may be a problem with a tool or a part. If the hand goes out of the golden range (e.g., usual work area), there is room for improvement in the efficiency of the hand movement. If there is interference with other tasks, there is a delay in neighboring processes.

[0096] The size of the mark (e.g., a circle) changes depending on the severity of the event. The size of a circle that indicates the fast / slow hand movement differs from the size of a circle that indicates the longer time staying.

[0097] A dot in a frame image indicates a position of a hand. It is indicated here by a dot, but it may be indicated by a triangle to indicate the direction of the hand movement. The direction of the movement may be indicated by using an another method.

[0098] The lookup table used in the example 2 is the same as the lookup table used in Example 1. However, the drawing parameters of Example 1 and Example 2 are different. Fig. 34 and Fig. 35 show lookup tables used in the example 2.

[0099] Example 3 　　Example 3 provides side-by-side display or superimposed display.

[0100] It is possible to synchronously play back videos in a task unit that is finer than the cycle. A task division function of the action analysis unit 10 is used. The difference between the actions of the skilled person and the target worker is detected, and the part where the difference is large is highlighted. The color of the trajectory is changed, the dot is made bigger, or the area is filled. The highlighting may be blinking the primitives when the video playbacks, or creating a polygon surrounding the primitives contained in the trajectory and superimposing the polygon on a screen.

[0101] Fig. 36 is a block diagram of the action analysis device 40. The data preparation unit 11 is the same as in Example 1.

[0102] The task association unit 14 associates task starting points between workers (a target worker (e.g., myself) and a skilled worker). This association is performed by matching trajectories of hands of workers. Fig. 37 shows an association table showing association results.

[0103] A visualization unit 60 determines and displays a marker position indicating a task start point on the time line based on information of start time points of individual tasks in the association table. The visualization unit 60 synchronizes the side by side or superimposed videos based on the associated start points.

[0104] Fig. 38 and Fig. 39 show the side-by-side display. The videos can be arranged vertically. Triangle marks on the time display bar indicates the heading points for a subtask, where the shape of the mark is not limited to a triangle, a mark of any shape may be used. The heading points are displayed based on the association table. When a heading point is selected (clicked or tapped), the right video and the left video are synchronized. If the end time of the task is different from the start time of the next task, the association table further stores the end time. If the mark of a first task start time of the task on the seek bar of a first video that is a video of a first person is selected, the visualization unit 60 moves a playback position of the first video to the first task start time, and moves playback position of second video that is a video of a second person to a second task start time of the task that is associated with the first task time in the task time association table, and playback the first and second videos simultaneously.

[0105] Referring to Fig. 39, when a task (e.g., task C) is selected (clicked or tapped) from the task list, the association table is referenced and scenes of the selected task is playbacked. When the end time of the selected task is reached, the video playback may stop automatically. When a task in the task list is selected, the visualization unit 60 highlights the task in the task list, gets the task start time of the task of each person based on the task time association table, and moves the playback position of each video to the task start of the video, and playbacks the videos simultaneously.

[0106] Fig. 40 is a flowchart showing the flow for visualizing the detected event. Comparing Fig. 40 with Fig. 23, steps S71 to S79 are added.

[0107] After step S45, a task is drawn on the seek bar (step S71).

[0108] If playback update is not requested (NO at step S47), or after step S410, the action analysis device 40 determines whether the seek bar of the target worker is clicked (step S72). If the seek bar is clicked (YES at step S72), the action analysis device 40 gets the time position clicked from the seek bar and sets the playback position of the target worker to the time position (step S73). Next, the action analysis device 40 calculates and sets the playback position of the skilled worker (step S74). Next, the action analysis device 40 requests an update of the seek bar and the playback position (step S75).

[0109] If the seek bar of the target worker is not clicked (NO at step S 72), or after step S75, the action analysis device 40 determines whether the seek bar of the skilled worker is clicked (step S76). If the seek bar of the skilled worker is clicked, the action analysis device 40 gets time position and set the playback position of the skilled worker to it (Step S77). Next, the action analysis device 40 calculates and sets the playback position of the target worker (Step S78). Next, the action analysis device 40 requests an update of the seek bar and the playback (Step S79).

[0110] If the seek bar of the skilled worker is not clicked (NO at Step S76), or after Step S79, the next step will be performed.

[0111] An example of the calculation method of Step S74 and Step S78 will be described. The playback position of the skilled worker is calculated from the playback position of the target worker. Similarly, the playback position of the target worker is calculated from the playback position of the skilled worker. The formula is expressed, for example, as follows.

[0112] P2 = S2 + {(P1 - S1) / (E1 - S1)} * (E2 - S2)

[0113] P represents the playback position. S represents the start time of the task where P is in. E represents the end time of the task where P is in. The end time of the task may be the start time of the next task. P and S are obtained from the task association table. P, S and E with index 1 are for the target worker, and those with index 2 are for the skilled worker. When calculating the playback position of the target worker is calculated from the playback position of the skilled worker, just change the index 1 and 2 in the above expression. Playback can be started immediately after the playback position is set. Also, the playback end time may be set, and playback will stop when the playback end time is reached. When both the target worker's playback and the skilled worker's playback stop, the playback position may be set to the start time of the next task, and playback may start automatically.

[0114] Fig. 41 shows a flow for drawing a task int the seek bar. First, the action analysis device 40 gets a task (step S101). Next, the action analysis device 40 gets a start time position of the task (step S102). Next, the action analysis device 40 calculates the coordinate at the seek bar based on the start time position (step S103). Next, the action analysis device 40 draws the start time position of the task (step S104). Next, the action analysis device 40 determines if all the tasks were processed (step S105). If all the tasks were processed (YES at step S105), the action analysis device 40 ends the processing, and if any task was not processed (NO at step S105), the action analysis device 40 performes step S101.

[0115] Fig. 42 shows a task on the seek bar. A start time position of a task is drawn with a triangle mark, but the shape of the mark is not limited to a triangle, a mark of any shape may be used.

[0116] Fig. 43 shows a flow for visualizing the detected event. The action analysis device 40 reads events, time series data, and frame images (step S201). Next, the action analysis device 40 sets the playback positions of the target worker and the skilled worker to the beginning of the videos (step S202). Next, the action analysis device 40 requests an update of the seek bars and playbacks (step S203). Next, the action analysis device 40 determines whether the screen is closed (step S204). If the screen is closed (YES at step S204), the action analysis device 40 ends the processing. If the screen is not closed (NO at step S204), the action analysis device 40 determines whether an update of the task list is requested (step S205). If an update of the task list is requested (YES at step S205), the action analysis device 40 draws the task list (step S206). If an update of the task list is not requested (NO at step S205), or after step S206, the action analysis device 40 determines whether a task in the task list is clicked (step S207). If the task is clicked (YES at step S207), the action analysis device 40 gets the start times for both the target worker and the skilled worker from the task association table, and sets the playback positions of the target worker and the skilled worker (step S208). Next, the action analysis device 40 requests an update of the playbacks of the target worker and the skilled worker (step S209).

[0117] The playback may start immediately after the playback position is set. Also, the playback end time may be set. Playback will stop when the end time is reached. When playbacks of both the target worker and the skilled worker stop, the next task in the task list may be automatically selected, the playback positions may be set to the start times of the next task, and playbacks may start automatically.

[0118] The videos of the target worker and the skilled worker may be superimposed. Fig. 44 shows a display in which two videos are superimposed. The scene to be played back is selected from the task list. Tasks are arranged in time series order int the task list. After matching the start times of the task, the video of the skilled worker is superimposed with the video of the target worker. The playback position may be selected (clicked or tapped) in the time bar. When the playback position is selected in one time bar, the playback position in the other time bar is automatically adjusted based on the association table. When a task in the task list is selected, the visualization unit 60 highlights the task in the task list, gets the task start time of the task of each person based on the task time association table, generates a frame by superimposing video frames of the plurality of persons respectively, each video frame corresponding to a task time after the task start time, and playbacks a video containing the frame.

[0119] Fig. 45 shows a display screen. The trajectory of the left hand and the trajectory of the right hand of the worker are distinguished from each other. The tracking results from a certain time back from the present time to the present time are displayed. The tracking results are not connected, and each detection result is indicated by a dot.

[0120] The dot indicates the position of the hand. The position of the hand may be indicated by a triangle. The orientation of the triangle indicates the direction of the movement of a hand. A primitive other than the triangle that can indicate an orientation may be used.

[0121] Reffering to Fig. 46, actions in each cycle are indicated by circles. The condition generation unit 12 of the action analysis device sets periodicity of the action based on the position of the slider on the slider bar. The condition generation unit 12 calculates an appearance probability of each action based on the action analysis result of a plurality of cycles of a person. The condition generation unit 12 determines the perioditicity of the action based on the appearance probability. The condition generation unit 12 sets the condition that the action of which the periodicity is low is detected. The event detection unit 13 may issue the event if the action satisfying the condition is detected. When the slider is moved to the bottom, all the actions are displayed. When the slider is moved upward, actions whose perioditicy is high are displayed. When the slider is moved to the top of the slider, the action that is not obserbed in other cycles is displayed.

[0122] Referring to Fig. 47, actions of each worker are indicated by circles. When the slider is moved to the bottom, all the actions are displayed. The condition generation unit 12 calculates an appearance probability of each action based on the action analysis results of a plurality of cycles of a plurality of persons. The condition generation unit 12 determines an individuality of the action based on the appearance probability. The condition generation unit 12 sets the condition that the action of which the individuality is high is detected. The event detection unit 13 may issue the event if the action satisfying the condition is detected. When the slider is moved upward, actions whose individuality is high are displayed. When the slider is moved to the top of the slider, the action that is very unique to each worker is displayed. When "Low" is set to the slider, all actions are shown, and when "High" is set to the slider, a common action among the plurality of workers is filtered out, and an action unique to the worker is shown.

[0123] Fig. 48 is a display of an action that interferes with other work area. A hand that reaches the hand area of workbench 2 is shown by a circle. Any shape may be used in place of the circle. In addition, the buttons "Filter by periodicity" and "Filter by Individuality" are buttons for filtering events by periodicity or individuality. The slider bar to adjust prdiodicity or individuality may be shown when a user clicks or taps the buttons.

[0124] Example 4 　　Referring to Fig. 49, a cycle list is displayed on the left side, and a video is displayed on the right side. Detected events are shown in each cycle in the cycle list. When an event is clicked, the video is jumped to a scene corresponding to the event.

[0125] Fig. 50 shows the flow for visualizing the detected event. At step S301, the action analysis device 40 determines whether an event in the cycle list is clicked. If an event in the cycle list is clicked (YES at Step S301), the action analysis device 40 gets the start time of the clicked event and changes the playback position to the start time (Step S302). Next, the action analysis device 40 requests an update of the seek bar and the playback (step S303).

[0126] Embodiment 2 　　Embodiment 2 balances workload of workers on assembly lines. Embodiment 2 can optimize an assembly line even in the absence of a production engineer who addresses productivity improvement for the assembly line.

[0127] Fig. 51 is a block diagram of the action analysis device 40. The action analysis device 40 includes a video input unit 50, an analysis unit 10, a data storage unit 81, a shift table input unit 82, a line balance optimization unit 83, and a visualization unit 60. The visualization unit 60 includes a line balance display unit 84, and a task display unit 85.

[0128] The video input unit 50 receives and decodes a video file or video stream, and stores frame images in the data storage unit 81 for analysis.

[0129] The analysis unit 10 analyzes frame images stored in a data storage unit 81. The analysis unit 10 includes the data preparation unit 11, the condition generation unit 12, and the event detection unit 13.

[0130] The data preparation unit 11 of the analysis unit generates time series data and stores it in a data storage unit 81. The time series data includes a hand trajectory, a detected action, a specified cycle and a task, and an event detected in the action, the cycle, and the task. When tasks are specified, the analysis unit 10 measures an idle time between cycles and tasks.

[0131] The condition generation unit 12 gets, from the action analysis result and a cycle detection result, at least one of a usual task duration, usual movement speed of a person or a part of a person, and a usual idle time based on the action detection result of a plurality of cycles of usual time included in time series data of a certain period, and sets the condition that the at least one of the usual task duration, the usual movement speed, and the usual idle time.

[0132] The event detection unit 13 gets at least one of a task duration, movement speed of the person or the part, and an idle time based on the time series data after the condition is set, and issues an event if the task duration, the movement speed, or the idle time differs from the usual task duration, the usual movement speed, or the usual time by a predetermined value. The event detection unit 13 detects an idle time event, an unusual task duration event, or a slower hand movement speed event. The idle time event indicates that a worker is waiting for a previous worker to finish his / her work. The unusual task duration event or the slower hand movement event indicates that the worker is intentionally slows down the pace of work. The event detection unit 13 may compare the idle times of adjacent persons and issues the idle time event if the difference between the idle times becomes large.

[0133] The analysis unit 10 may also use object detection to directly detect the number of products between workbenches. The event detection unit 13 detects a work object existing between the plurality of persons from a video, and issues an unusual task duration event if the number of the detected object becomes large.

[0134] The data storage unit 81 keeps time series data and videos.

[0135] The shift table input unit 82 receives the shift table and the initial task distribution for a target assembly line through CUI, GUI, or API call then stores it in the data storage unit 81.

[0136] The line balance optimization unit 83 reads the shift table, the current task distribution, and the estimated task time by the workers in the shift table from the data storage unit 81. The line balance optimization unit 83 estimates the line throughput upon request by a line leader or detection of an event related to work stagnation. The line balance optimization unit 83 calculates, if the event is issued, a throughput of a line when task assignment of the plurality of persons is changed, determines the task assignment of which the throughput becomes maximum, and generates work assignment information based on the determined task assignment. The line balance optimization unit 83 estimates the line throughput by collecting the task times done by the workers at the workbench for each re-assignment case in which the assignment of works to the workers is changed. The line balance optimization unit 83 selects one re-assignment case which yields the best line throughput as a suggestion. The line balance optimization unit 83 stores the suggestion in the data storage unit. The tasks can be labeled as re-assignable or not. If so, only re-assignable tasks are reassigned when creating a re-assignment case.

[0137] The line balance display unit 84 provides GUI to show current and suggested task assignment, and line throughput for the target assembly line. The line balance display unit 84 may generates a screen for presenting the work assignment information to a line leader.

[0138] The task display unit 85 displays tasks assigned for each worker. The task display unit generates a screen for presenting the wok assignment instruction to each worker based on the work assignment information. The task display unit 85 may further displays previous work assignment, and highlights a part of the determined task assignment that is changed from the previous work assignment. The task display unit 85 presents not only the work assignment of the target worker but also the work assignment of an adjacent worker who is adjacent to the target worker in the line.

[0139] Fig. 52 shows an example of the flow to suggest work distribution. (1) Referring to the first figure from above, usual cycle / task durations are individually measured by the analysis unit and stored in the data storage unit. Usually the line throughput for this line is 40 s / unit with skilled workers A, B, and C.

[0140] (2a) Referring to the second figure from the top, one day, a temporary worker D is at the last workbench to cover absence of the worker C. The event detection unit detects an unusual task duration event at the last workbench.

[0141] (2b) Referring to the third figure from the top, the workers on the upper workbenches sometimes dare to slow down their work according to the temporary worker's pace. The event detection unit detects an unusual task duration event.

[0142] (3) Referring to the fourth figure from the top, the action analysis device 40 creates a task reassignment suggestion. The throughput of the task reassignment suggestion is predicted to be the best according to the task duration of the skilled workers at (1) and task duration by the temporary worker at (2a) and (2b).

[0143] Fig. 53 shows a display for suggesting the work distribution. The task distribution is changed over lunch time or break time, then new task assignment is displayed for the screen of each worker.

[0144] Fig. 54 shows an example of the flow to suggest work distribution. (4) Referring to the first figure from above, a few hours after the task assignment change, the temporary worker D gets used to the assigned tasks and the duration of the assigned tasks are shorter. The action analysis device 40 detects an unusual task duration event.

[0145] Referring to the second figure from the top, the action analysis device 40 creates a task re-assignment suggestion. The task reassignment with the best throughput is suggested according to the usual task duration by the skilled workers and the task duration by the temporary worker. Here, the throughput is unchanged from the case above, but throughput increase is predicted as the temporary worker is getting used to the newly assigned task.

[0146] Referring to the third figure from the top, a few hours after the task assignment change, the temporary worker D gets used to the newly assigned task.

[0147] As another example, the action analysis device 40 may reduce the number of tasks assigned to a worker once this worker's duration gets longer due to fatigue.

[0148] If an interferring in other work area event is detected, the action analysis device 40 may identify the interfered task done at the adjacent workbench and mark or label the identified task as reassignable. The interfered task can be interfered with another assembly line where similar products are being made. The action analysis device 40 can raise an alert once a missed action event is detected after task assignment change. Task distribution among workers can be changed to allow workers to take turn taking a certain idle time between cycles to prevent a specific worker (at the bottle neck in assembly lline) from getting too tired. The action analysis device 40 can create a task re-assignment suggestion even if number of workers at the assembly line is changed according to workers shift and production plan.

[0149] Embodiment 3 　　Embodiment 3 updates a work procedure table. The work procedure table specifies time durations of tasks at each workbench. When the durations are shortened through KAIZEN activities, a production engineer updates the work procedure table. Updating work procedure table currently requires manpower.

[0150] Fig. 55 is a block diagram of the action analysis device 40. The action analysis device 40 includes a video input unit 50, an analysis unit 10, a data storage unit 91, a work procedure document input unit 92, a work procedure document update unit 93, and an work procedure document review unit 94.

[0151] The work procedure document input unit 92 receives the work procedure table for a target assembly line through CUI, GUI, or API call then stores it in the data storage unit.

[0152] Upon request by the work procedure document review unit 94 , the work procedure table updating unit 93 updates time duration of the specific task with the estimation one which has been stored in the data storage unit.

[0153] The work procedure document review unit 94 generates work procedure document update information for correcting time information of each task in the work procedure document based on the task boundary candidate, while displaying the video corresponding to the task in the work procedure document. The task boundary candidate is calculated by the analysis unit 10. The work procedure document review unit 94 may select at least one candidate section determined by the task boundary candidate information, and edits the time of each task in the work procedure document by associating the at least one candidate section with the task. The work procedure document review unit 94 may edit the time of the task in the work procedure document by selecting a boundary position of the task from a plurality of task boundary candidates.

[0154] The work procedure document review unit 94 gets the work procedure table and the estimated task time from the data storage unit, and provides a user GUI to select one or more tasks corresponding to the specific task in the work procedure table. The work procedure document review unit 94 issues an update request to the work procedure document update unit 93.

[0155] The work procedure document review unit 94 can notify a user of a work procedure table that needs updating. This notification can be triggered by the detection of an unusual task duration event. As an example of the notification, an icon is shown at a specific work procedure table which needs updating in the list of work procedure tables. As another example, an email is sent to inform the user of updating the work procedure table.

[0156] Fig. 56 is a block diagram of the analysis unit 10. The analysis unit includes a data preparation unit 11, a condition generation unit 12, and an event detection unit 13.

[0157] The condition generation unit 12 calculates a task boundary candidate based on the action detection result in a plurality of cycles included in time series data of a certain period. An example to calculate the task boundary candidate will be described. The condition generation unit 12 receives time series data including action detection results of pre-determined number of cycles. Then, the condition generation unit 12 calculates the probability that each action is included in a cycle. The condition generation unit 12 determines the usual action based on the probability. The determined usual actions are the task boundary candidates. Another example to calculate the task boundary candidate will be described. The condition generation unit 12 refers to a trajectory pattern of the person or the part that is associated with each task when the work procedure document was generated or updated in the past, and calculates the task boundary candidate by determining similarity between the trajectory pattern and the tracking result of the person or the part of each cycle. If actions which corresponds to the start and end of a task, or the boundary of two tasks are pre-defined at the data preparation unit 11; such actions are the task boundaries as well.

[0158] The event detection unit 13 issues an unusual task duration event.

[0159] Fig. 57 shows an UI example. A work bench and a worker are selected through the UI.

[0160] Fig. 58 shows an UI example. A cycle is seleced through the UI. When a cycle is tapped or clicked, preview of the cycle is displayed.

[0161] Fig. 59 shows an UI example. An actual work scene is associated with a task in work procedure table through the UI. The duration of the task is updated based on the duration corresponding to the scene. The editing screen includes the task list display area for displaying a task list of tasks included in the work procedure document, a split result presentation area for presenting a plurality of split segments on a time line generated by splitting the video at task boundary candidate, and a video playback area for playing back the video. When a split segment is selected, the video of the split video segment is played back in the video playback area. When at least one split segment is selected with a task in the task list selected, the at least one split section is associated with the task, and total time of the at least one split section is set as the time of the task. The association may be done by dragging the at least one split segment selected to the task region. Note that multiple segments can be selected at once and can be associated with a task. In this case, the total time is a sum of durations of the selected segments.

[0162] Fig. 60 shows an UI example. The action analysis device 40 analyzes the hand movement and divides the cycle into small task segments beforehand. The task segments are shown on the left side. When a user selects a task segment, a scene relating to the task segment is displayed on the right side. The speed of the playback near the boundary between the taske segments can be slow. 　　The user drags the task segment to a task (e.g., task C) in the work procedure table. Thus, the work procedure table is updated.

[0163] Fig. 61 shows an UI example. The work process is automatically estimated by video analysis beforehand. The work procedure is updated by clicking or tapping a button on the screen. The condition generation unit 12 estimates task name of a split segment generated by splitting the video at the task boundary candidate, based on the action analysis or a trajectory pattern for each task registered when the work procedure document was generated or updated in the past. The editing screen includes a task list display area for displaying a task list of tasks included in the work procedure document, and a video playback area for playing back the video. The work procedure document review unit 94 displays the estimated task name (e.g., task C) of the split section on the editing screen while playing back a video segment corresponding to the split section, and generates the work procedure document updating information based on the split section when a predetermined area on the editing screen is selected. The task name is estimated by the condition generation unit 12. The condition generation unit 12 may estimate the task name based on the action analysis or a trajectory pattern for each task registered when the work procedure document was generated or updated in the past.

[0164] While playing a video, a work boundary (for example, the boundary between Task A and Task B) is specified by entering (for example, clicking the mouse) a break in each task. If there is an idle time between Task A and Task B, it may be possible to specify it (the idle time is specified and the task boundary is entered). The speed of video playback is set to be slow near the time of a candidate for break (split) obtained by task division unit, and set to be fast when it is not. This makes it easier for a user who perform mapping operations (or annotations on operation sections) to enter breaks in the tasks, and shortens the time to check the images of other parts. At this time, it is better to oversplit the task than the actual task. Instead of continuously changing the playback speed, parts other than the task break candidates may be skipped and played only at the time near the task break.

[0165] Fig. 62 shows an UI example. Candidates for split are marked on the timeline. When the user presses the confirm split button, the end position of the task is confirmed, and the focus shifts to the next task. An editing screen includes a task list display area for displaying tasks included in the work procedure document, a video playback area for displaying the task boundary candidate as a mark on a seek bar, and a split determination input area that is selected when split positions of the video are determined. The work procedure document review unit 94 highlights a task in the task list when a video segment corresponding to the task is played back, determines an end position of the task when the split determination input unit is selected, and highlights the next task. A playback speed of the video is slow when playback position is within a predetermined range from the task boundary candidate, and the video playback speed is fast when the playback position is not within the predetermined range from the task boundary candidate.

[0166] Fig. 63 shows an UI example. When a task is selected from the task list, video clips which are specific for the task are shown in the playback screen. In the playback screen, silhouettes of workers from different video clips are superimposed. If there is a large difference between the silhouettes, the difference is displayed with a color (e.g., red). If clicked with the right button, the user can exclude the specific video clip which is deemed as wrongly estimated from the training data.

[0167] The program includes instructions (or software code) to cause the computer to perform one or more of the functions described in the embodiments when read into the computer. The program may be stored in a non-temporary computer readable medium or a substantial storage medium. By way of example, but not limitation, a computer readable medium or a substantial storage medium may include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disc storage or other magnetic storage device. The program may be transmitted on a temporary computer readable medium or communication medium. By way of example, but not limitation, a temporary computer readable medium or communication medium includes an electrical, optical, acoustic, or other form of propagating signal.

[0168] Although the embodiments of the present disclosure have been described in detail, the present disclosure is not limited to the embodiments described above, and modifications or modifications to the embodiments are included in the present disclosure to the extent that they do not deviate from the purpose of the present disclosure.

[0169] The program includes instructions (or software codes) that, when loaded into a computer, cause the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-transitory computer readable medium or a tangible storage medium. By way of example, and not limitation, non-transitory computer readable media or tangible storage media can include a random-access memory (RAM), a read-only memory (ROM), a flash memory, a solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disk (DVD), Blu-ray disc ((r): Registered trademark) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted on a transitory computer readable medium or a communication medium. By way of example, and not limitation, transitory computer readable media or communication media can include electrical, optical, acoustical, or other form of propagated signals.

[0170] Various combinations and selections of various disclosed elements (including each element in each example, each element in each drawing, and the like) are possible within the scope of the claims of the present disclosure. That is, the present disclosure naturally includes various variations and modifications that could be made by those skilled in the art according to the overall disclosure including the claims and the technical concept.

[0171] The whole or part of the embodiments disclosed above can be described as, but not limited to, the following supplementary notes. (Supplementary note 1) 　　An action analysis device comprising an analysis unit and a visualization unit, 　　wherein the analysis unit comprising: 　　a detection unit for detecting a person or a part of the person from video frames in which work of the person is captured; and 　　a tracking unit for tracking the person or the part between the video frames; and 　　the visualization unit acquires a tracking result from a predetermined time before a frame time to the frame time, arranges primitives at the positions of the person or the part included in the acquired tracking result, and superimposes the primitives on a video frame at the frame time, and presents the video frame. (Supplementary note 2) 　　The action analysis device according to Supplementary note 1, wherein 　　each primitive is a figure having directionality, and when a magnitude of a movement of the person or the part between the video frames is larger than a predetermined value, the direction of the figure is aligned with the direction of the movement. (Supplementary note 3) 　　The action analysis device according to Supplementary note 1 or 2, wherein the analysis unit comprising: 　　an action detection unit for analyzing an action of the person or the part based on detection information and position information of the person or the part included in the tracking result; 　　a cycle detection unit for detecting a start and an end of a cycle of work based on an action analysis result by the action analysis unit; 　　a condition generation unit for analyzing a video for each cycle using a cycle detection result, and generates a condition to be used for detecting an event based on the action analysis result; and 　　an event detection unit for detecting the event based on the condition determined by the condition generation unit; and 　　if the event is detected on a frame, the visualization unit changes a method for superimposing the primitive on the frame. (Supplementary note 4) 　　An action analysis device comprising: 　　a detection unit for detecting a person or a part of the person from video frames in which work of the person is captured, and 　　a tracking unit for tracking the person or the part between the video frames, 　　an action detection unit for analyzing an action based on a detection result of the person or the part, and detection information and position information of the person or the part included in a tracking result, 　　a cycle detection unit for detecting a start and an end of a cycle of the work based on an action analysis result; 　　a condition generation unit for analyzing a video for each cycle using the detection result of the cycle and for generating a condition to be used for detecting an event based on the action analysis result; and 　　an event detection unit for detecting the event based on the condition. (Supplementary note 5) 　　The action analysis device according to Supplementary note 3 or 4, wherein 　　the action detection unit extracts a feature value from the detection result of the person or the part, and the detection information and the position information of the person or the part included in the tracking result, recognizes the action based on a pattern of a movement of the person or the part, and / or determines a boundary of a task determined by one or more actions based on a recognition result of the action; 　　the cycle detecting unit detects the cycle by specifying the actions of the start and the end of the cycle from among the recognized actions; 　　the condition generation unit generates the condition based on a statistic value for a plurality of cycles, and 　　the event detecting unit detects the event by comparing the recognized actions for each cycle or the feature value with the condition generated by the condition generation unit. (Supplementary note 6) 　　The action analysis device according to Supplementary note 5, wherein 　　the feature value includes at least one of position, direction, speed, acceleration, degree of directional change of a trajectory of the action, the orientation of the person or the part, and a size of the person or the part, and 　　the statistic value includes at least one of a statistic value of the position of the person or the part, the speed of movement of the person or the part, appearance timing, duration, appearance interval, probability of appearance, and appearance order of the actions. (Supplementary note 7) 　　The action analysis device according to Supplementary notes 3 to 6, wherein 　　the condition generation unit obtains an appearance probability of each action from the action analysis result in a plurality of cycles included in time series data of a certain period, and sets the condition that an action whose probability of appearance is less than a first threshold value is unusual action, and an action whose probability of appearance is greater than a second threshold value is a usual action, and 　　the event detection unit issues an unusual action event when the unusual action is detected in each cycle, and issues a missed action detection event when the usual action is not detected. (Supplementary note 8) 　　The action analysis device according to Supplementary notes 3 to 6, wherein 　　the condition generation unit obtains a first appearance probability of each action from the action analysis result in a plurality of cycles included in time series data for a certain period of time, and sets the condition based on the first appearance probability, and 　　the event detection unit obtains the second appearance probability of each action using data of a predetermined number of cycles obtained from the time series data after the condition is set, and issues an unusual action appearance probability event if the second appearance probability differs from the first appearance probability by a predetermined value. (Supplementary note 9) 　　The action analysis device according to Supplementary notes 3 to 6, wherein 　　the condition generation unit sets the condition based on a first order of actions detected from the action analysis result in a plurality of cycles included in time series data of a certain period, 　　the event detection unit issues an unusual order of action event, if a second order of actions detected in a cycle does not match the first order of actions. (Supplementary note 10) 　　The action analysis device according to Supplementary notes 3 to 6, wherein 　　the condition generation unit obtains a first interval of actions from the action analysis result of a plurality of cycles included in time series data of a certain period, and sets the condition based on the first interval of actions, 　　the event detection unit issues an unusual interval event, if a second interval of actions in a cycle after the condition is set does not match the condition of the first interval of actions. (Supplementary note 11) 　　The action analysis device according to Supplementary notes 3 to 6, wherein 　　the condition generation unit reads a plurality of tasks from a work procedure document, and sets the condition by associating each task with an action included in the task, and 　　the event detection unit issues the event, if an action that is not specified in the condition is detected. (Supplementary note 12) 　　The action analysis device according to Supplementary notes 3 to 6, wherein 　　the condition generation unit calculates a number of occurrences of an action in time series data for each cycle, and sets, as the condition, a usual range of the number of occurrences, and 　　the event detection unit issues the event, if a number of occurrences of the action in a cycle after the condition is set deviates from the usual range. (Supplementary note 13) 　　The action analysis device according to Supplementary notes 3 to 6, wherein 　　the condition generation unit sets, as the condition, a usual work area for a person based on the action detection result of a plurality of cycles included in time series data of a certain period, 　　the event detection unit issues, as the event, an out of work area event if at least one of a first condition, a second condition, and a third condition is satisfied, and 　　the first condition is that the position of the person or the part deviates from the usual work area, 　　the second condition is that the time that the person or the part stays outside the usual work area exceeds a certain time, and 　　the third condition is that the total distance the person or the part moves outside the usual work area exceeds a certain distance. (Supplementary note 14) 　　The action analysis device according to Supplementary notes 3 to 6, wherein 　　the condition generation unit sets, as the condition, a usual work area for each person from the action analysis result of a plurality of cycles included in time series data, 　　the event detection unit issues the event, if a person or the part of the person enters a usual work area of another person after the condition is set. (Supplementary note 15) 　　The action analysis device according to Supplementary notes 3 to 6, wherein 　　the condition generation unit sets, as the condition, a range of usual movement speeds of the person or the part for each action from the action analysis result of a plurality of cycles included in time series data of a certain period of time, and 　　the event detection unit issues the event, if an action for which a movement speed of the person or the part deviates from the range is detected after the condition is set. (Supplementary note 16) 　　The action analysis device according to Supplementary notes 3 to 6, wherein 　　the condition generation unit sets, as the condition, a maximum duration for which the person or the part is allowed to stay in a registered area, and 　　the event detection unit issues the event, if a duration for which the person or the part stays in the registered area exceeds the maximum duration after the condition is set. (Supplementary note 17) 　　The action analysis device according to Supplementary notes 3 to 6, wherein 　　the condition generation unit sets, as the condition, a maximum duration for which the person or the part stops, 　　the event detection unit issues the event, if a duration for which the person or the part is stops exceeds the maximum duration. (Supplementary note 18) 　　The action analysis device according to Supplementary notes 10, 13 to 15, wherein 　　the condition generation unit sets the condition using the time series data of a skilled worker. (Supplementary note 19) 　　The action analysis device according to Supplementary notes 3, 5 to 18, wherein 　　the visualization unit superimposes a mark indicating a time of the event on a seek bar for controlling playback time of the video, and playbacks the video from time position corresponding to the time of the event, if the mark is selected, (Supplementary note 20) 　　The action analysis device according to Supplementary note 19, wherein 　　the visualization unit changes a display parameter of the mark superimposed on the seek bar according to a type or importance of the event. (Supplementary note 21) 　　The action analysis device according to Supplementary notes 3, 5 to 20, wherein 　　the visualization unit highlights a trajectory of the action in which the event is detected. (Supplementary note 22) 　　The action analysis device according to Supplementary note 21, wherein 　　highlighting by the visualization unit includes at least one of increasing the size of the primitives by changing the display parameter, changing color around the primitives, blinking the primitives when the video playbacks, and creating a polygon surrounding the primitives contained in the trajectory and superimposing the polygon on a screen. (Supplementary note 23) 　　The action analysis device according to Supplementary note 13 or 14, wherein 　　the visualization unit superimposes the usual work area on a screen. (Supplementary note 24) 　　The action analysis device according to Supplementary note 23, wherein 　　the visualization unit obtains the polygon surrounding the trajectory of the person or the part in time segment in which the event is issued, overlaps the polygon as an out of work area on the screen, and moves, if the out of work area is selected, playback position of the video to time position corresponding to the time segment. (Supplementary note 25) 　　The action analysis device according to Supplementary notes 1 to 24, wherein 　　the analysis unit analyzes videos of a plurality of persons, and 　　the action analysis unit creates a task time association table in which task start times of a plurality of persons are associated with each other based on boundary information of a task included in the action analysis result. (Supplementary note 26) 　　The action analysis device according to Supplementary note 25, wherein 　　the visualization unit displays the videos of the plurality of persons side by side, displays a mark at the task start time on the seek bar of each video, moves, if the mark of a first task start time of the task on the seek bar of a first video that is a video of a first person is selected, a playback position of the first video to the first task start time, and moves playback position of second video that is a video of a second person to a second task start time of the task that is associated with the first task time in the task time association table, and playback the first and second videos simultaneously. (Supplementary note 27) 　　The action analysis device according to Supplementary note 26, wherein 　　the visualization unit stops the playback of each video when the playback position reaches the end of the task. (Supplementary note 28) 　　The action analysis device according to Supplementary note 26 or 27, wherein 　　the visualization unit further comprises a task list visualization unit for displaying a task list in which a plurality of tasks are arranged in time series order, and 　　when a task in the task list is selected, the visualization unit highlights the task in the task list, gets the task start time of the task of each person based on the task time association table, and moves the playback position of each video to the task start of the video, and playbacks the videos simultaneously. (Supplementary note 29) 　　The action analysis device according to Supplementary note 27, wherein 　　the visualization unit stops the playback of each video when the playback of a video section of the selected task of the video is finished, moves a highlight position in the task list to the next task when the playbacks of the videos of the plurality of persons are finished. (Supplementary note 30) 　　The action analysis device according to Supplementary note 29, wherein 　　the visualization unit overlaps the videos of the same task of the plurality of persons, and 　　the visualization unit comprises a task list visualization unit for displaying a task list in which tasks are arranged in time series order, and 　　when a task in the task list is selected, the visualization unit highlights the task in the task list, gets the task start time of the task of each person based on the task time association table, generates a frame by superimposing video frames of the plurality of persons respectively, each video frame corresponding to a task time after the task start time, and playbacks a video containing the frame. (Supplementary note 31) 　　The action analysis device according to Supplementary note 30, wherein 　　the visualization unit stops the playback of the video when the playback of the video section of the selected task is finished, and highlights the next task in the task list. (Supplementary note 32) 　　The action analysis device according to Supplementary notes 3 to 6, and 18 to 31, wherein 　　the condition generation unit calculates an appearance probability of each action based on the action analysis results of a plurality of cycles of a person, determines a periodicity of the action for the person based on the appearance probability, and sets the condition that the action of which the periodicity is low is detected, and 　　the event detection unit issues the event if the action satisfying the condition is detected. (Supplementary note 33) 　　The action analysis device according to Supplementary notes 3 to 6, and 18 to 31, wherein 　　the condition generation unit calculates an appearance probability of each action based on the action analysis results of a plurality of cycles of a plurality of persons, determines an individuality of the action based on the appearance probability, and sets the condition that the action of which the individuality is high is detected, and 　　the event detection unit issues the event if the action satisfying the condition is detected. (Supplementary note 34) 　　The action analysis device according to Supplementary notes 3, 5 to 25, and 32 to 33, wherein 　　the visualization unit includes a cycle list display unit for displaying a plurality of cycles in time series order, 　　the cycle list display unit displays the plurality of cycles according to a time line that is a common scale among the plurality of cycles, cycle start times of the plurality of cycles are aligned to the origin of the time line, and the cycle list display unit further displays the event according to the time line, and 　　when the event is selected, the visualization unit moves playback position of the video corresponding to the event. (Supplementary note 35) 　　The action analysis device according to Supplementary note 4, wherein 　　the action analysis device further comprises a balance optimization unit, 　　the condition generation unit gets, from the action analysis result and a cycle detection result of the cycle detection unit, at least one of a usual task duration, usual movement speed of the person or the part, and a usual idle time based on the action detection result of a plurality of cycles of usual time included in time series data of a certain period, and sets the condition that the at least one of the usual task duration, the usual movement speed, and the usual idle time, 　　the event detection unit gets at least one of a task duration, movement speed of the person or the part, and an idle time based on the time series data after the condition is set, and issues the event if the task duration, the movement speed, or the idle time differs from the usual task duration, the usual movement speed, or the usual time by a predetermined value, and 　　the balance optimization unit calculates, if the event is issued, a throughput of a line when task assignment of the plurality of persons is changed, determines the task assignment of which the throughput becomes maximum, and generates work assignment information based on the determined task assignment. (Supplementary note 36) 　　The action analysis device according to Supplementary note 35, wherein 　　the event detection unit detects a work object existing between the plurality of persons from a video, and issues the event if the number of the detected object becomes large. (Supplementary note 37) 　　The action analysis device according to Supplementary note 35, wherein 　　the event detection unit compares the idle times of adjacent persons and issues the event when the difference between the idle times becomes large. (Supplementary note 38) 　　The action analysis device according to Supplementary note 35, wherein 　　the action analysis device further comprises a visualization unit comprising: 　　a line balance display unit for generating a screen for presenting the work assignment information to a leader, and 　　a task display unit for generating a screen for presenting work assignment instruction to each worker based on the work assignment information. (Supplementary note 39) 　　The action analysis device according to Supplementary note 38, wherein 　　the task display unit further displays previous work assignment, and highlights a part of the determined task assignment that is changed from the previous work assignment. (Supplementary note 40) 　　The action analysis device according to Supplementary note 38 or 39, wherein 　　the task display unit presents not only the work assignment of the person but also the work assignment of an adjacent person who is adjacent to the person in the line. (Supplementary note 41) 　　The action analysis device according to Supplementary note 4, wherein 　　the action analysis device further comprises a work procedure document update unit and a work procedure document review unit, 　　the condition generation unit calculates a task boundary candidate based of the action detection result in a plurality of cycles included in time series data of a certain period, 　　the work procedure document review unit generates work procedure document update information for correcting time information of each task in the work procedure document based on the task boundary candidate, while displaying the video corresponding to the task in the work procedure document, and 　　the work procedure document updating unit updates the time of the task in the work procedure document based on the work procedure document updating information. (Supplementary note 42) 　　The action analysis device according to Supplementary note 41, wherein 　　the condition generation unit refers to a trajectory pattern of the person or the part that is associated with each task when the work procedure document was generated or updated in the past, and calculates the task boundary candidate by determining similarity between the trajectory pattern and the tracking result of the person or the part of each cycle. (Supplementary note 43) 　　The action analysis device according to Supplementary note 41 or 42, wherein 　　the work procedure document review unit selects at least one candidate section determined by the task boundary candidate information, and edits the time of each task in the work procedure document by associating the at least one candidate section with the task. (Supplementary note 44) 　　The action analysis device according to Supplementary note 43, wherein 　　the work procedure document review unit comprises, on an editing screen, a task list display unit for displaying a task list of tasks included in the work procedure document, a split result presentation unit for presenting a plurality of split segments on a time line generated by splitting the video at the task boundary candidate, and a video playback unit for playing back the video, 　　when a split segment is selected, the video of the split video segment is played back in the video playback unit, and 　　when at least one split segment is selected with a task in the task list being selected, the at least one split section is associated with the task, and total time of the at least one split section is set as the time of the task. (Supplementary note 45) 　　The action analysis device according to Supplementary note 41 or 42, wherein 　　the condition generation unit estimates task name of a split segment generated by splitting the video at the task boundary candidate, based on the action analysis or a trajectory pattern for each task registered when the work procedure document was generated or updated in the past, 　　the work procedure document review unit comprises, on an editing screen, a task list display unit for displaying a task list of tasks included in the work procedure document, and a video playback unit for playing back the video, and 　　the work procedure document review unit displays the estimated task name of the split section on the editing screen while playing back a video segment corresponding to the split section, and generates the work procedure document updating information based on the split section when a predetermined area on the editing screen is selected. (Supplementary note 46) 　　The action analysis device according to Supplementary note 41 or 42, wherein 　　the work procedure document review unit edits the time of the task in the work procedure document by selecting a boundary position of the task from a plurality of task boundary candidates. (Supplementary note 47) 　　The action analysis device according to Supplementary note 46, wherein 　　the work procedure document review unit comprises, on an editing screen, a task list display unit for displaying tasks included in the work procedure document, a video playback unit for displaying the task boundary candidate as a mark on a seek bar, and a split determination input unit that is selected when split positions of the video are determined, and 　　the work procedure document review unit highlights a task in the task list when a video segment corresponding to the task is played back, determines an end position of the task when the split determination input unit is selected, and highlights the next task. (Supplementary note 48) 　　The action analysis device according to Supplementary note 47, wherein 　　a playback speed of the video is slow when playback position is within a predetermined range from the task boundary candidate, and the video playback speed is fast when the playback position is not within the predetermined range from the task boundary candidate. (Supplementary note 49) 　　An action analysis method including: 　　detecting a person or a part of the person from video frames in which work of the person is captured; 　　tracking the person or the part between the video frames; 　　acquiring a tracking result from a predetermined time before a frame time to the frame time, arranges primitives at the positions of the person or the part included in the acquired tracking result, and superimposing the primitives on a video frame at the frame time, and presents the video frame. (Supplementary note 50) 　　A program for causing computer to perform processes including: 　　detecting a person or a part of the person from video frames in which work of the person is captured; 　　tracking the person or the part between the video frames; 　　acquiring a tracking result from a predetermined time before a frame time to the frame time, arranges primitives at the positions of the person or the part included in the acquired tracking result, and superimposing the primitives on a video frame at the frame time, and presents the video frame.

[0172] It will be appreciated by a person skilled in the art that numerous variations and / or modifications may be made to the present disclosure as shown in the specific embodiments without departing from the spirit or scope of this disclosure as broadly described. The present embodiments are, therefore, to be considered in all respects to be illustrative and not restrictive.

[0173] The present application is based upon and claims the benefit of priority from Singapore Patent Application No. 10202303654W, filed on December 27, 2023, the entire contents of which are hereby incorporated by reference.

[0174] 10　　analysis unit 11　　data preparation unit 111　　hand detection unit 112　　object tracking unit 113　　action detection unit 114　　cycle detection unit 12　　condition generation unit 13　　event detection unit] 14　　task association unit 20　　camera 30　　display 40　　action analysis device 50　　video input unit 60　　visualization unit 81　　data storage unit 82　　shift table input unit 83　　line balance optimization unit 84　　line balance display unit 85　　task display unit 91　　data storage unit 92　　work procedure document input unit 93　　work procedure document update unit 94　　work procedure document review unit

Claims

1. An action analysis device comprising an analysis unit and a visualization unit, 　　wherein the analysis unit comprising: 　　a detection unit for detecting a person or a part of the person from video frames in which work of the person is captured; and 　　a tracking unit for tracking the person or the part between the video frames; and 　　the visualization unit acquires a tracking result from a predetermined time before a frame time to the frame time, arranges primitives at the positions of the person or the part included in the acquired tracking result, and superimposes the primitives on a video frame at the frame time, and presents the video frame.

2. The action analysis device according to claim 1, wherein 　　each primitive is a figure having directionality, and when a magnitude of a movement of the person or the part between the video frames is larger than a predetermined value, the direction of the figure is aligned with the direction of the movement.

3. The action analysis device according to claim 1 or 2, wherein the analysis unit comprising: 　　an action detection unit for analyzing an action of the person or the part based on detection information and position information of the person or the part included in the tracking result; 　　a cycle detection unit for detecting a start and an end of a cycle of work based on an action analysis result by the action analysis unit; 　　a condition generation unit for analyzing a video for each cycle using a cycle detection result, and generates a condition to be used for detecting an event based on the action analysis result; and 　　an event detection unit for detecting the event based on the condition determined by the condition generation unit; and 　　if the event is detected on a frame, the visualization unit changes a method for superimposing the primitive on the frame.

4. An action analysis device comprising: 　　a detection unit for detecting a person or a part of the person from video frames in which work of the person is captured, and 　　a tracking unit for tracking the person or the part between the video frames, 　　an action detection unit for analyzing an action based on a detection result of the person or the part, and detection information and position information of the person or the part included in a tracking result, 　　a cycle detection unit for detecting a start and an end of a cycle of the work based on an action analysis result; 　　a condition generation unit for analyzing a video for each cycle using the detection result of the cycle and for generating a condition to be used for detecting an event based on the action analysis result; and 　　an event detection unit for detecting the event based on the condition.

5. The action analysis device according to claim 3 or 4, wherein 　　the action detection unit extracts a feature value from the detection result of the person or the part, and the detection information and the position information of the person or the part included in the tracking result, recognizes the action based on a pattern of a movement of the person or the part, and / or determines a boundary of a task determined by one or more actions based on a recognition result of the action; 　　the cycle detecting unit detects the cycle by specifying the actions of the start and the end of the cycle from among the recognized actions; 　　the condition generation unit generates the condition based on a statistic value for a plurality of cycles, and 　　the event detecting unit detects the event by comparing the recognized actions for each cycle or the feature value with the condition generated by the condition generation unit.

6. The action analysis device according to claim 5, wherein 　　the feature value includes at least one of position, direction, speed, acceleration, degree of directional change of a trajectory of the action, the orientation of the person or the part, and a size of the person or the part, and 　　the statistic value includes at least one of a statistic value of the position of the person or the part, the speed of movement of the person or the part, appearance timing, duration, appearance interval, probability of appearance, and appearance order of the actions.

7. The action analysis device according to claims 3 to 6, wherein 　　the condition generation unit obtains an appearance probability of each action from the action analysis result in a plurality of cycles included in time series data of a certain period, and sets the condition that an action whose probability of appearance is less than a first threshold value is unusual action, and an action whose probability of appearance is greater than a second threshold value is a usual action, and 　　the event detection unit issues an unusual action event when the unusual action is detected in each cycle, and issues a missed action detection event when the usual action is not detected.

8. The action analysis device according to claims 3 to 6, wherein 　　the condition generation unit obtains a first appearance probability of each action from the action analysis result in a plurality of cycles included in time series data for a certain period of time, and sets the condition based on the first appearance probability, and 　　the event detection unit obtains the second appearance probability of each action using data of a predetermined number of cycles obtained from the time series data after the condition is set, and issues an unusual action appearance probability event if the second appearance probability differs from the first appearance probability by a predetermined value.

9. The action analysis device according to claims 3 to 6, wherein 　　the condition generation unit sets the condition based on a first order of actions detected from the action analysis result in a plurality of cycles included in time series data of a certain period, 　　the event detection unit issues an unusual order of action event, if a second order of actions detected in a cycle does not match the first order of actions.

10. The action analysis device according to claims 3 to 6, wherein 　　the condition generation unit obtains a first interval of actions from the action analysis result of a plurality of cycles included in time series data of a certain period, and sets the condition based on the first interval of actions, 　　the event detection unit issues an unusual interval event, if a second interval of actions in a cycle after the condition is set does not match the condition of the first interval of actions.

11. The action analysis device according to claims 3 to 6, wherein 　　the condition generation unit reads a plurality of tasks from a work procedure document, and sets the condition by associating each task with an action included in the task, and 　　the event detection unit issues the event, if an action that is not specified in the condition is detected.

12. The action analysis device according to claims 3 to 6, wherein 　　the condition generation unit calculates a number of occurrences of an action in time series data for each cycle, and sets, as the condition, a usual range of the number of occurrences, and 　　the event detection unit issues the event, if a number of occurrences of the action in a cycle after the condition is set deviates from the usual range.

13. The action analysis device according to claims 3 to 6, wherein 　　the condition generation unit sets, as the condition, a usual work area for a person based on the action detection result of a plurality of cycles included in time series data of a certain period, 　　the event detection unit issues, as the event, an out of work area event if at least one of a first condition, a second condition, and a third condition is satisfied, and 　　the first condition is that the position of the person or the part deviates from the usual work area, 　　the second condition is that the time that the person or the part stays outside the usual work area exceeds a certain time, and 　　the third condition is that the total distance the person or the part moves outside the usual work area exceeds a certain distance.

14. The action analysis device according to claims 3 to 6, wherein 　　the condition generation unit sets, as the condition, a usual work area for each person from the action analysis result of a plurality of cycles included in time series data, 　　the event detection unit issues the event, if a person or the part of the person enters a usual work area of another person after the condition is set.

15. The action analysis device according to claims 3 to 6, wherein 　　the condition generation unit sets, as the condition, a range of usual movement speeds of the person or the part for each action from the action analysis result of a plurality of cycles included in time series data of a certain period of time, and 　　the event detection unit issues the event, if an action for which a movement speed of the person or the part deviates from the range is detected after the condition is set.

16. The action analysis device according to claims 3 to 6, wherein 　　the condition generation unit sets, as the condition, a maximum duration for which the person or the part is allowed to stay in a registered area, and 　　the event detection unit issues the event, if a duration for which the person or the part stays in the registered area exceeds the maximum duration after the condition is set.

17. The action analysis device according to claims 3 to 6, wherein 　　the condition generation unit sets, as the condition, a maximum duration for which the person or the part stops, 　　the event detection unit issues the event, if a duration for which the person or the part is stops exceeds the maximum duration.

18. The action analysis device according to claims 10, 13 to 15, wherein 　　the condition generation unit sets the condition using the time series data of a skilled worker.

19. The action analysis device according to claims 3, 5 to 18, wherein 　　the visualization unit superimposes a mark indicating a time of the event on a seek bar for controlling playback time of the video, and playbacks the video from time position corresponding to the time of the event, if the mark is selected.

20. The action analysis device according to claim 19, wherein 　　the visualization unit changes a display parameter of the mark superimposed on the seek bar according to a type or importance of the event.

21. The action analysis device according to claims 3, 5 to 20, wherein 　　the visualization unit highlights a trajectory of the action in which the event is detected.

22. The action analysis device according to claim 21, wherein 　　highlighting by the visualization unit includes at least one of increasing the size of the primitives by changing the display parameter, changing color around the primitives, blinking the primitives when the video playbacks, and creating a polygon surrounding the primitives contained in the trajectory and superimposing the polygon on a screen.

23. The action analysis device according to claim 13 or 14, wherein 　　the visualization unit superimposes the usual work area on a screen.

24. The action analysis device according to claim 23, wherein 　　the visualization unit obtains the polygon surrounding the trajectory of the person or the part in time segment in which the event is issued, overlaps the polygon as an out of work area on the screen, and moves, if the out of work area is selected, playback position of the video to time position corresponding to the time segment.

25. The action analysis device according to claims 1 to 24, wherein 　　the analysis unit analyzes videos of a plurality of persons, and 　　the action analysis unit creates a task time association table in which task start times of a plurality of persons are associated with each other based on boundary information of a task included in the action analysis result.

26. The action analysis device according to claim 25, wherein 　　the visualization unit displays the videos of the plurality of persons side by side, displays a mark at the task start time on the seek bar of each video, moves, if the mark of a first task start time of the task on the seek bar of a first video that is a video of a first person is selected, a playback position of the first video to the first task start time, and moves playback position of second video that is a video of a second person to a second task start time of the task that is associated with the first task time in the task time association table, and playback the first and second videos simultaneously.

27. The action analysis device according to claim 26, wherein 　　the visualization unit stops the playback of each video when the playback position reaches the end of the task.

28. The action analysis device according to claim 26 or 27, wherein 　　the visualization unit further comprises a task list visualization unit for displaying a task list in which a plurality of tasks are arranged in time series order, and 　　when a task in the task list is selected, the visualization unit highlights the task in the task list, gets the task start time of the task of each person based on the task time association table, and moves the playback position of each video to the task start of the video, and playbacks the videos simultaneously.

29. The action analysis device according to claim 27, wherein 　　the visualization unit stops the playback of each video when the playback of a video section of the selected task of the video is finished, moves a highlight position in the task list to the next task when the playbacks of the videos of the plurality of persons are finished.

30. The action analysis device according to claim 29, wherein 　　the visualization unit overlaps the videos of the same task of the plurality of persons, and 　　the visualization unit comprises a task list visualization unit for displaying a task list in which tasks are arranged in time series order, and 　　when a task in the task list is selected, the visualization unit highlights the task in the task list, gets the task start time of the task of each person based on the task time association table, generates a frame by superimposing video frames of the plurality of persons respectively, each video frame corresponding to a task time after the task start time, and playbacks a video containing the frame.

31. The action analysis device according to claim 30, wherein 　　the visualization unit stops the playback of the video when the playback of the video section of the selected task is finished, and highlights the next task in the task list.

32. The action analysis device according to claims 3 to 6, and 18 to 31, wherein 　　the condition generation unit calculates an appearance probability of each action based on the action analysis results of a plurality of cycles of a person, determines a periodicity of the action for the person based on the appearance probability, and sets the condition that the action of which the periodicity is low is detected, and 　　the event detection unit issues the event if the action satisfying the condition is detected.

33. The action analysis device according to claims 3 to 6, and 18 to 31, wherein 　　the condition generation unit calculates an appearance probability of each action based on the action analysis results of a plurality of cycles of a plurality of persons, determines an individuality of the action based on the appearance probability, and sets the condition that the action of which the individuality is high is detected, and 　　the event detection unit issues the event if the action satisfying the condition is detected.

34. The action analysis device according to claims 3, 5 to 25, and 32 to 33, wherein 　　the visualization unit includes a cycle list display unit for displaying a plurality of cycles in time series order, 　　the cycle list display unit displays the plurality of cycles according to a time line that is a common scale among the plurality of cycles, cycle start times of the plurality of cycles are aligned to the origin of the time line, and the cycle list display unit further displays the event according to the time line, and 　　when the event is selected, the visualization unit moves playback position of the video corresponding to the event.

35. The action analysis device according to claim 4, wherein 　　the action analysis device further comprises a balance optimization unit, 　　the condition generation unit gets, from the action analysis result and a cycle detection result of the cycle detection unit, at least one of a usual task duration, usual movement speed of the person or the part, and a usual idle time based on the action detection result of a plurality of cycles of usual time included in time series data of a certain period, and sets the condition that the at least one of the usual task duration, the usual movement speed, and the usual idle time, 　　the event detection unit gets at least one of a task duration, movement speed of the person or the part, and an idle time based on the time series data after the condition is set, and issues the event if the task duration, the movement speed, or the idle time differs from the usual task duration, the usual movement speed, or the usual time by a predetermined value, and 　　the balance optimization unit calculates, if the event is issued, a throughput of a line when task assignment of the plurality of persons is changed, determines the task assignment of which the throughput becomes maximum, and generates work assignment information based on the determined task assignment.

36. The action analysis device according to claim 35, wherein 　　the event detection unit detects a work object existing between the plurality of persons from a video, and issues the event if the number of the detected object becomes large.

37. The action analysis device according to claim 35, wherein 　　the event detection unit compares the idle times of adjacent persons and issues the event when the difference between the idle times becomes large.

38. The action analysis device according to claim 35, wherein 　　the action analysis device further comprises a visualization unit comprising: 　　a line balance display unit for generating a screen for presenting the work assignment information to a leader, and 　　a task display unit for generating a screen for presenting work assignment instruction to each worker based on the work assignment information.

39. The action analysis device according to claim 38, wherein 　　the task display unit further displays previous work assignment, and highlights a part of the determined task assignment that is changed from the previous work assignment.

40. The action analysis device according to claim 38 or 39, wherein 　　the task display unit presents not only the work assignment of the person but also the work assignment of an adjacent person who is adjacent to the person in the line.

41. The action analysis device according to claim 4, wherein 　　the action analysis device further comprises a work procedure document update unit and a work procedure document review unit, 　　the condition generation unit calculates a task boundary candidate based of the action detection result in a plurality of cycles included in time series data of a certain period, 　　the work procedure document review unit generates work procedure document update information for correcting time information of each task in the work procedure document based on the task boundary candidate, while displaying the video corresponding to the task in the work procedure document, and 　　the work procedure document updating unit updates the time of the task in the work procedure document based on the work procedure document updating information.

42. The action analysis device according to claim 41, wherein 　　the condition generation unit refers to a trajectory pattern of the person or the part that is associated with each task when the work procedure document was generated or updated in the past, and calculates the task boundary candidate by determining similarity between the trajectory pattern and the tracking result of the person or the part of each cycle.

43. The action analysis device according to claim 41 or 42, wherein 　　the work procedure document review unit selects at least one candidate section determined by the task boundary candidate information, and edits the time of each task in the work procedure document by associating the at least one candidate section with the task.

44. The action analysis device according to claim 43, wherein 　　the work procedure document review unit comprises, on an editing screen, a task list display unit for displaying a task list of tasks included in the work procedure document, a split result presentation unit for presenting a plurality of split segments on a time line generated by splitting the video at the task boundary candidate, and a video playback unit for playing back the video, 　　when a split segment is selected, the video of the split video segment is played back in the video playback unit, and 　　when at least one split segment is selected with a task in the task list being selected, the at least one split section is associated with the task, and total time of the at least one split section is set as the time of the task.

45. The action analysis device according to claim 41 or 42, wherein 　　the condition generation unit estimates task name of a split segment generated by splitting the video at the task boundary candidate, based on the action analysis or a trajectory pattern for each task registered when the work procedure document was generated or updated in the past, 　　the work procedure document review unit comprises, on an editing screen, a task list display unit for displaying a task list of tasks included in the work procedure document, and a video playback unit for playing back the video, and 　　the work procedure document review unit displays the estimated task name of the split section on the editing screen while playing back a video segment corresponding to the split section, and generates the work procedure document updating information based on the split section when a predetermined area on the editing screen is selected.

46. The action analysis device according to claim 41 or 42, wherein 　　the work procedure document review unit edits the time of the task in the work procedure document by selecting a boundary position of the task from a plurality of task boundary candidates.

47. The action analysis device according to claim 46, wherein 　　the work procedure document review unit comprises, on an editing screen, a task list display unit for displaying tasks included in the work procedure document, a video playback unit for displaying the task boundary candidate as a mark on a seek bar, and a split determination input unit that is selected when split positions of the video are determined, and 　　the work procedure document review unit highlights a task in the task list when a video segment corresponding to the task is played back, determines an end position of the task when the split determination input unit is selected, and highlights the next task.

48. The action analysis device according to claim 47, wherein 　　a playback speed of the video is slow when playback position is within a predetermined range from the task boundary candidate, and the video playback speed is fast when the playback position is not within the predetermined range from the task boundary candidate.

49. An action analysis method including: 　　detecting a person or a part of the person from video frames in which work of the person is captured; 　　tracking the person or the part between the video frames; 　　acquiring a tracking result from a predetermined time before a frame time to the frame time, arranges primitives at the positions of the person or the part included in the acquired tracking result, and superimposing the primitives on a video frame at the frame time, and presents the video frame.

50. A program for causing computer to perform processes including: 　　detecting a person or a part of the person from video frames in which work of the person is captured; 　　tracking the person or the part between the video frames; 　　acquiring a tracking result from a predetermined time before a frame time to the frame time, arranges primitives at the positions of the person or the part included in the acquired tracking result, and superimposing the primitives on a video frame at the frame time, and presents the video frame.

Images