Work analysis apparatus and work analysis method

Abstract

To enable a user to efficiently check, even when a result of analyzing an operation state of an operator extends over a long time, the analysis result extending over the long time.SOLUTION: An operation analysis device recognizes, on the basis of a video obtained by capturing a state of operation performed by an operator, a target object handled by the operator, recognizes actions constituting the operation performed by the operator to acquire analysis result information including a recognition result of the target object and a recognition result of the actions, and outputs a display portion (overall analysis result display portion 108) of first analysis result display information in which the analysis result information on a first period is visualized and a display portion (detailed analysis result display portion 107) of second analysis result display information in which the analysis result information on a second period included in the first period is visualized while a time axis is extended more than that of the first analysis result display information.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a work analysis apparatus and a work analysis method that execute, by a processor, a process of outputting analysis result information regarding a work situation of a worker based on an image obtained by photographing the state of work of the worker.

Background Art

[0002] In a factory, by analyzing the work situation of a worker and reviewing the work procedure and the like based on the analysis result, it is possible to improve work efficiency. For this reason, a technology that can efficiently analyze the work situation of a worker is desired. Further, a technology that can present analysis result information regarding the work situation of a worker to a user in an easy-to-understand manner is desired.

[0003] As a technology related to such demands, conventionally, an image of a worker working is photographed with a camera, the work situation of the worker is analyzed from the image, and a Gantt chart visualizing the implementation status of work for each process, or a technology for presenting an image in which the state of the worker is photographed to a user is known (see Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the conventional technology, a user can relatively easily grasp the work situation of a worker in each process by browsing while comparing a Gantt chart and an image. On the other hand, when the image in which the state of the worker is photographed extends over a long time, the analysis result of the work situation of the worker also extends over a long time. For this reason, it is desired to be able to efficiently check an analysis result that extends over a long time. However, the conventional technology did not take any consideration for such demands at all.

[0006] Therefore, the main objective of the present invention is to provide a work analysis device and work analysis method that enable users to efficiently review analysis results, even when the analysis of a worker's work status takes a long time. [Means for solving the problem]

[0007] The present invention relates to a work analysis device that uses a processor to perform a process of outputting analysis result information regarding the worker's work status based on video footage of the worker's work, wherein the processor recognizes the object handled by the worker and the actions that constitute the worker's work based on the video footage, acquires the analysis result information including the recognition result of the object and the recognition result of the actions, and outputs first analysis result display information that visualizes the analysis result information for a first period. A first display unit that displays the following, A second analysis result display information is obtained by setting a second period included in the first period and expanding the time axis from the first analysis result display information to visualize the analysis result information. A display screen is generated that includes a second display unit for displaying the first period, a first operation unit for selecting one of a plurality of predetermined periods included in the first period, and a second operation unit for specifying an arbitrary period from the first period. When an arbitrary period is specified as the second period by the second operation unit, the period corresponding to the second period is highlighted in the first analysis result display information. When a predetermined period is selected as the second period by the first operation unit, the specified state of the second operation unit is made to correspond to that predetermined period. This will form the structure.

[0008] Furthermore, the present invention relates to a work analysis method in which a processor performs a process to output analysis result information regarding the worker's work status based on video footage of the worker's work, wherein the method recognizes the object handled by the worker and the actions constituting the worker's work based on the video footage, obtains the analysis result information including the recognition result of the object and the recognition result of the actions, and visualizes the analysis result information for a first period as first analysis result display information. A first display unit that displays the following, A second analysis result display information is obtained by setting a second period included in the first period and expanding the time axis from the first analysis result display information to visualize the analysis result information. A display screen is generated that includes a second display unit for displaying the first period, a first operation unit for selecting one of a plurality of predetermined periods included in the first period, and a second operation unit for specifying an arbitrary period from the first period. When an arbitrary period is specified as the second period by the second operation unit, the period corresponding to the second period is highlighted in the first analysis result display information. When a predetermined period is selected as the second period by the first operation unit, the specified state of the second operation unit is made to correspond to that predetermined period. This will be the structure. [Effects of the Invention]

[0009] According to the present invention, the user, The first period was covered.By viewing the first analysis results, it is possible to get an overview of the workers' work status over a long period of time. Furthermore, the user can, A portion of the first period is set using the first or second operating unit. By viewing the second set of analysis results, the worker's work status can be checked in detail. This allows for detailed analysis of the worker's work status, even if the analysis results are lengthy. Using two control units, This will allow users to efficiently review the results of that lengthy analysis. [Brief explanation of the drawing]

[0010] [Figure 1] Overall configuration diagram of the work analysis system according to this embodiment [Figure 2] Block diagram showing the general configuration of the server. [Figure 3] Block diagram showing the contents of the analysis processing performed on the server. [Figure 4] Diagram illustrating the status of the work cycle detection process performed on the server. [Figure 5] Block diagram showing the contents of the analysis result visualization process performed on the server. [Figure 6] Diagram illustrating the analysis results display screen shown on the administrator's terminal. [Figure 7] This diagram illustrates the process selection section, target period specification section, work video display section, playback operation section, and work cycle selection section of the analysis results display screen. [Figure 8] Detailed diagram showing the analysis results display section of the analysis results display screen. [Figure 9] An explanatory diagram showing the key parts of the detailed analysis result display unit in the event of a work error (abnormality). [Figure 10] An explanatory diagram showing the key parts of the detailed analysis result display unit in the event of a work error (abnormality). [Figure 11] An explanatory diagram showing another example of the detailed analysis results display section. [Figure 12] Diagram illustrating the overall analysis results display section of the analysis results display screen. [Figure 13] An explanatory diagram showing another example of the overall analysis results display section. [Modes for carrying out the invention]

[0011] A first invention made to solve the above problems is a work analysis device that executes, by a processor, a process of outputting analysis result information regarding a worker's work situation based on a video capturing the state of the worker's work. The processor recognizes an object to be handled by the worker based on the video, recognizes actions constituting the worker's work, and obtains the analysis result information including the recognition result of the object and the recognition result of the actions. The processor visualizes the analysis result information for a first period as first analysis result display information. A first display unit that displays the following, The processor sets a second period included in the first period, expands the time axis from the first analysis result display information, and visualizes the analysis result information as second analysis result display information. A display screen is generated that includes a second display unit for displaying the first period, a first operation unit for selecting one of a plurality of predetermined periods included in the first period, and a second operation unit for specifying an arbitrary period from the first period. When an arbitrary period is specified as the second period by the second operation unit, the period corresponding to the second period is highlighted in the first analysis result display information. When a predetermined period is selected as the second period by the first operation unit, the specified state of the second operation unit is made to correspond to that predetermined period. It is configured as such.

[0012] According to this, when the user The first period was covered. views the first analysis result display information, the user can overview the work situation of the worker for a long period. Also, when the user A portion of the first period is set using the first or second operating unit. views the second analysis result display information, the user can check the work situation of the worker in detail. Thus, even when the analysis result of the worker's work situation extends over a long time, Using two control units, the user can efficiently check the analysis result that extends over that long time.

[0013] Also, in a second invention, the processor is configured to output the video so that it can be reproduced in a state associated with the analysis result display information.

[0014] According to this, when the user views while comparing the analysis result display information and the video, the user can easily grasp the work situation of the worker. In this case, the video may be reproduced such that the video is associated with the second analysis result display information presenting the detailed work situation of the worker for the second period. Also, for example, a timeline cursor indicating the current reproduction position may be displayed on the second analysis result display information.

[0015] Furthermore, the third invention is, The display screen further includes a period specification section for inputting the start and end times of the period during which the video footage is stored, in order to specify the first period. This will be the structure.

[0016] According to this, users As information regarding the first period for which the first analysis result display information is displayed, the start time and end time can be entered as text. In this case, the date can be changed by user operation. It may be designated as such.

[0017] Furthermore, the fourth invention is, The first operating unit is a work cycle selection unit that selects one of a plurality of work cycles consisting of a predetermined series of actions included in the first period. This will be the structure.

[0018] According to this, Analysis results for each cycle of a task consisting of a predetermined series of actions This can be easily confirmed using the second analysis result display information. can.

[0019] Furthermore, the fifth invention is, The second operating unit is a slider bar that specifies any period on the time axis of the first analysis result display information included in the first period. The work analysis apparatus according to feature 1. This will be the structure.

[0020] According to this, in the first analysis results display information covering a broad first period, the user... Using a slider bar on the time axis The second period of the second analysis result display information must be appropriately specified. can.

[0021] Furthermore, the sixth invention is configured such that when the processor detects an abnormality in the worker's work status, it outputs the analysis result display information which includes information regarding the abnormality detection result.

[0022] According to this, users can easily check for abnormalities in the worker's work status. In this case, for example, abnormalities related to excesses or deficiencies in actions, the order of actions, and the duration of the work cycle may be detected.

[0023] Furthermore, the seventh invention provides that the second analysis result display information includes, as display information relating to the anomaly detection result, at least one of the following: highlighting of the display field for the target object, displaying a mark indicating an action related to the anomaly, and displaying characters representing the content of the anomaly.

[0024] This allows users to more easily identify anomalies in the workers' work status. In particular, the second analysis result display information, which uses a shorter time axis, displays anomaly detection results on an action-by-action basis, allowing users to check the details of the anomaly.

[0025] Furthermore, the eighth invention is that the first analysis result display information is included in the first period as display information relating to the anomaly detection result. The aforementioned Display of the number of anomaly detections within the specified period. Either the above, or the above-mentioned highlighting of the specified period during which the above-mentioned abnormality was detected. The configuration will include the following:

[0026] According to this, users can more easily identify abnormalities in the workers' work status. In particular, in the first analysis result display information, where the time axis is long, Number of anomaly detections within the specified period This display allows users to easily identify abnormal situations. For example, for each of the multiple work cycles included in the first period. The number of times anomalies have been detected is displayed. It may be considered as such.

[0027] Furthermore, the ninth invention is a work analysis method in which a processor performs a process to output analysis result information regarding the worker's work status based on video footage of the worker's work, wherein the processor recognizes the object the worker is handling and the actions that constitute the worker's work based on the video footage, obtains analysis result information including the recognition result of the object and the recognition result of the actions, and visualizes the analysis result information for a first period as first analysis result display information. A first display unit that displays the following, A second analysis result display information is obtained by setting a second period included in the first period and expanding the time axis from the first analysis result display information to visualize the analysis result information. A display screen is generated that includes a second display unit for displaying the first period, a first operation unit for selecting one of a plurality of predetermined periods included in the first period, and a second operation unit for specifying an arbitrary period from the first period. When an arbitrary period is specified as the second period by the second operation unit, the period corresponding to the second period is highlighted in the first analysis result display information. When a predetermined period is selected as the second period by the first operation unit, the specified state of the second operation unit is made to correspond to that predetermined period. This will be the structure.

[0028] According to this, similar to the first invention, even if the analysis results of analyzing the worker's work situation are over a long period of time, Using two control units, This will allow users to efficiently review the results of that lengthy analysis.

[0029] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0030] Figure 1 is an overall diagram of the work analysis system according to this embodiment.

[0031] This system analyzes the work status of workers in factories and other workplaces and presents the analysis results to the user. The system consists of a camera (1), a recorder (2), a server (work analysis device) (3), and an administrator terminal (4).

[0032] Camera 1 films the workers performing their tasks at their assigned positions.

[0033] Recorder 2 records the video output from camera 1.

[0034] Server 3 acquires video from Recorder 2, performs work analysis based on that video to analyze the workers' work status, and outputs the analysis results. Server 3 is installed in a facility (such as a factory) and performs work analysis on that facility. Alternatively, Server 3 may be configured as a cloud computer and perform work analysis on multiple facilities.

[0035] Administrator terminal 4 is used by system administrators and work managers (users) and consists of PCs, tablet devices, etc. System administrators use administrator terminal 4 to perform various settings related to the processing performed on server 3. Additionally, administrator terminal 4 displays the analysis results output from server 3 for work managers to view.

[0036] Next, we will describe the general configuration of Server 3. Figure 2 is a block diagram showing the general configuration of Server 3.

[0037] Server 3 comprises a video input unit 11, a screen output unit 12, a storage unit 13, and a processor 14.

[0038] The video input unit 11 receives video recorded by the recorder 2.

[0039] The screen output unit 12 outputs the analysis results display screen generated by the processor 14, and the analysis results display screen (see Figure 6) is displayed on the administrator terminal 4.

[0040] The memory unit 13 stores programs executed by the processor 14, etc. The memory unit 13 also stores video footage for analysis acquired from the recorder 2. Furthermore, the memory unit 13 stores the object behavior recognition model (object behavior estimation model) used by the processor 14, the analysis result video generated by the processor 14, and the analysis result information.

[0041] The processor 14 performs various processes by executing programs stored in the memory unit 13. In this embodiment, the processor 14 performs analysis processing P1 and analysis result visualization processing P2, etc.

[0042] In analysis processing P1, the processor 14 analyzes the worker's work status based on video footage of the worker performing the work and generates analysis result information.

[0043] In the analysis result visualization process P2, the processor 14 visualizes the analysis result information acquired in the analysis process P1 and presents it to the user. In this embodiment, an analysis result display screen (see Figure 6) is generated.

[0044] Next, we will explain the analysis process P1 performed on server 3. Figure 3 is a block diagram showing the contents of analysis process P1.

[0045] On server 3, processor 14 performs analysis processing P1. Analysis processing P1 includes video acquisition processing P11, object behavior recognition processing P12, analysis result video generation processing P13, work cycle detection processing P14, anomaly detection processing P15, and output processing P16.

[0046] In the video acquisition process P11, the processor 14 acquires the video to be analyzed from the storage unit 13.

[0047] In the object and action recognition process P12, the processor 14 performs object recognition to recognize the type of object (person, part, and work tool) from the video to be analyzed, and also performs action recognition (action estimation) to recognize the type of action of a person. In action recognition, the person's action is recognized from the person's movement and the type of object the person comes into contact with. In the object and action recognition process P12, the recognition result information obtained includes the action ID assigned to the action, the type of object (target object) associated with the action, and the start and end times of the action. The object and action recognition process P12 can use an object and action recognition model (machine learning model) constructed using machine learning such as deep learning.

[0048] In the analysis result video generation process P13, the processor 14 generates an analysis result video with recognition result information added. Specifically, it generates the analysis result video by superimposing a rectangular frame surrounding the object (person, part, work tool) recognized in the object action recognition process P12, characters representing the type of recognized object, and characters representing the type of recognized action onto the original video being analyzed.

[0049] In the work cycle detection process P14, the processor 14 detects a work cycle (specified period) consisting of a predetermined series of actions from among the worker actions recognized in the object action recognition process P12. In the work cycle detection process P14, the detection result information of the work cycle is obtained as the cycle ID assigned to the work cycle, and the start time and end time of the work cycle.

[0050] In the anomaly detection process P15, the processor 14 detects work errors (work anomalies) by comparing the worker's work procedure with the regular work procedure. Specifically, the processor 14 detects work errors by comparing the actions performed by the worker with the actions included in the regular work procedure for the target work cycle. Work errors include, for example, excessive actions, insufficient actions, and changes in the order of actions. An excessive action is a state in which an action is performed extra at a time when it should not be performed. An insufficient action is a state in which an action that should be performed is not performed.

[0051] Furthermore, in the anomaly detection process P15, the processor 14 detects anomalies in a work cycle by comparing the worker's work time for one work cycle with the standard time. Specifically, for example, if the work cycle time is longer than the standard time by a predetermined limit, such as when the line is temporarily stopped due to some trouble, it is determined to be an anomaly in the work cycle. Also, for example, if the work cycle is canceled midway, it is determined to be an anomaly in the work cycle by a predetermined limit, such as when the work cycle is canceled midway.

[0052] In output processing P16, the processor 14 outputs analysis result information, which includes information about the worker's actions acquired in object action recognition processing P12 (type of action, start time, and end time, etc.), information about the work cycle acquired in work cycle detection processing P14 (start time and end time, etc.), and detection result information from anomaly detection processing P15 (work anomalies (work errors), work cycle anomalies). The analysis result information is stored in the storage unit 13.

[0053] Next, we will explain the work cycle detection process P14 performed on server 3. Figure 4 is an explanatory diagram showing the status of the work cycle detection process P14.

[0054] Server 3 performs a work cycle detection process P14 to detect a work cycle consisting of a predetermined series of actions. Specifically, processor 14 detects a trigger action from among the recognized actions that defines the start timing of the work cycle. The timing at which the trigger action is detected is set as the start timing of the work cycle, and the timing at which the action immediately preceding the next trigger action is detected is set as the end timing of the work cycle.

[0055] Figure 4 shows an example of actions included in one work cycle. In this case, first, as shown in Figure 4(A), the worker grasps the main body with their hand. Next, as shown in Figure 4(B), the worker grasps the screwdriver with their hand. Next, as shown in Figure 4(C), the worker takes the screw out of the parts box. Next, as shown in Figure 4(D), the worker tightens the screw. Next, as shown in Figure 4(E), the worker carries the main body. Next, as shown in Figure 4(F), the worker grasps the main body with their hand.

[0056] Here, as shown in Figures 4(A) and (F), the action of the worker grasping the main body with their hand becomes a trigger action that defines the start timing of the work cycle, and the timing when that trigger action begins is the start timing of the work cycle. Also, the timing when the action immediately preceding the next trigger action is completed is the end timing of the work cycle.

[0057] Furthermore, for each recognized action, the recognition result information includes the action ID assigned to the action, the type of object (target object) associated with the action, and the start and end times of the action. Additionally, the work cycle detection result information includes the cycle ID assigned to the work cycle and the start and end times of the work cycle.

[0058] Next, we will explain the analysis result visualization process P2 performed on server 3. Figure 5 is a block diagram showing the contents of the analysis result visualization process P2.

[0059] On server 3, processor 14 performs analysis result visualization processing P2. Analysis result visualization processing P2 includes video acquisition processing P21, analysis result acquisition processing P22, and screen generation processing P23.

[0060] In the video acquisition process P21, the processor 14 acquires the analysis result video from the storage unit 13. The analysis result video consists of a rectangular frame surrounding the recognized object (person, part, work tool), characters representing the type of recognized object, and characters representing the type of recognized action, superimposed on the original video being analyzed.

[0061] In the analysis result acquisition process P22, the processor 14 acquires analysis result information from the storage unit 13. The analysis result information includes information about the worker's actions (type of action, start time and end time, etc.), information about the work cycle (start time and end time, etc.), and information about the anomaly detection result (work anomaly (work error), work cycle anomaly).

[0062] In screen generation process P23, the processor 14 generates an analysis result display screen (see Figure 6) that visualizes the analysis result information. This analysis result display screen is output from the screen output unit 12 to the administrator terminal 4, and the analysis result display screen is displayed on the administrator terminal 4. In this embodiment, the analysis result display screen displays the analysis result information in a visualized manner and also displays an analysis result video.

[0063] Next, the analysis results display screen shown on the administrator terminal 4 will be described. Figure 6 is an explanatory diagram showing the analysis results display screen. Figure 7 is an explanatory diagram showing the process selection unit 102, target period specification unit 103, work video display unit 104, playback operation unit 105, and work cycle selection unit 106 of the analysis results display screen. Figure 8 is an explanatory diagram showing the detailed analysis results display unit 107 of the analysis results display screen. Figures 9 and 10 are explanatory diagrams showing the main parts of the detailed analysis results display unit 107 when a work error (abnormality) occurs. Figure 11 is an explanatory diagram showing another example of the detailed analysis results display unit 107. Figure 12 is an explanatory diagram showing the overall analysis results display unit 108 of the analysis results display screen. Figure 13 is an explanatory diagram showing another example of the overall analysis results display unit 108.

[0064] As shown in Figure 6, the administrator terminal 4 displays the analysis results display screen 101. The analysis results display screen 101 includes a process selection unit 102, a target period specification unit 103, a work video display unit 104, a playback operation unit 105, a work cycle selection unit 106, a detailed analysis results display unit 107 (a unit for displaying the second analysis results display information), and an overall analysis results display unit 108 (a unit for displaying the first analysis results display information).

[0065] As shown in Figure 7, the process selection unit 102 is equipped with buttons 111 for each process (picking, pressing, welding, assembly, inspection). When a user selects a process by operating a button 111, the analysis results for the selected process are displayed on the analysis results display screen 101.

[0066] In the target period specification unit 103, the user can input the date and time (start time and end time) in text as information about the target period (first period) for which the analysis results will be displayed. In addition, the user can specify any period included in the period during which video is stored in the recorder 2, that is, the period during which work analysis results based on the video can be obtained.

[0067] The work video display unit 104 displays the analysis result video 121. In the analysis result video 121, a rectangular frame 122 surrounding the recognized object (person, part, work tool), a character 123 representing the type of recognized object, and a character 124 representing the type of recognized action are superimposed on the video footage of the worker performing the work. In this example, a rectangular frame 122 and character 123 representing a person, a rectangular frame 122 and character 123 representing a screwdriver, and a character 124 representing the action of tightening a screw are displayed.

[0068] The playback control unit 105 is equipped with a button 131 that serves both as a play and pause button, and a button 132 for double speed.

[0069] The work cycle selection unit 106 is equipped with buttons 141 for each work cycle (1 to 6), a previous skip button 142, and a next skip button 143. When the user selects a work cycle by operating button 141, the analysis results for the selected work cycle are displayed on the work video display unit 104 and the detailed analysis results display unit 107. When the user operates the previous skip button 142, the system moves to the beginning of the next work cycle. When the user operates the next skip button 143, the system returns to the beginning of the currently displayed work cycle.

[0070] As shown in Figure 8, the detailed analysis results display unit 107 is provided with a first action list display unit 201. The first action list display unit 201 visualizes and presents to the user the short-term analysis results regarding the status of actions related to each object. In the first action list display unit 201, action display fields 202 are arranged vertically for each object related to the worker's actions. In this example, action display fields 202 are provided for each object related to the worker's actions, including parts (main body, part #1, part #2, and part #3) and work tools (screwdriver, tweezers, and marker).

[0071] The action display area 202 displays an action bar 203. The action bar 203 is an image that represents the period during which the action was performed. In other words, in the action display area 202, the horizontal axis represents time, and the action bar 203 is drawn within the range corresponding to the period during which the action was performed. This allows the status of actions performed for each type of object to be represented chronologically.

[0072] The action bar 203 displays different patterns depending on the type of action, specifically, actions such as grasping parts or work tools in the hand, or tightening screws with a screwdriver. In this example, the shading pattern (drawing pattern) drawn on the action bar 203 differs depending on the type of action. Alternatively, the fill color of the action bar 203 may also differ depending on the type of action.

[0073] Furthermore, in the first action list display unit 201, if a work cycle is selected in the work cycle selection unit 106, the analysis results for the entire period of that selected work cycle are displayed. Also, if an arbitrary period is specified in the overall analysis result display unit 108, the analysis results for that specified period are displayed in the first action list display unit 201.

[0074] Furthermore, the first action list display unit 201 displays a timeline cursor 204 that indicates the current playback position (playback time) of the analysis result video 121 displayed on the work video display unit 104. The timeline cursor 204 moves horizontally (in the time axis direction) as the playback time of the analysis result video 121 on the work video display unit 104 progresses.

[0075] Furthermore, the detailed analysis results display unit 107 is equipped with a time display unit 205. The time display unit 205 displays the start time, end time, and duration of the range (display period) of the analysis results displayed in the first action list display unit 201. Here, when a work cycle is selected in the work cycle selection unit 106, and the entire duration of one work cycle is displayed in the first action list display unit 201, the displayed duration represents the work time of the selected work cycle.

[0076] Furthermore, in the detailed analysis results display unit 107, if a work error (abnormality) is detected, the action display field 202 related to the object (tool, part) associated with the detected work error is highlighted. Specifically, the display form of the background of the corresponding action display field 202 (for example, the fill color, shading pattern, etc.) changes. Work errors (abnormalities) include situations where actions are excessive, actions are insufficient, or the order of actions is reversed.

[0077] Furthermore, the detailed analysis results display unit 107 displays an alert mark 206 when a work error (abnormality) is detected. The alert mark 206 is displayed in the action display section 202 related to the object involved in the detected work error, at a position corresponding to the timing of the action related to the work error. For example, if an action is insufficient, the alert mark 206 is displayed at the position where that action should have been performed. Also, if an action is superfluous, the alert mark 206 is displayed at the position where that superfluous action was performed.

[0078] Alternatively, instead of, or in addition to, the alert mark 206, the action bar 203 related to the work error may be highlighted by changing its display format. For example, the border of the action bar 203 may be drawn as a dotted line, or the transparency of the action bar 203 may be changed.

[0079] Here, Figures 9(A-1) and (A-2) show examples of actions related to part #3. In this case, the part handling operation consists of the action of grasping the part with the hand. In the worker's procedure shown in Figure 9(A-2), the action of grasping the part with the hand is one step more than in the normal procedure shown in Figure 9(A-1). This is detected as a work error (abnormality), and alert mark 206 is displayed.

[0080] Figures 9(B-1) and (B-2) show examples of actions related to a screwdriver. In this case, the screw-tightening operation consists of the actions of grasping the screwdriver in the hand, taking the screw out of the parts box, and tightening the screw. The actions of taking the screw out of the parts box and tightening the screw are repeated three times. In the worker's procedure shown in Figure 9(B-2), the action of taking the screw out of the parts box is performed one less time compared to the normal procedure shown in Figure 9(B-1). This is detected as a work error (abnormality), and alert mark 206 is displayed.

[0081] Figures 10(A-1) and (A-2) show examples of actions related to markers. In this case, the marking process consists of the actions of grasping the marker with the hand, opening the cap, applying the mark, and closing the cap. In the worker's procedure shown in Figure 10(A-2), the action of closing the cap is missing compared to the standard procedure shown in Figure 10(A-1). This is detected as a work error (abnormality), and alert mark 206 is displayed.

[0082] Furthermore, as shown in Figure 11, the detailed analysis result display unit 107 may be configured to display an abnormality content display unit 211 when a work error (abnormality) is detected. The abnormality content display unit 211 displays characters representing the content of the detected work error. In this case, it displays that there was one extra action related to part #3, one less screw tightening action related to the screwdriver, and that the cap was not closed as an action related to the marker.

[0083] As shown in Figure 12, the overall analysis results display unit 108 is provided with a second action list display unit 301. The second action list display unit 301 visualizes and presents to the user the long-term analysis results regarding the status of actions related to each object. In the second action list display unit 301, similar to the first action list display unit 201, action display fields 302 are arranged vertically for each object related to the worker's actions. Similar to the first action list display unit 201, an action bar 303 is drawn in the action display field 302.

[0084] The second action list display unit 301 displays the analysis results for the period specified in the target period specification unit 103 (target period).

[0085] Furthermore, the overall analysis results display unit 108 is provided with a slider bar 304. The slider bar 304 allows the user to specify an arbitrary period for the analysis results to be displayed on the detailed analysis results display unit 107 by operating the left and right sliders 305. Here, the left slider 305 defines the start time, and the right slider 305 defines the end time. As a result, when a period is specified on the overall analysis results display unit 108, the analysis results for that specified period are displayed on the detailed analysis results display unit 107 in an enlarged state along the time axis.

[0086] Furthermore, when a user selects a work cycle in the work cycle selection unit 106, the left and right sliders 305 move to correspond to the duration (start time and end time) of the selected work cycle, and the analysis results of the selected work cycle are displayed in the detailed analysis results display unit 107.

[0087] Furthermore, the overall analysis results display unit 108 displays a count mark 306 when a work error (abnormality) is detected. The count mark 306 represents the number of times a work error (abnormality) was detected in one work cycle. The count mark 306 is displayed in the display area of ​​the work cycle in which the work error was detected. In this example, one work error was detected in work cycle #1, and three work errors were detected in work cycle #4.

[0088] Furthermore, in the overall analysis results display unit 108, if an abnormality in a work cycle is detected, the work cycle in which the abnormality was detected is highlighted. Specifically, a rectangular image 307 is displayed for the duration of the relevant work cycle. The rectangular image 307 is highlighted, for example, by a fill color or a shading pattern. In this example, an abnormality has been detected in work cycle #4. Specifically, an abnormality in a work cycle refers to a situation where the work cycle time is longer than the standard time by a predetermined limit, or where the work cycle time is shorter than the standard time by a predetermined limit.

[0089] Furthermore, as shown in Figure 13, the overall analysis results display unit 108 highlights the period specified by the slider bar 304, that is, the period during which the analysis results were displayed in the detailed analysis results display unit 107. Specifically, a rectangular image 311 is displayed during the corresponding period. The rectangular image 311 is highlighted, for example, by a fill color or a shading pattern. The rectangular image 311 is displayed in a different format from the rectangular image 307 so that it can be distinguished from the rectangular image 307 that highlights the work cycle in which an anomaly was detected. In this example, work cycle #5 is selected, and the rectangular image 311 is displayed for work cycle #5.

[0090] In this embodiment, the user can arbitrarily specify the period (target period, first period) of the analysis results to be displayed on the overall analysis results display unit 108 using the target period specification unit 103. The user can also arbitrarily specify the period (part of the target period, second period) of the analysis results to be displayed on the detailed analysis results display unit 107 using the slider bar 304. Furthermore, the user can display the analysis results for one work cycle on the detailed analysis results display unit 107 using the work cycle selection unit 106.

[0091] The first period covered by the overall analysis results display unit 108 is the entire target period extracted from the total period during which the worker's work was performed, while the second period covered by the detailed analysis results display unit 107 is a part of that target period. In other words, the time axis scale differs between the detailed analysis results display unit 107 and the overall analysis results display unit 108, and a portion of the analysis results displayed in the overall analysis results display unit 108 is displayed in the detailed analysis results display unit 107 in an enlarged state along the time axis. This allows the user to get an overview of the worker's work status over a wide period by viewing the first analysis results display information. Furthermore, the user can get a detailed view of the worker's work status by viewing the second analysis results display information.

[0092] The specific procedure involves, for example, the user first using the work cycle selection unit 106 to display and view the analysis results for one work cycle on the detailed analysis results display unit 107. If necessary, the user can adjust the period of analysis results displayed on the detailed analysis results display unit 107 using the slider bar 304, thereby viewing analysis results for various ranges. For example, the user can use the slider bar 304 to display only the analysis results for any range of interest within a single work cycle on the detailed analysis results display unit 107. The user can also use the slider bar 304 to display analysis results spanning two or more work cycles on the detailed analysis results display unit 107. Although the explanation focused on work processes in a factory as an example of work status analysis, it can also be applied to work outside of factories, i.e., work that occurs regularly in various facilities (e.g., loading and unloading of goods, cleaning work, patrol security work).

[0093] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments. [Industrial applicability]

[0094] The work analysis device and work analysis method according to the present invention have the effect of enabling users to efficiently review analysis results that have been conducted for a long time, even when the analysis results of analyzing the worker's work situation are long-term, and are useful as a work analysis device and work analysis method that uses a processor to perform a process of outputting analysis result information regarding the worker's work situation based on video footage of the worker's work. [Explanation of Symbols]

[0095] 1 Camera 2 Recorder 3. Server (work analysis device) 4. Administrator terminal 11. Video Input Section 12. Screen output section 13 memory 14 processors 101 Analysis result display screen 102 Process Selection Unit 103 Target Period Specification Section 104 Operation video display unit 105 Playback operation section 106 Work Cycle Selection Section 107 Detailed analysis result display section 108 Overall analysis result display section 141 Work Cycle Select button 201 First Action List Display Section 202 Behavior display field 203 Action Bar 204 Timeline Cursor 206 Alert Mark 211 Abnormality content display section 301 Second Action List Display Section 302 Behavior display field 303 Action Bar 304 Slider Bar 305 Slider 306 count marks

Claims

1. A work analysis device that uses a processor to perform a process that outputs analysis result information regarding the worker's work status based on video footage of the worker's work, The aforementioned processor, Based on the aforementioned video, the system recognizes the object being handled by the worker and the actions constituting the worker's work, and obtains the analysis result information including the recognition result of the object and the recognition result of the actions. A display screen is generated that includes: a first display unit that displays first analysis result display information which visualizes the analysis result information for a first period; a second display unit that sets a second period included in the first period and displays second analysis result display information which visualizes the analysis result information with an expanded time axis compared to the first analysis result display information; a first operation unit for selecting one of a plurality of predetermined periods included in the first period; and a second operation unit for specifying an arbitrary period from the first period. The work analysis apparatus is characterized in that, when an arbitrary period is specified as the second period by the second operating unit, the period corresponding to the second period is highlighted in the first analysis result display information, and when a predetermined period is selected as the second period by the first operating unit, the specified state of the second operating unit is made to correspond to the predetermined period.

2. The aforementioned processor, The work analysis apparatus according to claim 1, characterized in that the aforementioned video is output in a manner that allows playback in association with the aforementioned analysis result display information.

3. The display screen is Furthermore, the work analysis apparatus according to claim 1 is characterized by including a period specification unit for inputting the start time and end time of the accumulation of the video for specifying the first period.

4. The first operating unit is The work analysis apparatus according to claim 1, characterized in that it is a work cycle selection unit that selects one of a plurality of work cycles consisting of a predetermined series of actions included in the first period.

5. The second operating unit is The work analysis apparatus according to claim 1, characterized in that it is a slider bar for specifying an arbitrary period on the time axis of the first analysis result display information included in the first period.

6. The aforementioned processor, The work analysis device according to claim 1, characterized in that when it detects an abnormality in the worker's work status, it outputs analysis result display information including information regarding the abnormality detection result.

7. The second analysis result display information is, as display information relating to the anomaly detection result, Highlighting of the display area for the aforementioned object, Display of a mark indicating the behavior related to the aforementioned abnormality, The work analysis apparatus according to claim 6, characterized in that it includes at least one of the following: a display of characters indicating the nature of the abnormality.

8. The first analysis result display information is, as display information relating to the anomaly detection result, The work analysis apparatus according to claim 6, characterized in that it includes either a display of the number of abnormality detections detected in the specified period included in the first period, or a highlighting of the specified period in which the abnormality was detected.

9. A work analysis method in which a processor performs a process to output analysis result information regarding the worker's work status based on video footage of the worker's work, Based on the aforementioned video, the system recognizes the object being handled by the worker and the actions constituting the worker's work, and obtains the analysis result information including the recognition result of the object and the recognition result of the actions. A display screen is generated that includes: a first display unit that displays first analysis result display information which visualizes the analysis result information for a first period; a second display unit that sets a second period included in the first period and displays second analysis result display information which visualizes the analysis result information with an expanded time axis compared to the first analysis result display information; a first operation unit for selecting one of a plurality of predetermined periods included in the first period; and a second operation unit for specifying an arbitrary period from the first period. A work analysis method characterized in that, when an arbitrary period is specified as the second period by the second operation unit, the period corresponding to the second period is highlighted in the first analysis result display information, and when the specified period is selected as the second period by the first operation unit, the specified state of the second operation unit is made to correspond to the specified period.

Images