Information processing apparatus, information processing method, and program

Abstract

To provide an information processing method capable of shortening the time required for investigating a factor of low recognition accuracy of a learned model.SOLUTION: An information processing method includes assigning, by a processor, a classification label selected from a plurality of classification labels representing types of work to each of a plurality of sections in a moving image obtained by capturing an image of a worker according to an input to a user interface, and performing, by the processor, statistical analysis on the classification labels assigned to the plurality of sections and outputting an analysis result to the user interface.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, a technology has been developed that uses a trained model to recognize work types from videos obtained by filming workers. The trained model is generated using training data that includes frames of the video and classification labels that represent the work types. As an annotation device for creating training data, International Publication No. 2022 / 234692 (Patent Document 1) discloses an information processing device that adds information tags related to factory work to target sections of video data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2022 / 234692 Summary of the Invention [Problem to be solved by the invention]

[0004] FIG. 15 is a flowchart showing an outline of the flow of conventional task recognition processing. Task recognition processing generally includes step S101 of selecting a training video, step S102 of creating multiple classification labels representing task types, and step S103 of assigning classification labels to each section of the training video. Steps S101 to S103 generate training data. The task recognition processing also includes step S104 of generating a trained model through machine learning using the training data. The task recognition processing further includes step S105 of inputting the recognition target video into the generated trained model to recognize the task type of each section of the recognition target video, and step S106 of visualizing the recognition results.

[0005] The user evaluates the recognition accuracy of the trained model based on the recognition results. The recognition accuracy depends on the granularity of multiple classification labels, the accuracy of the assigned classification labels, various parameters during machine learning, and the like. Therefore, if the recognition accuracy of the trained model is low, the user must check the appropriateness of the processing content of steps S102, S103, and S104 to investigate the cause. As a result, it takes time to investigate the cause of the low recognition accuracy.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and its purpose is to provide an information processing device, an information processing method, and a program that can reduce the time required to investigate the causes of low recognition accuracy of a trained model. [Means for solving the problem]

[0007] According to an example of the present disclosure, an information processing device includes an annotation unit and an analysis unit. The annotation unit assigns a first classification label selected from a plurality of first classification labels representing task types to each of a plurality of sections in a video obtained by filming a worker in accordance with an input to a user interface. The analysis unit performs statistical analysis on the first classification labels assigned to the plurality of sections and outputs the analysis results to the user interface.

[0008] According to this disclosure, a user can check the appropriateness of the assignment of multiple first classification labels or first classification labels based on the analysis results, and appropriately review inappropriate processing. Therefore, if the recognition accuracy of a trained model generated using first classification labels assigned to each section is low, the user does not need to check the details of the processing by the annotation unit to investigate the cause. Alternatively, the user can simply recheck the analysis results. This allows the user to reduce the time it takes to investigate the cause of the low recognition accuracy of the trained model.

[0009] In the above disclosure, the analysis results include a first stacked graph that stacks the time for sections to which each of the multiple first classification labels is assigned, and a second stacked graph that stacks the standard time required for work of the work type corresponding to each of the multiple first classification labels.

[0010] According to this disclosure, a user can identify a classification label for which the stacked times between the first stacked graph and the second stacked graph are significantly different, and as a result, the user can realize that there is an error in the assignment of the identified classification label.

[0011] In the above disclosure, the analysis result includes a graph representing, for each of the plurality of first classification labels, the variation in the duration of the interval to which the first classification label is assigned.

[0012] According to this disclosure, by checking the graph, a user can identify classification labels with relatively large variations in time length. This allows the user to consider deleting, splitting, or merging the identified classification labels with other classification labels. Alternatively, by investigating the appropriateness of the assignment of the identified classification labels, the user can become aware of errors in the assignment of the classification labels.

[0013] In the above disclosure, the analysis unit outputs a Gantt chart to a user interface, the Gantt chart having a first axis indicating the plurality of first classification labels and a second axis indicating time, and the Gantt chart visualizes each of the plurality of intervals at a corresponding position.

[0014] According to this disclosure, by checking the Gantt chart, a user can easily identify sections to which classification labels have been assigned in an order that differs from the standard procedure.

[0015] In the above disclosure, the information processing device further includes a conversion unit that converts a first classification label assigned to each of a plurality of sections into a corresponding second classification label among a plurality of second classification labels based on a correspondence relationship between the plurality of first classification labels and a plurality of second classification labels representing task types. The granularity of the plurality of second classification labels differs from that of the plurality of first classification labels. The information processing device further includes a generation unit that performs machine learning using training data indicating each frame of the plurality of sections and the corresponding second classification label to generate a trained model that receives input of a frame of a video to be recognized and outputs a second classification label among the plurality of second classification labels into which the frame of the video to be recognized is classified.

[0016] According to this disclosure, a user can selectively use first classification labels suitable for annotation and second classification labels suitable for generating a trained model.

[0017] In the above disclosure, the conversion unit outputs to the user interface a first timeline on which first classification labels assigned to a plurality of sections are visualized in chronological order, or a second timeline on which second classification labels corresponding to a plurality of sections are visualized in chronological order, and the conversion unit switches the display of the first timeline or the second timeline to the other in the user interface in response to a switching instruction input to the user interface.

[0018] According to this disclosure, the user can easily understand the first classification label and the second classification label corresponding to each section.

[0019] In the above disclosure, the annotation unit can access work procedure information indicating a standard work procedure. The work procedure information indicates that a second type of work is performed for a standard time after a first type of work. The annotation unit assigns a first classification label representing a second type of work from among the multiple first classification labels to a section having a standard time after the target section based on the work procedure information, in response to assigning a first classification label representing a first type from among the multiple first classification labels to the target section based on the work procedure information. This disclosure allows a user to reduce the effort required to assign first classification labels.

[0020] In the above disclosure, the analysis unit extracts a notable section that satisfies a predetermined abnormal condition from among a plurality of sections, and notifies the user of the notable section via a user interface.

[0021] According to this disclosure, the user can easily notice that there is an error in assigning the first classification label to the section of interest.

[0022] According to one example of the present disclosure, an information processing method includes a processor assigning a classification label selected from a plurality of classification labels representing work types to each of a plurality of sections in a video obtained by filming a worker in accordance with an input to a user interface, and the processor performing a statistical analysis of the classification labels assigned to the plurality of sections and outputting the analysis results to the user interface.

[0023] According to yet another example of the present disclosure, a program causes a computer to execute the above information processing method.

[0024] These disclosures also enable users to reduce the time it takes to investigate the causes of low recognition accuracy in trained models. [Effects of the Invention]

[0025] According to the present disclosure, users can reduce the time it takes to investigate the causes of low recognition accuracy of a trained model. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a diagram illustrating an example of a system to which an information processing device according to an embodiment is applied; [Figure 2] 2 is a flowchart showing an outline of a processing flow in the system shown in FIG. 1. [Figure 3] 2 is a diagram illustrating an example of a main functional configuration of the task recognition PC illustrated in FIG. 1. FIG. [Figure 4] 2 is a diagram showing an example of an annotation window generated by the annotation tool shown in FIG. 1. FIG. [Figure 5] FIG. 10 is a diagram illustrating an example of a plurality of classification labels. [Figure 6] FIG. 2 is a diagram illustrating an example of annotation data. [Figure 7] FIG. 10 is a diagram illustrating an example of switching the display of a timeline. [Figure 8] FIG. 10 shows an example of a window containing analysis results. [Figure 9] FIG. 10 shows another example of a window containing analysis results. [Figure 10] FIG. 10 is a diagram showing an example of a window that supports checking the appropriateness of the assignment of classification labels. [Figure 11] FIG. 10 is a diagram illustrating an example of a Gantt chart output to a user interface. [Figure 12] FIG. 10 is a diagram showing another example of a window that supports checking the appropriateness of the assignment of classification labels. [Figure 13] FIG. 10 is a diagram showing an example of a screen for notifying a section of interest. [Figure 14] FIG. 10 is a diagram illustrating a function of automatically assigning classification labels. [Figure 15] 1 is a flowchart showing the flow of a conventional task recognition process. DETAILED DESCRIPTION OF THE INVENTION

[0027] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, the same or corresponding parts are designated by the same reference numerals and the description thereof will not be repeated.

[0028] §1 Application Examples FIG. 1 is a diagram illustrating a schematic example of a system to which an information processing device according to an embodiment is applied. The system 1 illustrated in FIG. 1 includes a production site 10, an equipment management server 20, an activity recognition personal computer (hereinafter referred to as an "activity recognition PC 30"), and a visualization server 40. The activity recognition PC 30 is an example of an "information processing device" of the present disclosure. Each of the equipment management server 20, the activity recognition PC 30, and the visualization server 40 may be configured with one or more computers, and may include a virtual machine or a container constructed in a cloud environment, or a configuration consisting of at least a part of these.

[0029] One or more pieces of equipment 11 and one or more network cameras 12 are installed at the production site 10. The one or more pieces of equipment 11 include, for example, manufacturing equipment, sensors, conveyance equipment, etc. The one or more pieces of equipment 11 output equipment signals indicating their operating status to the equipment management server 20 at predetermined control cycles.

[0030] One or more network cameras 12 capture images of workers 13 performing work at production site 10. One or more network cameras 12 output video data obtained by capturing images (hereinafter simply referred to as "video") to task recognition PC 30.

[0031] The equipment management server 20 receives equipment signals from one or more pieces of equipment 11 and manages the values ​​of one or more variables that indicate the state of the one or more pieces of equipment 11. The equipment management server 20 includes an equipment database 21 that stores the values ​​of the one or more variables for each control period.

[0032] The task recognition PC 30 recognizes tasks performed by workers 13 at the production site 10. The task recognition PC 30 is configured by a computer with a general-purpose architecture. The task recognition PC 30 includes, as its main components, a processor 31, a memory 32, and a user interface 38. The processor 31 includes, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The memory 32 includes, for example, a volatile storage device such as a DRAM (Dynamic Random Access Memory). The user interface 38 includes a display 38a and an input device 38b. The input device 38b includes, for example, a keyboard, a mouse, or a touchpad.

[0033] The task recognition PC 30 further includes a recording service 35, an task recognition application 36, and an annotation tool 37 as programs executed by the processor 31. These programs are stored, for example, in a storage device such as a hard disk or in a storage medium. Storage media include volatile storage media, nonvolatile storage media, general-purpose semiconductor storage devices such as Compact Flash (CF) or Secure Digital (SD), magnetic storage media such as Flexible Disks, and optical storage media such as Compact Disk Read Only Memory (CD-ROM). Some or all of the functions provided by these programs may be realized by dedicated hardware circuits (for example, an Application Specific Integrated Circuit (ASIC) or a Field-Programmable Gate Array (FPGA)). The task recognition PC 30 further includes a task recognition database 34.

[0034] The recording service 35 outputs a recording instruction to the network camera 12 and causes the processor 31 to execute an operation of collecting video obtained by shooting in accordance with the recording instruction. The video collected by executing the recording service 35 includes learning video for generating teacher data and recognition target video that is the target of task type recognition. The learning video is output to the annotation tool 37. The recognition target video is output to the task recognition application 36.

[0035] The annotation tool 37 causes the processor 31 to perform operations related to the generation of training data.

[0036] The task recognition application 36 causes the processor 31 to execute a process for generating a trained model, a process for recognizing task types using the trained model, and a process for storing the task type recognition results in the task recognition database 34. The task recognition database 34 may store equipment data indicating some or all of the values ​​of one or more variables for each control period stored in the equipment database 21.

[0037] The visualization server 40 provides a visualization service 41 that visualizes the recognition results obtained by the task recognition PC 30. For example, in response to a request from a user terminal (not shown), the visualization service 41 accesses the task recognition database 34 to acquire data to be visualized (visualization data). The visualization service 41 generates a screen showing the task type recognition results based on the visualization data and outputs the generated screen to the user terminal. This allows the user to check the task type recognition results using the user terminal.

[0038] Fig. 2 is a flowchart showing an outline of the processing flow in the system shown in Fig. 1. The main processing in the system 1 includes steps S1 to S6. Steps S1 to S4 are executed by the processor 31 in accordance with the annotation tool 37 of the task recognition PC 30. Steps S5 and S6 are executed by the processor 31 in accordance with the task recognition application 36. Step S7 is executed by the visualization server 40.

[0039] First, in step S1, the processor 31 selects a learning video obtained by filming the worker 13 in accordance with an input to the user interface .

[0040] In the next step S2, the processor 31 creates a plurality of classification labels representing work types in accordance with input to the user interface 38. The plurality of classification labels are created according to a series of work performed at the production site 10.

[0041] In the next step S3, processor 31 assigns a classification label selected from a plurality of classification labels to each of a plurality of sections in the training video in accordance with input to user interface 38. The assignment of classification labels is also referred to as "annotation." Based on the annotation results, processor 31 generates training data that associates each frame of the training video with the classification label of the section to which the frame belongs.

[0042] In the next step S4, the processor 31 performs a statistical analysis on the classification labels assigned to the multiple sections, and outputs the analysis results to the user interface .

[0043] By checking the analysis results, the user can notice sections to which an incorrect classification label has been assigned. For example, by checking the analysis results showing the time variation of sections to which a certain classification label has been assigned, the user can notice that an error has been made in assigning a classification label to a section that is abnormally longer in duration than other sections.

[0044] Alternatively, the user can consider revising multiple classification labels by checking the analysis results. For example, if the time variance of a section to which a certain classification label is assigned is large, the user can consider deleting the classification label, splitting the classification label into multiple other classification labels, or merging the classification label into one or more other classification labels.

[0045] In this way, the user returns the process to steps S2 and S3 as appropriate depending on the analysis results output in step S4, resulting in the generation of training data suitable for generating a trained model.

[0046] If the user confirms that there are no problems with the analysis results of step S4, the process proceeds to step S5. In step S5, processor 31 generates a trained model by performing machine learning using training data. The trained model receives input frames of a video and recognizes the type of work depicted in the frame. Specifically, the trained model calculates the probability that the frame is classified into each of multiple classification labels and outputs the classification label with the highest probability.

[0047] In the next step S5, processor 31 inputs each frame of the recognition target video into the trained model, and recognizes the task type of the task shown in each frame based on the output of the trained model. The recognition result is stored in task recognition database 34 in association with the frame capture time.

[0048] In the next step S6, the visualization service 41 of the visualization server 40 provides the user terminal with a screen that visualizes the recognition results stored in the task recognition database 34 in response to a request from the user terminal.

[0049] According to the system 1 of this embodiment, a statistical analysis is performed on the classification labels assigned to multiple sections, and the analysis results are output. As a result, as described above, the process is returned to steps S2 and S3 as appropriate depending on the analysis results. As a result, training data suitable for generating a trained model is generated. As a result, if the user determines that the recognition accuracy of the trained model is low based on the recognition result provided in step S7, the user only needs to check the processing content of step S5 to investigate the cause, and does not need to check the processing content of steps S2 and S3 in detail. Alternatively, the user can simply recheck the analysis results output in step S4. This allows the user to reduce the time it takes to investigate the cause of low recognition accuracy.

[0050] §2 Specific examples <Functional configuration of task recognition PC> 3 is a diagram showing an example of the main functional configuration of an activity recognition PC. The activity recognition PC 30 includes a video collection unit 301, a label creation unit 302, an annotation unit 303, a conversion unit 304, an analysis unit 305, a teacher data output unit 306, a generation unit 307, and a recognition unit 308. The video collection unit 301 is realized by the processor 31 executing a recording service 35. The label creation unit 302, the annotation unit 303, the conversion unit 304, the analysis unit 305, and the teacher data output unit 306 are realized by the processor 31 executing an annotation tool 37. The generation unit 307 and the recognition unit 308 are realized by the processor 31 executing an activity recognition application 36.

[0051] (Video Collection Department) The video collection unit 301 outputs a recording instruction to the network camera 12 and collects video obtained by capturing images in accordance with the recording instruction. The video collection unit 301 may output a recording instruction at a predetermined cycle. Alternatively, the video collection unit 301 may output a recording instruction in response to an input to the user interface 38. The collected video is used as either a learning video or a recognition target video. Specifically, the video collected in the learning phase is used as the learning video. The video collected in the operation phase after the learning phase is used as the recognition target video.

[0052] (Annotation window) 4 is a diagram showing an example of an annotation window generated by the annotation tool 50. The annotation window 50 shown in FIG.

[0053] The annotation window 50 includes input areas 51 to 54, a display area 55, a seek bar 56, a timeline 57, and buttons 58 and 59. The input area 51 is used for creating classification labels and annotation. The input area 52 is used for selecting a training video. The input area 53 is used for playing the training video. The input area 54 is used for annotation. The display area 55 is used for playing the training video. The seek bar 56 indicates the playback position in the training video. The timeline 57 shows the annotation results. The button 58 is used as a trigger for saving the annotation results. The button 59 is used as a trigger for starting statistical analysis.

[0054] (Label Creation Department) The label creation unit 302 shown in Fig. 3 creates a plurality of classification labels representing work types in accordance with input to the input area 51 shown in Fig. 4. The label creation unit 302 may create a plurality of classification labels in association with work standards defined in Standard Operating Procedures (SOP) (or work standards or work procedure manuals).

[0055] The label creation unit 302 may create two classification label sets with different granularities. Specifically, the label creation unit 302 creates a first classification label set used for annotation and a second classification label set used for generating training data. The first classification label set includes a plurality of first classification labels. The second classification label set includes a plurality of second classification labels. The granularity of the plurality of second classification labels differs from the granularity of the plurality of first classification labels.

[0056] For a user familiar with standard operating procedures, it is easy to assign classification labels corresponding to each task specified in the standard operating procedures to each segment of a video. This reduces the assignment of incorrect classification labels. On the other hand, from the perspective of the recognition accuracy of a trained model, classification labels with a different granularity from the classification labels corresponding to each task specified in the standard operating procedures may be preferred. For example, if a short task lasting a few seconds is specified in the standard operating procedures, the recognition accuracy of the trained model for that task may be low. In such a case, coarse-grained classification labels are suitable for generating a trained model. Therefore, the label creation unit 302 creates, for example, a classification label set in accordance with the task standard as a first classification label set, and a classification label set with a coarser granularity than the first classification label set as a second classification label set.

[0057] Fig. 5 is a diagram showing an example of multiple classification labels. Table 310 shown in Fig. 5 is created by label creation unit 302. Table 310 includes columns 311 to 313 indicating standard operating procedures, column 314 indicating a first classification label set, and column 315 indicating a second classification label set.

[0058] Column 311 indicates the order in which a plurality of tasks are performed. Column 312 indicates the name of each task. Column 313 indicates the standard time for each task. Columns 311 to 313 are created in advance based on, for example, a standard operating procedure manual.

[0059] Each of columns 314 and 315 is created according to an input to input area 51 and indicates a plurality of classification labels. The plurality of classification labels indicated by column 314 correspond one-to-one to a plurality of tasks defined in, for example, a standard operating procedure. In the example shown in FIG. 5, the plurality of classification labels indicated by column 314 are treated as a first classification label set used for annotation. The plurality of classification labels indicated by column 315 have a different granularity from the plurality of classification labels indicated by column 314. In the example shown in FIG. 5, the granularity of the plurality of classification labels indicated by column 315 is coarser than the granularity of the plurality of classification labels indicated by column 314. In the example shown in FIG. 5, the plurality of classification labels indicated by column 315 are treated as a second classification label set used for generating training data.

[0060] In each record of table 310, a first classification label written in a field of column 314 corresponds to a second classification label written in a field of column 315. For example, the second classification label "Label_1" corresponds to the first classification labels "Work_a" and "Work_b." In this way, table 310 shows the correspondence between multiple first classification labels and multiple second classification labels.

[0061] (Annotation section) The annotation unit 303 shown in FIG. 3 assigns classification labels to each of the multiple sections of the learning video in accordance with inputs made to the input areas 51 to 54 shown in FIG.

[0062] The annotation unit 303 selects a learning video in accordance with the input to the input area 52. Furthermore, the annotation unit 303 plays the learning video in the display area 55 in accordance with the input to the input area 52. Furthermore, the annotation unit 303 changes the position of the slider 56a of the seek bar 56 in accordance with the frame displayed in the display area 55.

[0063] The annotation unit 303 determines a section in the learning video in response to inputs to the input areas 51 and 54, and assigns a classification label to the section. For example, the annotation unit 303 receives two consecutive label assignment instructions in response to inputs to the input area 54. The annotation unit 303 identifies the section from the frame displayed in the display area 55 when the first label assignment instruction was received to the frame displayed in the display area 55 when the second label assignment instruction was received. The annotation unit 303 assigns the classification label selected in the input area 51 ("Work_b" in the example shown in FIG. 4) to the identified section.

[0064] The annotation unit 303 updates the timeline 57 in response to the assignment of the classification labels. The timeline 57 visualizes the classification labels assigned to multiple sections in chronological order. Specifically, the timeline 57 arranges boxes 57a corresponding to each section in chronological order. Note that, for simplification, in FIG. 4, only one box to which the classification label "Work_c" is assigned is labeled with the symbol "57a." Furthermore, the display format of each box 57a corresponds to the classification label assigned to the corresponding section. In the example shown in FIG. 4, a character identifying the assigned classification label is displayed in each box 57a.

[0065] A series of tasks are usually performed repeatedly at the production site 10. The timeline 57 shown in Fig. 4 indicates that tasks of three task types corresponding to the classification labels "Work_a," "Work_b," and "Work_c" are repeated four times.

[0066] The width of the box 57a represents the time length of the corresponding section. The position of the left side of the box 57a represents the start timing of the corresponding section. The position of the right side of the box 57a represents the end timing of the corresponding section. The annotation unit 303 can accept instructions to move the positions of the left and right sides of each box 57a. In response to receiving an instruction to move the left or right side of the box 57a, the annotation unit 303 changes the start timing or end timing of the corresponding section. This allows the user to adjust the start timing or end timing of each section using the timeline 57.

[0067] Furthermore, the annotation unit 303 may correct the classification label assigned to each section in accordance with an input to the input area 54. This allows the user to correct a classification label that has been assigned in error.

[0068] When the label creating unit 302 creates two classification label sets, the annotation unit 303 assigns the first classification label included in the first classification label set used for annotation.

[0069] The annotation unit 303 generates annotation data indicating the classification labels assigned to each section of the learning video in response to the operation of the button 58. Annotation data 60 is generated for each learning video.

[0070] FIG. 6 is a diagram showing an example of annotation data. FIG. 6 shows annotation data 60 corresponding to the learning video "Video_A." As shown in FIG. 6, the annotation data 60 associates each section with an assigned classification label. Each section is defined by a start timing and an end timing. The start timing and end timing may be represented by the elapsed time from the beginning of the learning video, or may be represented by the shooting time of the corresponding frame.

[0071] (Conversion section) The conversion unit 304 operates when the label creation unit 302 creates two classification label sets. The conversion unit 304 refers to table 310 shown in FIG. 4 and converts the first classification label assigned to each of the multiple sections of the training video into a corresponding second classification label from among multiple second classification labels. For example, the conversion unit 304 refers to table 310 and converts the first classification label "Work_a" into the second classification label "Label_1."

[0072] The conversion unit 304 may switch the timeline 57 of the annotation window 50 in response to a switching instruction to the user interface 38.

[0073] 7 is a diagram showing an example of switching the display of a timeline. The conversion unit 304 displays a first timeline 57b, which visualizes first classification labels assigned to multiple sections of a learning video in chronological order, or a second timeline 57c, which visualizes second classification labels corresponding to multiple sections in chronological order, in the annotation window 50. The conversion unit 304 switches the display from one of the first timeline 57b and the second timeline 57c to the other in response to a switching instruction input to the user interface 38.

[0074] (Analysis Department) In response to the operation of button 58, analysis unit 305 performs a statistical analysis of the classification labels assigned to multiple sections of the learning video, and outputs the analysis results to user interface 38.

[0075] 8 is a diagram showing an example of a window containing the analysis results. The analysis results window 70 shown in FIG. 8 is generated by the analysis unit 305 and displayed on the display 38a of the user interface 38.

[0076] The analysis result window 70 includes a first stacked graph 71 that stacks the time for sections assigned with each of a plurality of classification labels, and a second stacked graph 72 that stacks the standard time required for work of the work type corresponding to each of the plurality of classification labels.

[0077] The analysis unit 305 may generate the second stack graph 72 based on the column 313 included in the table 310. The analysis unit 305 preferably generates the second stack graph 72 according to the number of times a series of tasks in the learning video is repeated. That is, the analysis unit 305 generates the second stack graph 72 by stacking up the time obtained by multiplying the standard time described in the column 313 by the number of repetitions.

[0078] The analysis unit 305 may determine the number of repetitions based on, for example, equipment data stored in the task recognition database 34. The multiple variables whose values ​​are indicated by the equipment data may include a trigger variable that indicates a trigger for starting a series of tasks. The analysis unit 305 may determine, as the number of repetitions, the number of times the value of the trigger variable changed during the filming period of the training video. Alternatively, the analysis unit 305 may determine, as the number of repetitions, the number of sections in the training video to which a specific classification label is assigned.

[0079] The user can identify a classification label for which the stacked times are significantly different between first stacked graph 71 and second stacked graph 72. As a result, the user can notice that there is an error in the assignment of the identified classification label.

[0080] 9 is a diagram showing another example of a window containing analysis results. The analysis results window 73 shown in FIG. 9 is generated by the analysis unit 305 and displayed on the display 38a of the user interface 38.

[0081] The analysis result window 73 includes a box plot 74 as an example of a graph representing the variation in the duration of sections to which a classification label is assigned for each of a plurality of classification labels. Note that the graph representing the variation in the duration of sections to which a classification label is assigned is not limited to the box plot 74, and may be a graph in another format (for example, a histogram).

[0082] The user can identify classification labels with relatively large variations in time length by checking the box plot 74. For example, the user may determine that it is difficult to generate a trained model that can accurately recognize the task type corresponding to the identified classification label, and may consider deleting the identified classification label, splitting it, or merging it with another classification label.

[0083] Alternatively, the user can correct an error in the assignment of a classification label by investigating whether the assignment of the identified classification label is appropriate. For example, if one whisker in the box-and-whisker plot 74 is extremely long, the user can notice that the start timing or end timing of the section corresponding to the end of the whisker is incorrect.

[0084] 10 is a diagram showing an example of a window that supports checking the appropriateness of the assignment of classification labels. As shown in Fig. 10, the analysis unit 305 outputs to the user interface 38 a pop-up window 78 that displays a section corresponding to a specified point 90 in the box-and-whisker plot 74. This allows the user to easily check the appropriateness of the classification label assigned to the section corresponding to point 90, or the appropriateness of the start or end timing of the section corresponding to point 90.

[0085] Furthermore, the analysis unit 305 may output a Gantt chart having a first axis showing multiple classification labels and a second axis showing time to the user interface 38. On the first axis, the multiple classification labels are arranged according to the procedures defined in the standard operating procedures.

[0086] FIG. 11 is a diagram showing an example of a Gantt chart output to a user interface. The chart window 75 shown in FIG. 11 includes a button 76 for selecting a learning video and a Gantt chart 77 that visualizes multiple sections of the selected learning video at their corresponding positions. In the Gantt chart 77, the vertical axis represents multiple classification labels, and the horizontal axis represents time. By checking the Gantt chart 77, a user can easily identify sections in which classification labels have been assigned in an order that differs from the procedure specified in the standard operating procedure. The user can then investigate whether the assignment of classification labels to the identified sections is appropriate.

[0087] 12 is a diagram showing another example of a window that supports checking the appropriateness of the assignment of classification labels. As shown in Fig. 12, the analysis unit 305 outputs a pop-up window 79 to the user interface 38, in which a specified section 92 in a Gantt chart 77 is displayed. This allows the user to easily check the appropriateness of the classification label assigned to the specified section 92.

[0088] (Teacher data output section) The training data output unit 306 outputs training data according to the annotation results. Typically, the training data output unit 306 generates training data indicating each frame of the training video and the classification label assigned to the section to which the frame belongs. Note that when the label creation unit 302 creates a first classification label set and a second classification label set, the training data output unit 306 generates training data indicating each frame of the training video and the second classification label converted from the first classification label assigned to the section to which the frame belongs.

[0089] (Generation part) The generation unit 307 generates a trained model by performing machine learning using the training data output from the training data output unit 306. The trained model receives a frame as input and recognizes the classification label into which the frame is classified from among multiple classification labels. Note that if the label creation unit 302 creates two classification label sets, the generation unit 307 receives a frame as input and generates a trained model that outputs the second classification label into which the frame is classified from among multiple second classification labels.

[0090] (recognition part) The recognition unit 308 inputs each frame of the video to be recognized into the trained model, and recognizes the type of work shown in each frame based on the output of the trained model.

[0091] <Modification> (Variation 1) The analysis unit 305 may extract sections from multiple sections of the learning video that satisfy predetermined abnormal conditions as noteworthy sections in which either the assigned classification label, start timing, or end timing is incorrect, and notify the user interface 38 of the noteworthy sections.

[0092] The predetermined abnormal condition includes, for example, the following condition (a) or condition (b). Condition (a): The time difference between the standard time for the task type corresponding to the assigned classification label exceeds a threshold. Condition (b): The order of the classification labels assigned to the previous section, the classification labels assigned to the target section, and the classification labels assigned to the subsequent section differs from the procedure specified in the standard operating procedure.

[0093] FIG. 13 is a diagram showing an example of a screen notifying a user of a section of interest. The annotation window 50 shown in FIG. 13 differs from the annotation window 50 shown in FIG. 4 in that it includes a button 80. In response to the operation of the button 80, the analysis unit 305 extracts a section 81 that satisfies the condition (a) as a section of interest and highlights the section 81. This allows the user to check the classification label assigned to the section 81, the start timing of the section 81, or the end timing of the section 81, and correct them as necessary. As a result, more appropriate training data is generated.

[0094] (Variation 2) The annotation unit 303 may have a function of automatically assigning classification labels by accessing columns 311 to 313 of a table 310 shown in FIG.

[0095] FIG. 14 is a diagram illustrating the function of automatically assigning classification labels. As described above, columns 311 to 313 of table 310 correspond to work procedure information indicating standard work procedures. That is, columns 311 to 313 indicate the order in which work of multiple work types is performed and the standard time for each work. For example, columns 311 to 313 shown in FIG. 14 indicate that after work of the work type "circuit board division," work of the work type "component assembly (1)" is performed for a standard time of "10 seconds," and then work of the work type "heat sink assembly" is performed for a standard time of "15 seconds." The work type "circuit board division" is an example of a "first type" in the present disclosure. The work type "component assembly (1)" is an example of a "second type" and also an example of a "first type" in the present disclosure. The work type "component assembly (2)" is an example of a "second type" in the present disclosure.

[0096] The annotation unit 303 automatically assigns classification labels based on columns 311 to 313 as follows: In response to assigning the classification label "Work_a" representing the work type "board division" to section 82 in accordance with input to user interface 38, the annotation unit 303 assigns the classification label "Work_b" representing the work type "component assembly (1)" to section 83, which has a standard time of "10 seconds" after section 82. Furthermore, in response to assigning the classification label "Work_b" representing the work type "component assembly (1)" to section 83, which has a standard time of "15 seconds", the annotation unit 303 assigns the classification label "Work_c" representing the work type "component assembly (2)" to section 84, which has a standard time of "15 seconds" after section 83.

[0097] This saves the user the trouble of assigning classification labels to segments 83 and 84. If the duration of a segment to which a classification label has been automatically assigned is inappropriate, the user can simply check the learning video and adjust the start or end timing of the segment.

[0098] (Variation 3) In the above explanation, an example was given in which the granularity of the second classification label set is coarser than that of the first classification label set. However, the granularity of the second classification label set may be finer than that of the first classification label set. For example, the classification label set described in column 315 of table 310 may be created as the first classification label set, and the classification label set described in column 314 may be created as the second classification label set. In this case, conversion unit 304 converts the classification labels included in the first classification label set into classification labels included in the second classification label set based on columns 312 to 314 of table 310.

[0099] For example, the conversion unit 304 identifies the tasks "Board Dividing" and "Component Assembly (1)" corresponding to the first classification label "Label_1" based on column 312. Furthermore, the conversion unit 304 identifies the standard times of "5 seconds" and "10 seconds" for the tasks "Board Dividing" and "Component Assembly (1)," respectively, based on column 313. The conversion unit 304 divides the section assigned with the first classification label "Label_1" into a first sub-section and a second sub-section in accordance with the identified time length ratio of "5:10." The first sub-section is a section having a time length (T × 5 / 15) from the beginning of the section assigned with the classification label "Label_1" (having a time length T). The second sub-section is the remaining section. The conversion unit 304 converts the first classification label "Label_1" assigned to the first sub-section into a second classification label "Work_a" corresponding to the task "Board Dividing." The conversion unit 304 converts the first classification label "Label_1" assigned to the second sub-section into the second classification label "Work_b" corresponding to the work "Part installation (1)".

[0100] §3 Supplementary Note As described above, the present embodiment includes the following disclosures.

[0101] (Configuration 1) an annotation unit (31, 303) that assigns a first classification label selected from a plurality of first classification labels representing a type of work to each of a plurality of sections in a video obtained by shooting the worker (13) in accordance with an input to a user interface (38); an analysis unit (31, 305) that performs statistical analysis on the first classification labels assigned to the plurality of sections and outputs the analysis results to the user interface (38).

[0102] (Configuration 2) The information processing device (30) according to configuration 1, wherein the analysis results include a first stacked graph (71) that stacks the times of sections to which each of the plurality of first classification labels is assigned, and a second stacked graph (72) that stacks the standard times required for work of the work types corresponding to each of the plurality of first classification labels.

[0103] (Configuration 3) The information processing device (30) according to configuration 1 or 2, wherein the analysis results include a graph (74) representing, for each of the plurality of first classification labels, the variation in the duration of the sections to which the first classification label is assigned.

[0104] (Configuration 4) the analysis unit (31, 305) outputs a Gantt chart (77) having a first axis indicating the plurality of first classification labels and a second axis indicating time to the user interface (38); 4. The information processing device (30) according to any one of configurations 1 to 3, wherein the Gantt chart (77) visualizes each of the plurality of sections at a corresponding position.

[0105] (Configuration 5) The information processing device (30) further includes a conversion unit (31, 304) that converts the first classification label assigned to each of the plurality of sections into a corresponding second classification label among the plurality of second classification labels based on a correspondence relationship (310) between the plurality of first classification labels and a plurality of second classification labels that represent work types, and the granularity of the plurality of second classification labels is different from the granularity of the plurality of first classification labels, and the information processing device (30) further includes: An information processing device (30) according to any one of configurations 1 to 4, comprising a generation unit (31, 307) that generates a trained model by performing machine learning using training data indicating each frame of the plurality of sections and the corresponding second classification label, and that receives an input of a frame of a video to be recognized and outputs a second classification label among the plurality of second classification labels into which the frame of the video to be recognized is classified.

[0106] (Configuration 6) The conversion unit (31, 304) outputting, to the user interface (38), a first timeline (57b) in which the first classification labels assigned to the plurality of sections are visualized in chronological order, or a second timeline (57c) in which the second classification labels corresponding to the plurality of sections are visualized in chronological order; The information processing device (30) according to configuration 5, wherein the display of the first timeline (57b) and the second timeline (57c) is switched from one to the other in the user interface (38) in response to input of a switching instruction to the user interface (38).

[0107] (Configuration 7) The annotation unit (31, 303) can access work procedure information (311 to 313) that indicates a standard work procedure, The work procedure information (311 to 313) indicates that a second type of work is performed for a standard time after a first type of work, The information processing device (30) according to any one of configurations 1 to 6, wherein the annotation unit (31, 303) assigns a first classification label representing the second type from among the plurality of first classification labels to a section having the standard time after the target section in response to assigning a first classification label representing the first type from among the plurality of first classification labels to the target section based on the work procedure information (311 to 313).

[0108] (Configuration 8) The information processing device (30) according to any one of configurations 1 to 7, wherein the analysis unit (31, 305) extracts a notable section from the plurality of sections that satisfies a predetermined abnormal condition, and notifies the notable section on the user interface (38).

[0109] (Configuration 9) a processor (31) assigning a classification label selected from a plurality of classification labels representing work types to each of a plurality of sections in a video obtained by filming the worker in accordance with an input to a user interface (38); The information processing method comprises the processor (31) performing a statistical analysis on the classification labels assigned to the plurality of sections and outputting the analysis results to the user interface (38).

[0110] (Configuration 10) A program (37) for causing a computer to execute an information processing method, The information processing method includes: assigning a classification label selected from a plurality of classification labels representing work types to each of a plurality of sections in the video obtained by filming the worker in accordance with an input to a user interface (38); and performing a statistical analysis on the classification labels assigned to the plurality of sections and outputting the analysis results to the user interface.

[0111] Although the embodiments of the present invention have been described, the embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0112] 1 System, 10 Production site, 11 Equipment, 12 Network camera, 13 Worker, 20 Equipment management server, 21 Equipment database, 30 Task recognition PC, 31 Processor, 32 Memory, 34 Task recognition database, 35 Recording service, 36 Task recognition application, 37 Annotation tool, 38 User interface, 38a Display, 38b Input device, 40 Visualization server, 41 Visualization service, 50 Annotation window, 51-54 Input area, 55 Display area, 56 Seek bar, 56a Slider, 57 Timeline, 57a Box, 57b First timeline, 57c Second timeline, 58, 59, 76, 80 Button, 60 Annotation data, 70, 73 Analysis result window, 71 First stacked graph, 72 Second stacked graph, 74 Box plot, 75 Chart window, 77 Gantt chart, 78, 79 Pop-up window, 92 specified section, 301 video collection section, 302 label creation section, 303 annotation section, 304 conversion section, 305 analysis section, 306 training data output section, 307 generation section, 308 recognition section, 310 table, 311-315 columns.

Claims

1. an annotation unit that assigns a first classification label selected from a plurality of first classification labels that represent work types to each of a plurality of sections in a video obtained by filming a worker in accordance with an input to a user interface; an analysis unit that performs a statistical analysis on the first classification labels assigned to the plurality of sections and outputs an analysis result to the user interface.

2. 2. The information processing device according to claim 1, wherein the analysis results include a first stacked graph that stacks the times of sections to which each of the plurality of first classification labels is assigned, and a second stacked graph that stacks the standard times required for work of work types corresponding to each of the plurality of first classification labels.

3. The information processing device according to claim 1 , wherein the analysis result includes a graph representing, for each of the plurality of first classification labels, a variation in the duration of the sections to which the first classification label is assigned.

4. the analysis unit outputs a Gantt chart to the user interface, the Gantt chart having a first axis indicating the plurality of first classification labels and a second axis indicating time; The information processing device according to claim 1 , wherein the Gantt chart visualizes each of the plurality of sections at a corresponding position.

5. The information processing device further includes a conversion unit that converts the first classification label assigned to each of the plurality of sections into a corresponding second classification label among the plurality of second classification labels based on a correspondence relationship between the plurality of first classification labels and a plurality of second classification labels that represent task types, wherein the granularity of the plurality of second classification labels is different from the granularity of the plurality of first classification labels, and the information processing device further includes:

5. The information processing device according to claim 1, further comprising: a generation unit that generates a trained model that receives an input of a frame of a video to be recognized and outputs a second classification label, among the plurality of second classification labels, into which the frame of the video to be recognized is classified, by performing machine learning using training data indicating each frame of the plurality of sections and the corresponding second classification label.

6. The conversion unit outputting, to the user interface, a first timeline in which the first classification labels assigned to the plurality of sections are visualized in chronological order, or a second timeline in which the second classification labels corresponding to the plurality of sections are visualized in chronological order; The information processing device according to claim 5 , wherein the display of the first timeline and the second timeline is switched to the other in the user interface in response to an input of a switching instruction to the user interface.

7. the annotation unit is capable of accessing work procedure information indicative of standard work procedures; the work procedure information indicates that a second type of work is to be performed for a standard time after a first type of work, 5. The information processing device according to claim 1, wherein the annotation unit assigns a first classification label representing the second type among the plurality of first classification labels to a section having the standard time after the target section in response to assigning a first classification label representing the first type among the plurality of first classification labels to the target section based on the work procedure information.

8. The information processing device according to claim 1 , wherein the analysis unit extracts a notable section from the plurality of sections that satisfies a predetermined abnormal condition, and notifies the user of the notable section through the user interface.

9. a processor assigning a classification label selected from a plurality of classification labels representing work types to each of a plurality of sections in a video obtained by filming the worker in accordance with an input to a user interface; The information processing method comprises: the processor performing a statistical analysis on the classification labels assigned to the plurality of sections; and outputting the analysis results to the user interface.

10. A program for causing a computer to execute an information processing method, The information processing method includes: assigning a classification label selected from a plurality of classification labels representing work types to each of a plurality of sections in the video obtained by filming the worker in accordance with an input to a user interface; performing a statistical analysis on the classification labels assigned to the plurality of sections, and outputting the analysis results to the user interface.

Images