Information processing system, information processing method, and program

Abstract

To provide a system, a method, and a program that can reduce a time and an effort for machine learning of a learning model.SOLUTION: A system has: data acquisition means that acquires photographed data obtained by photographing a series of operation including one or more operation steps; registration means that registers operation identification information identifying the operation included in the operation steps; linking means that generates linkage information obtained by linking the operation identification information with a start position and an end position of the photographed data representing a time section of each operation; receiving means that receives the operation identification information of the operation executed in the operation steps; and data generation means that generates learning data for the operation steps. The data generation means acquires a start position and an end position of each operation identified by the operation identification information on the basis of the operation identification information and linkage information, acquires photographed data of a portion corresponding to the start position and the end position from the photographed data, and generates the learning data used for machine learning of a learning model for every operation step on the basis of the acquired photographed data.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing system that performs machine learning of a learning model, an information processing method, and a program for causing a computer to execute the machine learning of the learning model. [Background technology]

[0002] Training data is used for machine learning of a learning model, and the training data is created by performing a process called annotation, in which information (labels) for identifying the target data is added to the target data.

[0003] As a technique for creating learning data by annotation, a technique is known in which sensor data arranged continuously in a time series is divided into time intervals and a label is assigned to each time interval (see, for example, Patent Document 1). Summary of the Invention [Problem to be solved by the invention]

[0004] Learning models are used to confirm the work procedures when assembling products in factories, etc. In order to perform machine learning for such learning models, users need to take pictures of a series of work steps that are made up of multiple work processes and perform annotations from the captured image data.

[0005] However, in the conventional technology, when the work performed in each work process is changed, the photographed data and annotated data created once before the change cannot be reused. Therefore, in order to perform machine learning of the learning model after the work is changed, it is necessary to re-photograph the series of work after the change. Furthermore, every time the work performed in each work process is changed, the user has to repeatedly perform annotation from the photographed photographed data. This causes a problem that it takes time and effort to perform machine learning of the learning model.

[0006] Therefore, it is desirable to provide a system, method, and program that can reduce the effort and time required for machine learning of a learning model. [Means for solving the problem]

[0007] In view of the above problems, the present invention provides an information processing system for performing machine learning of a learning model, comprising: A data acquisition means for acquiring photographed data of a series of operations including one or more work steps; A registration means for registering task identification information for identifying each task included in a task process; a linking means for generating linking information that links a start position and an end position of the shooting data that indicate a time period of each task with task identification information, using the shooting data; A receiving means for receiving work identification information of each work to be performed in each work process; A data generating means for generating learning data for each work process; having The data generating means includes: acquiring a start position and an end position of shooting data representing a time period of each task identified by the task identification information received by the reception means based on the task identification information received by the reception means and the linking information generated by the linking means; From the photographic data acquired from the data acquisition means, photographic data corresponding to a start position and an end position of the photographic data representing a time period of each task identified by the task identification information accepted by the acceptance means is acquired; An information processing system is provided that generates learning data to be used for machine learning of a learning model for each work process based on acquired photographic data. Effect of the Invention

[0008] According to the present invention, it is possible to reduce the effort and time required for machine learning of a learning model. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of annotated data. [Diagram 2] FIG. 1 is a diagram showing an example of allocation of tasks to each process. [Diagram 3] FIG. 1 is a diagram for explaining a method for creating learning data to be used for machine learning of a learning model for detecting tasks within each process. [Figure 4] FIG. 1 is a diagram showing a hardware configuration of an information processing apparatus as an example of an information processing system according to an embodiment of the present invention. [Diagram 5] FIG. 2 is a block diagram showing an example of a functional configuration of an information processing device. [Figure 6] FIG. 4 is a diagram showing an example of a label stored in a label storage unit. [Figure 7] FIG. 4 is a diagram showing an example of work definition information stored in a work definition storage unit. [Figure 8] FIG. 4 is a diagram showing an example of learning data stored in a learning data storage unit. [Figure 9] FIG. 11 is a sequence diagram showing the process flow for registering a video of an assembly operation, a label, operation definition information, and linking information. [Figure 10] A sequence diagram showing the process flow of generating learning data and performing machine learning on a learning model. [Figure 11] This is a sequence diagram showing the process flow for performing machine learning on a learning model by changing the process when learning data has already been generated. [Figure 12] FIG. 11 is a diagram showing an example of a method for creating work definition information. [Figure 13] FIG. 13 is a diagram showing an example of grouping each process. [Figure 14] A diagram showing an example of assigning groups to be used based on a schedule. [Figure 15] FIG. 13 is a diagram showing an example of a screen for registering a new label and deleting or editing a registered label. [Figure 16] FIG. 13 is a diagram showing an example of a screen for adding and deleting a process and adding and deleting an operation. [Figure 17] 13 is a diagram showing an example of a screen for registering linking information that links the start and end positions of a video representing a time period of each task with a label. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The information processing system, information processing method, and program of the present invention are a system, method, and program that perform machine learning of a learning model that infers what tasks a worker performed from data capturing a series of tasks.

[0011] A series of tasks is made up of multiple individual tasks performed consecutively, and includes one or more work processes (hereinafter simply referred to as a process). An example of a series of tasks is assembly work for the same machine in a factory that performs line production. Assembly work may also include inspection after assembly. A process represents the granularity of the assembly work; when one worker performs all the tasks in the assembly work, it is one process, and when the assembly work is shared among two or more workers, it is two or more processes. Below, a series of tasks is described as assembly work in a factory, but a series of tasks is not limited to assembly work in a factory.

[0012] The captured data of an assembly operation is, for example, video data (moving image). A moving image is composed of a plurality of still images captured successively in time. The still images that make up a moving image are called frames. The assembly operation includes a plurality of individual tasks, such as removing a part, attaching a part, removing a screw, removing a screwdriver, and fixing the part with a screw.

[0013] The learning model that infers tasks from video makes it possible to provide machine learning solutions that monitor whether assembly procedures are being followed correctly and measure the time taken for each task, with the aim of using this information as input for improvement.

[0014] To improve the accuracy of detecting tasks, the learning model undergoes machine learning (training) using learning data. The learning model includes algorithms and parameters, and the parameters are adjusted by training. There are three types of machine learning: supervised learning, unsupervised learning, and reinforcement learning. However, for a learning model that infers tasks, supervised learning is used because it is necessary to predict the correct answer for what task it is.

[0015] Teacher data is data in a format in which an example question and its corresponding correct answer are paired. In supervised learning, a learning model is trained so that when example data is input, correct answer data is output. This makes it possible to derive and output data that is expected to be the correct answer for unknown data in the same category as the example question.

[0016] To create such learning data, it is necessary to assign correct answer data to example data. For learning data used in a learning model that infers tasks from video, the time interval from the start to the end of each task performed in the video is determined, and the video of that time interval is created as example data. Then, learning data is created by assigning information (labels) to the example data to identify what task it is as correct answer data. The task of assigning this label is called annotation.

[0017] To train a learning model, multiple videos and each time segment of the videos must be annotated as learning data.

[0018] FIG. 1 is a diagram showing an example of annotated data. The number of frames from the start of shooting of a video is used as frame identification information (frame ID) for identifying each frame, and the start and end positions of the video representing the time period of each task are identified. The start and end positions of the video representing the time period of each task are identified by a start frame (frame_start) ID as the start position of the video representing the time period of the task and an end frame (frame_end) ID as the end position. A label (action) is assigned to the start frame ID and end frame ID of the video representing the time period of each task as task identification information for identifying each task. Meanwhile, "None", which indicates no task, is assigned to a time period in which the worker is not performing a task.

[0019] In the example shown in Fig. 1, the video has about 1242 frames and is classified into 15 time intervals. The time interval from frame 0 to frame 344 is a time interval in which no work is being performed, and is given the label "None." On the other hand, the time interval from frame 345 to frame 386 is a time interval in which the work of removing part A is being performed, and is given the label "remove part A."

[0020] In factories that use production lines, even when assembling the same machines, the number of workers is increased or decreased depending on the increase or decrease in the number of units produced, so the amount of work per person generally changes. Such increases and decreases in the number of units produced are often carried out periodically, about several times a year.

[0021] FIG. 2 is a diagram showing an example of each process and task allocation. FIG. 2(a) is a diagram showing the workload per person when the number of units produced is small, and FIG. 2(b) is a diagram showing the workload per person when the number of units produced is large. Within the assembly work, the range of tasks performed by one worker is defined as one process. In the example shown in FIG. 2(a), there are three processes, and four tasks are performed for each process. On the other hand, in the example shown in FIG. 2(b), there are four processes, and three tasks are performed for each process.

[0022] In the example shown in Figure 2(a), the number of tasks performed by one person is four, which is a large number, so the work time per person is long and the number of units produced is small. On the other hand, in the example shown in Figure 2(b), the number of tasks performed by one person is three, which is a small number, so the work time per person can be reduced and the number of units produced can be increased.

[0023] FIG. 3 is a diagram explaining a method for creating learning data used for machine learning of a learning model for detecting work in each process. The learning model is a learning model that infers the work performed by each worker. Therefore, when a process is changed from the example shown in FIG. 2(a) to the example shown in FIG. 2(b) or vice versa, the range of work performed by each worker changes, so each time a process change occurs, photography must be taken at each process, annotation must be performed, and training must be performed.

[0024] In the example shown in Figure 3, when the new process was introduced, each person was supposed to perform four tasks, but in order to increase the number of units produced, the process was changed to three tasks per person, and one worker was added.

[0025] In creating the learning data, photographed data (video) of each operation is obtained. The video is, for example, 10 cycles, that is, 10 assembly operations. Here, 10 cycles is used, but the number is not limited to 10 cycles, and may be 9 cycles or less, or 11 cycles or more.

[0026] The videos include videos of each task and videos of no task, and these videos are identified by a start frame ID and an end frame ID. The tasks performed by each worker are defined by a task definition as tasks performed within a process. The learning data is created by linking the videos of each task within the process defined by the task definition with a label.

[0027] When introducing a new process, each task in each process needs to be photographed and annotated.

[0028] The learning model is trained using the annotated data. Training the learning model requires photographing and annotating each task in each process every time a change is made, which is time-consuming and labor-intensive. This makes the investment too high for the benefits of introducing the solution.

[0029] Referring to Figure 3, when a process is changed, the photographed data taken when a new process was introduced already exists, labels for identifying each task have been created, and the video of each task has been linked to the label. The only difference between when a new process is introduced and when a process is changed is the task definition. Therefore, when a process is changed, if only the task definition is created, learning data can be created without time or effort, and the learning data can be used to train the learning model.

[0030] The following describes in detail a system, method, and program that allows training of a learning model without such effort and time.

[0031] 4 is a diagram illustrating a hardware configuration of an information processing device as an example of a system that performs machine learning of a learning model. The information processing system according to the present embodiment may be any system that can perform machine learning of a learning model, and is not limited to an information processing device.

[0032] The information processing device 10 is a general computer, and may be, for example, a notebook PC (Personal Computer), a desktop PC, a tablet PC, a wearable PC, a smartphone, a PDA (Personal Digital Assistant), an image forming device such as an MFP (Multi-Function Peripheral), a PJ (Projector), an IWB (Interactive White Board: an electronic whiteboard having a blackboard function capable of mutual communication), etc. Below, the information processing device 10 is assumed to be a PC (server) and its hardware configuration will be briefly described.

[0033] The information processing device 10 includes, as hardware, a CPU (Central Processing Unit) 20, a ROM (Read Only Memory) 21, a RAM (Random Access Memory) 22, a HD (Hard Disk) 23, and a HDD (Hard Disk Drive) controller 24. The information processing device 10 also includes a display 25, an external device connection I / F (Interface) 26, a network I / F 27, a bus line 28, a keyboard 29, a pointing device 30, a DVD-RW (Digital Versatile Disk Rewritable) drive 31, and a media I / F 32.

[0034] The CPU 20 controls the overall operation of the information processing device 10. The ROM 21 stores programs used to drive the CPU 20, such as an IPL (Initial Program Loader). The RAM 22 provides a working area for the CPU 20. The HD 23 stores various data such as programs. The HDD controller 24 controls the reading and writing of various data from and to the HD 23 under the control of the CPU 20.

[0035] The display 25 displays various information such as a cursor, a menu, a window, characters, or an image. The external device connection I / F 26 is an interface for connecting various external devices. The external devices are, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 27 is an interface for communicating via a network. The bus line 28 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 20.

[0036] The keyboard 29 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 30 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 31 controls reading and writing of various data from a DVD-RW 33 as an example of a removable recording medium. The recording medium is not limited to the DVD-RW 33, and may be a DVD-R, a CD-RW, a CD-R, etc. The media I / F 32 controls reading and writing of data from a recording medium 34 such as a flash memory.

[0037] 5 is a block diagram showing an example of a functional configuration of the information processing device 10. The information processing device 10 has a plurality of functions for performing machine learning of the above-mentioned learning model, and each function can be implemented by one or a plurality of processing circuits. The processing circuit includes a processor programmed to execute each function by software such as a CPU implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), a conventional circuit module, or other device designed to execute each function.

[0038] The information processing device 10 includes an annotation tool 40. The annotation tool 40 includes a moving image registration unit 41, a label registration unit 42, a work definition unit 43, and a linking unit 44. In addition to the annotation tool 40, the information processing device 10 also includes a moving image storage unit 45, a label storage unit 46, a work definition storage unit 47, and a linking information storage unit 48.

[0039] The video registration unit 41 functions as a data acquisition means, acquiring shooting data (video) of assembly work from an imaging device or the like, and saving and managing the data in the video storage unit 45. A file name or the like for identifying the video is associated with the video. The label registration unit 42 functions as a registration means, dividing the assembly work into the granularity of individual tasks to be detected, and registering task identification information for identifying each task as a label. The label registration unit 42 saves the task names and the like as labels in the label storage unit 46 and manages them.

[0040] The work definition unit 43 functions as a receiving means, and receives and registers work definition information that defines which work is to be performed in each process of the line production. A work is identified by a label stored in the label memory unit 46. The work definition unit 43 stores and manages the work definition information in the work definition memory unit 47.

[0041] The linking unit 44 functions as a linking means, and links the start and end positions of the videos stored in the video storage unit 45, which represent the time interval of each task, i.e., from which frame to which frame a task was performed, with a label. The linking unit 44 generates linking information that links the start and end positions of the videos that represent the time interval of each task with a label, and stores and manages the information in the linking information storage unit 48. The start and end positions of the videos that represent the time interval of each task are, for example, the start frame ID and end frame ID of the video in which each task was filmed, and the linking information is information that links a label to the start frame ID and end frame ID. The linking information can also be linked to the file name of the video, etc.

[0042] The label storage unit 46 stores a list of labels. The work definition storage unit 47 stores work definition information for each process. The linking information storage unit 48 stores linking information.

[0043] The information processing device 10 further includes a learning data generating unit 49, a learning data storage unit 50, a training unit 51, a learning model storage unit 52, a schedule setting unit 53, and a setting storage unit .

[0044] The learning data generating unit 49 functions as a data generating means, and acquires a start frame ID and an end frame ID of a video that represents a time interval of each task that is associated with a label of each task performed in each process, based on the task definition information received by the task definition unit 43 and the association information generated by the association unit 44 and stored in the association information storage unit 48. The learning data generating unit 49 also functions as an association information acquiring means, and can acquire association information from the association information storage unit 48. The learning data generating unit 49 acquires a part of the video that corresponds to the start frame ID and the end frame ID of the video that represents the time interval of each acquired task, from the video stored by the video registration unit 41. Then, the learning data generating unit 49 generates learning data to be used for machine learning of a learning model for each process, based on the acquired video.

[0045] When generating learning data, the learning data generation unit 49 detects skeletal information including feature points such as joints required to estimate the posture for each frame of the acquired video in order to detect what work the worker is performing by focusing on the worker's posture.

[0046] The learning data generation unit 49 treats the detected skeletal information as one set, assigns a label to the set, and generates learning data for one task. In this way, the learning data generation unit 49 generates learning data for each task in each process, and generates learning data to be used for training the learning model for each process. The learning data generation unit 49 stores the generated learning data in the learning data storage unit 50.

[0047] The training unit 51 functions as a judgment means, a model acquisition means, and a learning means, and refers to the work definition information of the process to be trained, and judges what label of learning data is required for that process. The training unit 51 acquires the training target learning data from the learning data storage unit 50, and trains the learning model of the training target process.

[0048] A trained learning model (a trained learning model) makes it possible to predict what task is being performed from input data and output the predicted result.

[0049] The training unit 51 stores and manages the learned learning model in the learning model storage unit 52. The training unit 51 acquires from the learning model storage unit 52 the learning model of the process to be trained.

[0050] The schedule setting unit 53 sets the production plan for the factory and the learning model to be used that is linked to the plan. That is, the schedule setting unit 53 sets schedule information that defines which learning model is to be used at what time, and stores and manages the schedule information in the setting storage unit 54.

[0051] The training unit 51 also functions as an information acquiring unit, and acquires schedule information stored in the setting storage unit 54. Then, the training unit 51 acquires a corresponding learning model from the learning model storage unit 52 based on the schedule information.

[0052] The information processing device 10 shown in FIG. 5 includes a video storage unit 45, a label storage unit 46, an operation definition storage unit 47, a linking information storage unit 48, a learning data storage unit 50, a learning model storage unit 52, and a setting storage unit 54, but is not limited thereto. Videos, etc. may be stored in other devices, such as a server, other than the information processing device 10, and may be acquired by each acquisition means as necessary. In addition, instead of storing all information, such as videos, in other devices, a part of the information may be stored in one or more storage units in the information processing device 10, other information may be stored in other devices, and the information stored in the other devices may be acquired by one or more acquisition means.

[0053] In addition, in the example shown in FIG. 5, the training unit 51 is configured to have multiple functions such as a judgment means, a model acquisition means, a learning means, and an information acquisition means, but it may also be configured to have only the function as a learning means, and to have a judgment means, a model acquisition means, and an information acquisition means separately provided as a judgment unit, a model acquisition unit, and an information acquisition unit.

[0054] 6 is a diagram showing an example of labels stored in the label storage unit 46. The labels are assumed to be work names, such as "take part A," "attach part A," "take part B," and "attach part B." The labels are saved as a list of work for each process. Note that the labels must be unique in order to identify the work.

[0055] Fig. 7 is a diagram showing an example of work definition information stored in work definition storage unit 47. Work definition information is information that defines what work is to be performed for each process, and the work is defined by a label. In the example shown in Fig. 7, seven works are defined by seven labels in process 1. Six works are defined by six labels in process 2.

[0056] 8 is a diagram showing an example of the learning data stored in the learning data storage unit 50. The label "take part A" corresponds to frames 345 to 386 of the captured data, and based on the linking information, the label "take part A" is assigned to the set of frames 345 to 386, thereby generating learning data.

[0057] Similarly, the label “Attach part A” corresponds to frames 415 to 455 of the captured data, and based on the linking information, the label “Attach part A” is assigned to the set of frames 415 to 455, and learning data is generated.

[0058] The start frame ID and end frame ID of the video representing the time period of each task are set as a set of the smallest and largest frame IDs assigned the same label, such as 345 and 386, or 415 and 455, respectively, and one learning data may be generated for each label, but two or more learning data may also be generated.

[0059] The example shown in FIG. 8 shows an example in which two or more pieces of learning data are generated for each label, and one piece of learning data is generated for a total of 10 frames consisting of three frames before the target frame and six frames after the target frame. In the example shown in FIG. 8, the condition is that the frames in the learning data include seven or more frames belonging to the target label. Note that the learning data shown in FIG. 8 is an example, and is not limited to the configuration shown in FIG. 8. Therefore, the above condition may be a condition that eight or more frames belonging to the target label are included in a total of 10 frames, or a condition that nine or more frames are included. In addition, a condition may be that the number of frames belonging to the target label accounts for 80% or more of a total of 15 or 20 frames.

[0060] FIG. 9 is a sequence diagram showing the flow of a process for registering a video of an assembly work, a label, work definition information, and linking information. The user 60 displays a UI (User Interface) to register a video of an assembly work (S1), and instructs to register the video using the UI (S2). The user 60 designates a video to be registered, and instructs to register the video by pressing a registration button or the like. The user 60 can input a file name and instruct to register the video by linking the file name. The file name may be automatically generated using information such as the location and date and time of the video. The video registration unit 41 receives an instruction from the user 60, registers the video in the video storage unit 45, and saves it (S3). The registration of the video is repeated as many times as the number of videos that have been shot.

[0061] The user 60 displays a UI for registering a label (S4) and instructs the user to obtain a list of registered labels. The label registration unit 42 receives the instruction from the user 60 and obtains the list of registered labels (S5).

[0062] The user 60 creates a label by inputting characters and the like using the UI (S6). The creation of labels is repeated for the number of types of work to be performed in the target process. After creating a label, the user 60 instructs registration of the label by, for example, pressing a registration button (S7). In response to the instruction from the user 60, the label registration unit 42 registers and saves the label in the label storage unit 46 (S8).

[0063] User 60 displays the UI to register work definition information (S9). Work definition unit 43 acquires a list of registered work definition information as a list of defined works from work definition storage unit 47 (S10). Work definition unit 43 also acquires a list of registered labels from label storage unit 46 (S11). Work definition unit 43 displays the acquired list of registered work definition information and list of registered labels on the UI.

[0064] User 60 creates new work definition information for the target process (S12). User 60 assigns tasks to the work definition information, i.e., assigns labels (S13). This allows work definition unit 43 to accept the selection of tasks to be performed in each process. Work assignment is repeated the number of times equal to the number of types of tasks to be performed in the target process. User 60 then uses the UI to instruct work definition unit 43 to register the work definition information (S14). In response to the instruction from user 60, work definition unit 43 registers and saves the work definition information in work definition storage unit 47 (S15).

[0065] User 60 displays the UI to link the start frame ID and end frame ID of a video that represents a time interval of a task with a label (S16). The linking unit 44 acquires a list of registered videos from the video storage unit 45 (S17). The linking unit 44 also acquires a list of registered labels from the label storage unit 46 (S18). The linking unit 44 displays the acquired list of registered videos and the list of registered labels on the UI. User 60 uses the UI to select a video to be linked (S19). In response to user 60's selection, the linking unit 44 previews the selected video.

[0066] User 60 displays a UI to specify a label, and uses the UI to set the label to be assigned (S20). User 60 uses the UI to determine the start position of a video corresponding to the label (S21). The frame ID of a frame indicating the start position of the video is specified as a start frame ID. User 60 uses the UI to determine the end position of the video corresponding to the label (S22). The frame ID of a frame indicating the end position of the video is specified as an end frame ID.

[0067] When the assignment of labels to all frames of the video is completed, the user 60 instructs the linking unit 44 to register linking information linking the labels with the start position (start frame ID) and end position (end frame ID) of the video (S23). In response to the instruction from the user 60, the linking unit 44 registers and saves the linking information in the linking information storage unit 48 (S24).

[0068] The above tasks are annotations, and are always performed once when a new process is introduced.

[0069] 10 is a sequence diagram showing a process flow for generating learning data and training a learning model. A user 60 instructs the learning data generating unit 49 to generate learning data (S1). At this time, the user 60 specifies a target video by inputting or selecting a file name of the video. The learning data generating unit 49 acquires the target video from the video storage unit 45 (S2). The learning data generating unit 49 also acquires association information indicating what work was being performed in the frame of the video from the association information storage unit 48 (S3).

[0070] The learning data generation unit 49 plays the acquired video, performs skeletal inference for each frame, and acquires skeletal information (S4). The learning data generation unit 49 acquires the linking information from the linking information storage unit 48, extracts several frames that represent the same work, sets the skeletal information of the extracted several frames, and assigns a corresponding label to the sets, thereby generating learning data (S5). When the condition that seven or more frames out of a total of ten frames are target frames is set as described above, the frames are extracted so that the number of frames containing skeletal information representing the target work is seven or more. The learning data generation unit 49 registers and saves the learning data in the learning data storage unit 50 (S6).

[0071] The user 60 uses the UI to instruct the training unit 51 to start training the learning model (S7). When instructing the learning model to be trained, the user 60 specifies a process corresponding to the learning model. The training unit 51 receives the instruction from the user 60 and requests acquisition of work definition information of the target process from the work definition storage unit 47 based on the specified process (S8). The training unit 51 retrieves a list of labels assigned to the acquired work definition information (S9). The training unit 51 retrieves the target learning data from the learning data storage unit 50 based on the retrieved label list (S10). That is, the training unit 51 retrieves learning data for the specified process.

[0072] The training unit 51 trains the learning model using the learning data acquired from the learning data storage unit 50 (S11). After the training, the training unit 51 registers and stores the trained learning model in the learning model storage unit 52 (S12).

[0073] Figure 11 is a sequence diagram showing the process flow for performing machine learning of a learning model by changing a process when learning data has already been created. If learning data has already been generated, annotation has already been performed. In other words, since a label has been assigned to each task and learning data has been generated, it is possible to train a learning model simply by creating and saving task definition information that defines which tasks will be performed in a new process.

[0074] User 60 displays a UI for specifying a task and creates work definition information for a target process (S1). User 60 assigns a task to the work definition information (S2). Task assignment is repeated for each type of task performed in the target process. User 60 instructs work definition unit 43 to register the work definition information (S3). In response to the instruction from user 60, work definition unit 43 registers and saves the work definition information in work definition storage unit 47 (S4).

[0075] User 60 uses the UI to instruct training unit 51 to train a learning model (S5). When instructing the learning model to be trained, user 60 specifies a process corresponding to the learning model. Upon receiving the instruction from user 60, training unit 51 acquires work definition information of a target process from work definition storage unit 47 based on the specified process (S6). Training unit 51 extracts a list of labels assigned to the acquired work definition information, and acquires target learning data from learning data storage unit 50 based on the extracted label list (S7).

[0076] The training unit 51 trains the learning model using the learning data acquired from the learning data storage unit 50 (S8). After the training, the training unit 51 registers and stores the trained learning model in the learning model storage unit 52 (S9).

[0077] In this way, when annotation has already been done and learning data has been generated, and only a process change is required, it is only necessary to create and save work definition information, eliminating the need to photograph the assembly work, register labels, link them, and so on, thereby reducing the effort and time required to create a learning model.

[0078] Fig. 12 is a diagram showing an example of a method for creating work definition information. Even if annotation has already been performed and learning data has already been generated, and only a process change is to be made, it is necessary to create work definition information.

[0079] The work definition information is managed, for example, in a table, in order to define the work included in each process. The table 70 is provided with a button 71 for adding a process. The table 70 may also be provided with a button for deleting a process.

[0080] For example, if there are three processes and each process has four operations, table 70 is a table consisting of 3 columns and 4 rows, as shown in Fig. 12(a). When user 60 presses button 71, one process is added, and the table becomes a table consisting of 4 columns and 4 rows, as shown in Fig. 12(b).

[0081] In table 70 shown in Fig. 12(b), user 60 uses input means such as a mouse to move task 4, for example, by dragging and dropping it from the field in process 1 to the field in process 2 as shown in Fig. 12(c). Then, as shown in Fig. 12(d), task 4 moves to the field in process 2, and the number of tasks in process 2 increases by one, making it five in total. Meanwhile, the number of tasks in process 1 decreases by one, making it three in total.

[0082] User 60 divides the total of 12 tasks evenly into four processes, and moves tasks 7 and 8 of process 2 to a field in process 3 so that each process has three tasks, and moves tasks 10, 11, and 12 of process 3 to a field in newly added process 4, as shown in Fig. 12(e). Then, the number of tasks in each of processes 1 to 4 becomes three, as shown in Fig. 12(f).

[0083] The work definition information can be set by a simple operation such as pressing the button 71 and dragging and dropping. Note that this method is just an example, and the creation of the work definition information is not limited to this method.

[0084] 13 is a diagram showing an example in which group names are assigned as model identification information for identifying learning models according to the number of processes. Process changes can be implemented appropriately by user 60, but when there are only a small number of process changes, such as two or three patterns, group names can be assigned according to the number of processes and the learning models can be linked to the group names and stored in learning model storage unit 52. This makes it easy to switch learning models when a process change occurs.

[0085] In the example shown in FIG. 13, the group names "normal season" and "peak season" are assigned, and the normal season consists of three processes, processes 1 to 3, while the peak season adds one more process, making it four processes, processes 1 to 4.

[0086] By switching the group name from "Normal Season" to "Peak Season", the number of processes is automatically changed from three processes, processes 1 to 3, to four processes, processes 1 to 4. In this case, the total number of operations does not change, so if the total number of operations is 12, three operations will be automatically assigned to each process, as shown in Figure 11(f).

[0087] 14 is a diagram showing an example of allocating groups to be used based on a planning table (schedule). Even if groups are divided into normal periods and peak periods, the user 60 must specify which group's model to use at which period.

[0088] 14, by setting in advance which group's model to use at which time, the model linked to the group name set in the schedule 72 can be acquired from the learning model storage unit 52, and the operation according to the schedule 72 can be automatically realized. This makes it possible to automatically create work definition information without the user 60 having to specify it.

[0089] Fig. 15 is a diagram showing an example of a screen for registering labels when introducing a new process. The screen that is first displayed on the UI when registering a label is a screen that lists the work names as labels as a label list as shown in Fig. 15(a). This screen includes an add button 73 for adding a work name, a delete button 74 for deleting a work name, and an edit button 75 for editing the work name.

[0090] When an Add button 73 is pressed on the screen shown in Fig. 15(a), a screen as shown in Fig. 15(b) is displayed. In Fig. 15(b), a small screen (pop-up) 76 for adding a label is displayed so as to be superimposed on the label list screen. The pop-up 76 includes a Name button 77 for inputting the name of the task as a label, and a Description button 78 for inputting a description of the task. The pop-up 76 also includes a Register button 79 for registering the label, and a Cancel button 80 for canceling the registration.

[0091] When the user 60 presses the register button 79, the label is registered, and a list including the registered label is displayed in the label list, as shown in Fig. 15(c), for example. When the user presses the cancel button 80, the screen can be returned to, for example, the screen shown in Fig. 15(a).

[0092] Fig. 16 is a diagram showing an example of a screen for adding and deleting processes and adding and deleting work. The screen that is first displayed on the UI when changing a process is a screen listing process names as a process list as shown in Fig. 16(a). This screen includes an add button 81 for adding a process name, a delete button 82 for deleting a process name, and an edit button 83 for editing the process name.

[0093] When the user 60 presses the Add button 81 on the screen shown in Fig. 16(a), a pop-up 84 is displayed as shown in Fig. 16(b). The pop-up 84 includes a Name button 85 for inputting the name of the process to be added, and a Description button 86 for inputting a description of the process. The pop-up 84 also includes a Register button 87 for registering the process to be added, and a Cancel button 88 for canceling the registration.

[0094] When the user 60 presses the register button 87 in the pop-up 84 shown in Fig. 16(b), a screen of the process list with the process added, shown in Fig. 16(c), is displayed. Until then, processes 1 to 5 were registered in the process list, but by registering process 6, process 6 has been added to the list.

[0095] When the user 60 presses the edit button 83 on the screen shown in Fig. 16(d), a pop-up 89 is displayed as shown in Fig. 16(e). The pop-up 89 is a screen that allows the user to add or delete tasks to be performed in process 6. For this reason, the pop-up 89 includes an add button 90 for adding a task, a delete button 91 for deleting a task, a register button 92 for registering a task to be performed in process 6, and a cancel button 93 for canceling the editing of process 6.

[0096] When the user 60 presses the Add button 90 on the screen shown in Fig. 16(e), the pop-up 89 is switched to a pop-up 94 as shown in Fig. 16(f). The pop-up 94 shows a list of tasks to be added, and is configured so that any task can be selected from the task list. The pop-up 94 includes a scroll bar 95 for moving (scrolling) a task name that is not visible to a visible position, an Add button 96 for adding a task, a Register button 97 for registering the added task, and a Cancel button 98 for canceling the addition of the task.

[0097] In the pop-up 94 shown in Fig. 16(f), when the user 60 specifies the task name "Pick up part A" and presses the Add button 96 next to the task name, the Add button 96 next to the task name "Pick up part A" disappears, as shown in Fig. 16(g), indicating that the task has been added. Here, the disappearance of the Add button 96 next to the task name indicates that the task has been added, but the method of indicating that the task has been added is not limited to this.

[0098] On the screen shown in Fig. 16(g), the user 60 can also add another task. In the example shown in Fig. 16(g), the Add button 96 next to the task name "Pick up part B" is pressed to add that task name. When the task name "Pick up part B" is added, a pop-up 94 is displayed as shown in Fig. 16(h). That is, the Add buttons 96 next to the task names "Pick up part A" and "Pick up part B" have disappeared.

[0099] When the user 60 presses the register button 97 in the pop-up 94 shown in Fig. 16(h), the screen returns to the original pop-up 89 as shown in Fig. 16(i), and the task names "Pick up part A" and "Pick up part B" are added as tasks to be performed in process 6. If the user 60 is OK with these two tasks to be performed in process 6, he or she can register them by pressing the register button 92. In this way, the user 60 can register task definition information for tasks to be performed in each process.

[0100] Fig. 17 is a diagram showing an example of a screen for linking the start position and end position of a video that represents a time period of each task with a label. The start position and end position of a video that represents a time period of each task are represented by, for example, the start frame ID and end frame ID of the video. The screen shown in Fig. 17 includes a selection button 100 for selecting the file name of the video to be played, a window 101 that displays a preview of the video, a drop-down list 102 for selecting and displaying a label, and operation buttons 103 for playing, fast-forwarding, stopping, rewinding, etc. the video in order to set the start position and end position corresponding to the displayed label.

[0101] The user 60 uses the operation buttons 103 to detect the start position and end position corresponding to the displayed label. The screen shown in FIG. 17 includes a start position setting button 104 for setting the start position, and an end position setting button 105 for setting the end position. The user 60 presses the start position setting button 104 for the start position detected by the operation buttons 103 to set the position as the start position of the task. Similarly, the user 60 presses the end position setting button 105 for the end position detected by the operation buttons 103 to set the position as the end position of the task. This is performed similarly for other tasks. The start position is set as the start frame ID, and the end position is set as the end frame ID.

[0102] The screen shown in FIG. 17 includes a table 106 that associates labels with start frame IDs and end frame IDs, a registration button 107 for registering the table as linking information, and a cancel button 108 for canceling the registration.

[0103] Table 106 stores the label selected from drop-down list 102 in association with the start frame ID indicating the start position and the end frame ID indicating the end position set by start position setting button 104 and end position setting button 105.

[0104] As described above, by providing the system, method, and program of the present invention, it is possible to reduce the effort and time required to machine-train a learning model.

[0105] The present invention has been described so far with respect to the above-mentioned embodiments as an information processing system, an information processing method, and a program. However, the present invention is not limited to the above-mentioned embodiments, and can be modified within the scope of what a person skilled in the art can conceive, such as other embodiments, additions, modifications, deletions, etc. Furthermore, any embodiment is within the scope of the present invention as long as it provides the functions and effects of the present invention. [Explanation of symbols]

[0106] 10...information processing device, 20...CPU, 21...ROM, 22...RAM, 23...HD, 24...HDD controller, 25...display, 26...external device connection I / F, 27...network I / F, 28...bus line, 29...keyboard, 30...pointing device, 31...DVD-RW drive, 32...media I / F, 33...DVD-RW, 34...recording medium, 40...annotation tool, 41...video registration unit, 42...label registration unit, 43...work definition unit, 44...linking unit, 45...video memory unit, 46...label memory unit, 47...work definition memory unit, 48...linking information memory unit, 49...learning data generation unit, 50...learning data memory unit, 51...training unit, 52...learning model memory unit, 53...schedule setting unit, 54...setting memory unit, 60...user, 70...table, 71...button, 72...schedule, 73...Add button, 74...Delete button, 75...Edit button, 76...Pop-up, 77...Name button, 78...Description button, 79...Register button, 80...Cancel button, 81...Add button, 82...Delete button, 83...Edit button, 84...Pop-up, 85...Name button, 86...Description button, 87...Register button, 88...Cancel button, 89...Pop-up, 90...Add button, 91...Delete button, 92...Register button, 93...Cancel button, 94...Pop-up, 95...Scroll bar, 96...Add button, 97...Register button, 98...Cancel button, 100...Select button, 101...Window, 102...Drop-down list, 103...Operation button, 104...Start position setting button, 105...End position setting button, 106...Table, 107...Register button, 108...Cancel button [Prior art documents] [Patent documents]

[0107] [Patent Document 1] JP 2019-159819 A

Claims

1. An information processing system for performing machine learning of a learning model, A data acquisition means for acquiring photographed data of a series of operations including one or more work steps; A registration means for registering work identification information for identifying each work included in the work process; a linking means for generating linking information that links a start position and an end position of the shooting data that represent a time period of each of the tasks with the task identification information, using the shooting data; a receiving means for receiving work identification information of each of the work performed in each of the work processes; A data generating means for generating learning data for each of the work processes; having The data generating means acquiring a start position and an end position of shooting data representing a time period of each of the tasks identified by the task identification information received by the receiving means, based on the task identification information received by the receiving means and the linking information generated by the linking means; acquiring, from the photographing data acquired from the data acquisition means, photographing data corresponding to a start position and an end position of photographing data representing a time period of each of the tasks identified by the task identification information accepted by the acceptance means; An information processing system that generates learning data to be used for machine learning of a learning model for each work process based on the acquired photographic data.

2. The information processing system according to claim 1 , wherein the start and end positions of the image data representing the time period of each task are the start and end frames of the task among a plurality of frames constituting a video as the image data.

3. 3. The information processing system according to claim 2, wherein the data generating means detects skeletal information of the worker in each of the frames, and uses the linking information generated by the linking means to assign corresponding task identification information to one or more pieces of skeletal information for each of the tasks, thereby generating learning data for each of the task processes.

4. 2. The information processing system according to claim 1, wherein the accepting means displays a list of predefined tasks and accepts selection of task identification information of tasks to be performed in each of the task processes.

5. 2. The information processing system according to claim 1, further comprising: a model acquisition means for acquiring the learning model to which model identification information corresponding to the work process is linked; and a learning means for performing machine learning on the learning model acquired by the model acquisition means using learning data of the work process generated by the data generation means.

6. an information acquiring means for acquiring schedule information that defines which learning model of a plurality of model identification information is to be used at what time, The information processing system according to claim 5 , wherein the model acquisition means acquires the learning model linked to the corresponding model identification information based on the schedule information acquired by the information acquisition means.

7. A method for performing machine learning of a learning model by an information processing system, comprising: A first acquisition step of acquiring photographed data of a series of operations including one or more work steps; A step of registering work identification information for identifying each work included in the work process; a first generation step of generating, using the photographing data, association information that associates a start position and an end position of the photographing data that represent a time period of each of the tasks with the task identification information; receiving work identification information of each of the work performed in each of the work processes; a second acquisition step of acquiring a start position and an end position of shooting data representing a time period of each of the tasks identified by the task identification information received in the receiving step, based on the task identification information received in the receiving step and the linking information generated in the first generation step; a third acquisition step of acquiring, from the photographing data acquired in the first acquisition step, photographing data corresponding to a start position and an end position of photographing data representing a time period of each of the tasks identified by the task identification information received in the receiving step; A second generation step of generating learning data to be used for machine learning of a learning model for each of the work processes based on the photographing data acquired in the third acquisition step; An information processing method comprising:

8. A program for causing a computer to execute machine learning of a learning model, A first acquisition step of acquiring photographed data of a series of operations including one or more work steps; A step of registering work identification information for identifying each work included in the work process; a first generation step of generating, using the photographing data, association information that associates a start position and an end position of the photographing data that represent a time period of each of the tasks with the task identification information; receiving work identification information of each of the work performed in each of the work processes; a second acquisition step of acquiring a start position and an end position of shooting data representing a time period of each of the tasks identified by the task identification information received in the receiving step, based on the task identification information received in the receiving step and the linking information generated in the first generation step; a third acquisition step of acquiring, from the photographing data acquired in the first acquisition step, photographing data corresponding to a start position and an end position of photographing data representing a time period of each of the tasks identified by the task identification information received in the receiving step; A second generation step of generating learning data to be used for machine learning of a learning model for each of the work processes based on the photographing data acquired in the third acquisition step; A program to execute.

Images