PROGRAM, INFORMATION PROCESSING METHOD, AND INFORMATION PROCESSING APPARATUS

Abstract

To provide a program etc. capable of presenting a work situation photographed in a plurality of moving images by synchronizing divided moving images of each action in the moving images. [Solution] A computer divides a video taken of a workspace where a subject is working into segments for each action included in the work. The computer also outputs synchronized segmented videos of each action included in the multiple videos based on segmented videos obtained by dividing a plurality of videos, including videos taken of different workspaces where the subject is working on the same work or videos taken of workspaces where the arrangement of items related to the same work is different, into segments for each action.

Description

[Technical field]

[0001] The present disclosure relates to a program, an information processing method, and an information processing device. [Background technology]

[0002] Patent Document 1 discloses a technique for detecting partial videos corresponding to partial movements included in an overall movement from an overall video obtained by capturing the overall movement of a trainee performing training, and evaluating the overall movement and the partial movements by referring to the partial videos. The technique disclosed in Patent Document 1 evaluates the overall movement and the partial movements based on the presence or absence of partial videos corresponding to each partial movement, the order of the partial movements, the duration of the partial movements, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6757010 Summary of the Invention [Problem to be solved by the invention]

[0004] When evaluating the movements of each trainee, there is a demand for comparison with the movements of other trainees or the movements of the same trainee at different times. However, in Patent Document 1, evaluation is based on whether or not partial movements have been performed, whether or not partial movements have been performed in a predetermined order, and whether or not partial movements have been performed for a predetermined period of time, and does not take into consideration comparison with the movements of other trainees or the movements of the same trainee at different times.

[0005] The present disclosure aims to provide a program etc. capable of presenting a work situation captured in multiple videos by synchronizing segmented videos of each action in the videos. [Means for solving the problem]

[0006] A program according to one aspect of the present disclosure causes a computer to execute a process of dividing a video taken of a work space in which a subject is working into each action included in the work, and outputting synchronized divided videos of each action in a plurality of videos based on the divided videos divided into each action, including videos taken of different work spaces in which the subject is performing the same work, or videos taken of work spaces in which items related to the same work are arranged in different ways, based on the divided videos. Effect of the Invention

[0007] In one aspect of the present disclosure, a task situation captured in a plurality of videos can be presented by synchronizing segmented videos of each action in the videos. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is an explanatory diagram showing an example of a moving image to be processed; [Diagram 2] 1 is a block diagram illustrating an example of a configuration of an information processing system. [Diagram 3] FIG. 2 is an explanatory diagram illustrating an example of the configuration of a learning model. [Figure 4] FIG. 11 is an explanatory diagram showing an example of the record layout of a video DB and a ToDo list DB. [Diagram 5] 13 is a flowchart illustrating an example of an upload process and a cycle detection process. [Figure 6] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 7] 13 is a flowchart showing an example of a comparison process procedure for cycle videos. [Figure 8] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 9] 13 is a flowchart illustrating an example of a procedure for adding a comment. [Figure 10] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 11] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 12] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 13] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 14] 13 is a flowchart illustrating an example of a monitoring process procedure. [Figure 15] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 16] 13 is a flowchart showing an example of a comparison process procedure according to the second embodiment. [Figure 17] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 18] 13 is a flowchart showing an example of a comparison process procedure between different tasks. [Figure 19] FIG. 13 is an explanatory diagram showing an example of a screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a program, an information processing method, and an information processing device according to the present disclosure will be described in detail with reference to the drawings showing embodiments thereof.

[0010] (Embodiment 1) In this embodiment, an information processing system that compares and presents multiple videos of a worker (subject) performing work is described. FIG. 1 is an explanatory diagram showing an example of a video to be processed. The video to be processed in this embodiment is video data including multiple images (still images), for example, 30 or 15 per second, and is a captured image of a worker performing work, a work space where the worker is working, etc., as shown in FIG. 1A. The videos to be compared in this embodiment include, for one work (work of the same content), a video of the same worker in the same work space, a video of different workers in the same work space, and a video of workers (the same worker and different workers) in different work spaces. The different work spaces may be different work spaces in the same place (factory), or may be work spaces in different places. In addition, as long as the work is the same, each video can be compared even if the work spaces have different arrangements of the items to be worked on and the items used for the work (items related to the work) and the worker is left-handed or right-handed.

[0011] In addition, the video to be compared in this embodiment includes a video taken by a camera with a fixed shooting position and a video taken by a camera attached to a worker. The upper left and upper right of FIG. 1A show an example of a video taken by a camera with a fixed shooting position, where the upper left is a video of the worker and the work space taken from above in front of the worker, and the upper right is a video of the worker and the work space taken from above the left side of the worker. The lower left and lower right of FIG. 1A show an example of a video taken by a wearable camera attached to a worker, showing a video of the worker's hands and the work space. The wearable camera is, for example, an ear-hook type, a headband type (head strap type), a glasses type, or the like, and is used by being attached to the head (face) of the worker. The wearable camera may be attached to the temples of the glasses worn by the worker, the brim of the hat, or a helmet, or may be attached to the worker's clothes or body using a clip or a chest harness. Therefore, in the system of this embodiment, for the same task, videos shot in different locations and different work spaces, videos shot in different shooting directions and angles of view relative to the work space, etc. can be compared.

[0012] In addition, the video to be processed in this embodiment may not only be a video (hereinafter referred to as cycle video) of a worker performing a series of tasks, but may also be a video including multiple cycle videos as shown in FIG. 1B. The upper side of FIG. 1B shows a video of a worker repeatedly performing task 1 (assembly of case A), and the lower side of FIG. 1B shows a video of a worker performing task 1 (assembly of case A), task 2 (assembly of part A), and so on. In this way, the video to be processed may be a video including multiple cycle videos of the same task, or may be a video including multiple cycle videos of different tasks. Furthermore, the worker performing each task may be the same worker or different workers. The different tasks are, for example, multiple tasks required to assemble one product. Furthermore, the cycle video includes a video divided into multiple actions included in each task (hereinafter referred to as divided video). In the example shown at the top of Figure 1B, the videos to be processed can be divided into cycle videos of a worker performing task 1 (assembling case A), and each cycle video can be divided into segmented videos of a worker performing actions such as action 1 (removing the case), action 2 (tightening the screws), etc.

[0013] FIG. 2 is a block diagram showing an example of the configuration of an information processing system. The information processing system of this embodiment includes a server 10 and a user terminal 20, and the server 10 and the user terminal 20 are connected to each other via a network N. The network N may be the Internet or a public communication line, or may be a LAN (Local Area Network) constructed in a facility where the server 10 is installed. The server 10 is an information processing device capable of various information processing and sending and receiving information, and is composed of, for example, a server computer, a personal computer, a workstation, etc. The user terminal 20 is an information processing device capable of various information processing and sending and receiving information, and is composed of, for example, a personal computer, a tablet terminal, a smartphone, etc. The user terminal 20 is a terminal used by a user who uses the server 10. The users in this embodiment include the workers to be photographed, the workers' superiors or supervisors, the people in charge (for example, the factory manager) of each work site (factory, etc.), and the managers (for example, the person in charge at the head office) who manage each work site, and the like who perform various processes in the analysis process of the work status of each worker using the server 10.

[0014] The server 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, etc., and these units are connected to each other via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), or an AI chip (AI semiconductor). The control unit 11 executes the information processing and control processing to be performed by the server 10 by appropriately executing a program 12P stored in the storage unit 12. Note that when the control unit 11 has multiple processes, the control unit 11 may execute each process by a different processor.

[0015] The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), and the like. The storage unit 12 stores a program 12P (program product) executed by the control unit 11 and various data. The storage unit 12 also temporarily stores data and the like generated when the control unit 11 executes the program 12P. The storage unit 12 also stores a learning model M that has been trained by machine learning. The learning model M is expected to be used as a program module that constitutes artificial intelligence software. The learning model M performs a predetermined calculation on an input value and outputs the calculation result. The storage unit 12 stores information that defines the learning model M, such as information on layers included in the learning model M, information on nodes that constitute each layer, and weights (coupling coefficients) between nodes. The learning model M is prepared for each task performed by the worker. The storage unit 12 also stores a video DB 12a, a ToDo list DB 12b, and a video folder 12c. At least one of the video DB 12a, the ToDo list DB 12b, and the video folder 12c may be stored in another storage device connected to the server 10, or may be stored in another storage device with which the server 10 can communicate.

[0016] The communication unit 13 is a communication module for performing processes related to wired or wireless communication, and transmits and receives information to and from other devices via the network N. The input unit 14 accepts operation input by a user, and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 11. A part of the input unit 14 and the display unit 15 may be a touch panel configured as an integrated unit.

[0017] The reading unit 16 reads information stored in a portable storage medium 10a including a CD (Compact Disc), a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, etc. The program 12P and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 10a via the reading unit 16 and stored in the storage unit 12. In addition, the program 12P and various data may be written in the storage unit 12 during the manufacturing stage of the server 10, or the control unit 11 may download them from another device via the communication unit 13 and store them in the storage unit 12.

[0018] The server 10 is not limited to a single computer, but may be a multi-computer including multiple computers, or may be a virtual machine virtually constructed by software in one device. When the server 10 is configured with a server computer, the server 10 may be a local server or a cloud server connected for communication via the Internet or the like. In the following, the server 10 is described as being one computer. The program 12P may be arranged on a single computer or at one site, or may be distributed across multiple sites and distributed and executed on multiple computers connected to each other via a network. Furthermore, the input unit 14 and the display unit 15 are not essential for the server 10, and the server 10 may be configured to accept operations through a connected computer (e.g., a user terminal 20), or may be configured to output information to be displayed to an external display device (e.g., a user terminal 20).

[0019] The user terminal 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, a display unit 25, a reading unit 26, etc., and these units are connected to each other via a bus. The control unit 21, the storage unit 22, the communication unit 23, the input unit 24, the display unit 25, and the reading unit 26 of the user terminal 20 have the same configuration as the control unit 11, the storage unit 12, the communication unit 13, the input unit 14, the display unit 15, and the reading unit 16 of the server 10, so that the description thereof will be omitted. Note that, when the user terminal 20 is a terminal of a user who uploads a video of a worker (work space) captured by a camera to the server 10, the storage unit 22 stores a video file 22a acquired from the camera in addition to the program 22P (program product) executed by the control unit 21.

[0020] FIG. 3 is an explanatory diagram showing an example of the configuration of the learning model M. FIG. 3A is an explanatory diagram of the learning process of the learning model M, and FIG. 3B is an explanatory diagram of the estimation process using the learning model M. The learning model M is configured using algorithms such as a recurrent neural network (RNN), a long short-term memory (LSTM), a convolution neural network (CNN), a transformer, etc., and may also be configured by combining a plurality of algorithms. The learning model M is a model that has been pre-trained with videos (videos including human movements) of humans performing a wide variety of actions, and is generated by loading one training cycling video. In the learning process, the learning model M extracts features of the input training cycling video. For example, the learning model M detects a moving object in the training cycling video and extracts features in the area where the moving object is captured. Then, when the video to be estimated is input to the learning model M, which has already loaded the training cycle video, in the estimation process, the learning model M divides the training cycle video loaded in the learning process into multiple segment videos based on image feature amounts, extracts cycle videos having a high similarity score with the training cycle video from the video to be estimated based on the division results, and divides (partitions) each extracted cycle video into multiple segment videos (the same number as the segment videos of the training cycle video) based on the segment videos in the training cycle video. Note that the number of segment videos into which the cycle video is divided may be set in advance in the learning model M, for example, or may be changeable to an arbitrary number. Thus, the learning model M receives the video to be estimated, performs a calculation to extract cycle videos from the input video, and a calculation to divide each extracted cycle video into segment videos, and outputs the calculation results. Specifically, the learning model M has an input layer to which the video to be estimated is input, an intermediate layer that extracts features from the input video and performs various calculations, and an output layer that outputs the start time and end time of each cycle video extracted from the video based on the calculation results of the intermediate layer, as well as the start time and end time of each segment video in each cycle video.The start time and end time of the cycle video and the start time and end time of the section video are represented by the elapsed time from the start of playback of the video. By using such a learning model M, when the server 10 inputs a video to be estimated into the learning model M, it can extract cycle videos from the video to be estimated based on output information from the learning model M (the start time and end time of the cycle video and the start time and end time of the section video), and divide each cycle video into section videos. In addition, in the estimation process, when dividing the cycle video extracted from the video to be estimated into section videos, the learning model M divides the section videos into section videos with high similarity scores with each section video in the training cycle video, and also outputs the similarity score of each section video with each section video in the training cycle video.

[0021] Learning of the learning model M (reading of a training cycle video) may be performed by another learning device. The learned learning model M generated by learning performed by another learning device is downloaded from the learning device to the server 10 via, for example, the network N or the portable storage medium 10a and stored in the storage unit 12. The learning model M is not limited to a configuration in which a video of a worker is input. For example, the server 10 may be configured to perform skeleton estimation of the worker based on each frame (each image) included in the video, generate skeleton data, and input the time-series skeleton data to the learning model M. In this case, the control unit 11 of the server 10 performs skeleton estimation of the worker in each frame using a technique for extracting the joint positions of a person in an image, such as OpenPose, and extracts the joint positions of the worker. The control unit 11 may acquire time-series skeleton data by performing a process of extracting joint positions for each frame included in the video, and input the acquired time-series skeleton data to the learning model M.

[0022] FIG. 4 is an explanatory diagram showing an example of the record layout of the video DB 12a and the ToDo list DB 12b. FIG. 4A shows the video DB 12a, which stores information about videos uploaded to the server 10. The video DB 12a shown in FIG. 4A includes a video URL (Uniform Resource Locator) column, a shooting location column, a shooting target column, a registrant column, a cycle detection column, a comparison analysis column, and the like, and stores information about videos uploaded to the server 10 in association with the video URL. The video URL stores information for reading out video data stored in the video folder 12c. Note that instead of the video URL, a folder name and a file name indicating the storage location of the video data may be stored. The shooting location column stores information about the shooting location of the video, such as information indicating the location of the factory, information indicating the location of the work space in the factory, and the like. The shooting target column stores information about the worker who is the subject of the shooting, such as the worker's affiliation, name, employee number, and the like. Note that if a video includes multiple cycle videos of multiple workers, information about the multiple workers is stored. The registrant column stores information about the user (registrant) who uploaded the video to the server 10, and stores, for example, identification information (employee number, user ID, etc.) assigned to the registrant. The cycle detection column stores information about the process of extracting cycle videos from the video to be processed using the learning model M, and stores the execution date and time of the process, information about the executor, URL of each extracted cycle video, etc. The executor is a user who instructed the server 10 to execute the cycle video extraction process. Each cycle video extracted from the video to be processed may be stored in the video folder 12c. The comparison analysis column stores information about the process of displaying multiple cycle videos side by side and analyzing them, and stores the execution date and time of the process, information about the executor, URL of the cycle video to be compared, etc. The executor here is a user who instructed the server 10 to execute the comparison process of multiple cycle videos. The comparison target stores information about the cycle video displayed as the comparison target, and stores, for example, the URL of the cycle video. The stored contents of the video DB 12a are not limited to the example shown in FIG. 4A, and may store, for example, the shooting date and time of the video or the registration date and time.

[0023] FIG. 4B shows the ToDo list DB 12b, which stores information on cycle videos and section videos to which comments have been added among videos uploaded to the server 10, and information on cycle videos and section videos registered in the ToDo list. The ToDo list DB 12b shown in FIG. 4B includes a video URL column, a comment column, an improvement plan column, a status column, an improved video URL column, an improvement content column, and the like, and stores information on cycle videos and section videos registered in the ToDo list in association with video URLs (ToDo data). The video URL stores information for reading video data of cycle videos or section videos stored in the video folder 12c. Again, instead of the video URL, a folder name and a file name indicating the storage location of the video data may be stored. The comment column stores information on comments entered for cycle videos and section videos, and stores information on the person who entered the comment, the content, and the like. The improvement plan column stores information about the improvement plan input for the cycle video and the section video to which a comment has been added, and stores information about the person who input the information, the contents of the improvement plan, the person in charge of implementing the improvement plan, and the deadline, etc. The status column stores information indicating the implementation status of the improvement plan, and stores "Completed" if the implementation has been completed, and stores "Not yet" if it has not been completed. The improved video URL column stores the URL of the cycle video or section video that was filmed after the implementation of the improvement plan was completed. The improvement content column stores information indicating the improvement status due to the implementation of the improvement plan, and stores the time required before improvement, the time required after improvement, the improvement rate, etc. The time required before improvement is the time required (playback time) of the task or action in the cycle video or section video whose URL is registered in the video URL column, the time required after improvement is the time required (playback time) of the task or action in the cycle video or section video whose URL is registered in the improved video URL column, and the improvement rate is a ratio calculated, for example, by the time required after improvement / the time required before improvement. The contents stored in the ToDo list DB 12b are not limited to the example shown in FIG. 4B, and the items in the ToDo list DB 12b can be changed for each work site (for example, a factory or work space) or for each company, for example.For example, not only the improvement plan but also messages from the worker's superior or supervisor, information to be shared with other users, information for promoting improvements in work efficiency, etc. may be stored. Furthermore, the input date and time of each of the comments, improvement plan, status, improved video URL, and improvement content may be stored.

[0024] The following describes the process of uploading a video captured by a camera to the server 10 and the cycle detection process of extracting a cycle video from the video uploaded to the server 10 in the information processing system of this embodiment. Fig. 5 is a flowchart showing an example of the upload process procedure and the cycle detection process procedure, and Fig. 6 is an explanatory diagram showing an example screen.

[0025] FIG. 5A shows an upload process. In the process of FIG. 5A, the user of the user terminal 20 is a person in charge of uploading videos to the server 10, such as the worker himself or the worker's superior. A user who wants to upload a video copies the video stored in the camera to the storage unit 22 of the user terminal 20. Alternatively, the user connects a camera to the user terminal 20 so that the video stored in the camera can be acquired by the user terminal 20. The user performs a predetermined operation via the input unit 24 of the user terminal 20, and the control unit 21 of the user terminal 20 performs processing according to the operation content to display an upload screen on the display unit 25 and upload the video to the server 10 via the upload screen. For example, the control unit 21 accesses the server 10 and acquires the upload screen from the server 10. In addition, if an app (application program) for using the server 10 is stored in the storage unit 22, the control unit 21 may display the upload screen by starting the app.

[0026] Control unit 21 of user terminal 20 displays an upload screen as shown in Fig. 6A on display unit 25 (S11). The upload screen displays a list of file names of videos (video files 22a) stored in storage unit 22 and videos that can be acquired from a camera connected to user terminal 20, and has check boxes for selecting each video and an upload button for instructing uploading of the selected video.

[0027] The control unit 21 receives a selection of a video from the user on the upload screen via the input unit 24 (S12), and determines whether the upload button has been operated (S13). If the control unit 21 determines that the upload button has not been operated (S13: NO), it continues to receive the selection of the video. If the control unit 21 determines that the upload button has been operated (S13: YES), it reads out the video file 22a of the selected video from the storage unit 22 or the camera and transmits (uploads) it to the server 10 (S14). When uploading a video to the server 10, the control unit 21 receives information on the shooting location (e.g., a factory, a work space in the factory), information on the worker to be shot, information on the user (registered person) uploading, etc., via the input unit 24, for example, and transmits them to the server 10 together with the video file.

[0028] The control unit 11 of the server 10 acquires the video file uploaded from the user terminal 20 and stores it in the storage unit 12 (S15). For example, the control unit 11 stores the acquired video file in a video folder 12c in the storage unit 12, and generates a URL for reading out this video file. The control unit 11 then associates the generated URL with the information on the shooting location and the worker who was the subject of the shooting acquired from the user terminal 20, and the information on the registered person, and stores them in the video DB 12a. This enables the server 10 to collect videos shot in each factory.

[0029] FIG. 5B shows the cycle detection process. The user of the user terminal 20 in the process of FIG. 5B is a person in charge of performing cycle detection process to extract cycle videos from uploaded videos, such as the worker himself or the worker's superior. A user who wants to perform cycle detection process performs a predetermined operation via the input unit 24 of the user terminal 20, and the control unit 21 of the user terminal 20 performs processing according to the operation content, thereby displaying a cycle detection screen on the display unit 25 and requesting the server 10 to execute cycle detection process. The control unit 21 of the user terminal 20 displays a cycle detection screen as shown in FIG. 6B on the display unit 25 (S21). The cycle detection screen displays a list of file names of videos (video files) uploaded to the server 10, and has check boxes for selecting each video and a cycle detection button for instructing execution of cycle detection process for the selected video. For videos for which cycle detection process has already been performed, "cycle detected" may be displayed after the file name.

[0030] The control unit 21 accepts a video selection from the user via the input unit 24 on the cycle detection screen (S22), and determines whether or not the cycle detection button has been operated (S23). If it is determined that the cycle detection button has not been operated (S23: NO), the control unit 21 continues to accept the selection of videos, and if it is determined that the cycle detection button has been operated (S23: YES), it instructs the server 10 to execute cycle detection processing on the selected video (S24). The control unit 21 transmits information indicating the selected video (e.g., file name, URL, etc.) to the server 10 to request the execution of cycle detection processing.

[0031] The control unit 11 of the server 10 reads out the video for which the cycle detection process has been requested from the video folder 12c, and executes the cycle detection process (S25). Here, the control unit 11 inputs the selected video to the learning model M as a video to be estimated, and based on the output information from the learning model M, extracts cycle videos from the input video, and divides each cycle video into sectioned videos. The control unit 11 stores each extracted cycle video in the video folder 12c, generates a URL for reading out each cycle video (video file), and stores the execution date and time of the cycle detection process, information of the user (executor) who requested the execution of the cycle detection process, and the URL of each cycle video in association with each other in the cycle detection column of the video DB 12a. This allows the server 10 to register information on the cycle video extracted from each video.

[0032] The control unit 11 then generates a detection result screen that displays the results of the cycle detection process as shown in FIG. 6C (S26), and transmits the generated screen to the user terminal 20 (S27). The screen shown in FIG. 6C has a video display field R1 that displays the video to be processed, an information display field R2 that displays information on cycle videos extracted from this video, and a time display field R3 that displays the start time and end time of each cycle video relative to the playback time of the video to be processed. The information display field R2 displays "video 11" as the name (e.g., file name) of the video to be processed, and displays video 1, video 2, ... as the names of cycle videos extracted from this video. The time display field R3 displays a rectangle indicating the start time and end information of each cycle video relative to an indicator showing the playback time of the video to be processed.

[0033] The control unit 21 of the user terminal 20 receives the detection result screen transmitted by the server 10 and displays it on the display unit 25 (S28). As a result, the screen shown in Fig. 6C is displayed on the user terminal 20, and the user can check each cycle video included in the video that requested the cycle detection process.

[0034] Next, a process for comparing and presenting multiple cycle videos of the same work in the information processing system of this embodiment will be described. Fig. 7 is a flow chart showing an example of the cycle video comparison process procedure, and Fig. 8 is an explanatory diagram showing an example screen. The user of user terminal 20 in the process of Fig. 7 is a person in charge of checking and analyzing the work situation using the cycle videos, such as the worker himself or the worker's superior.

[0035] In this embodiment, a comparison of multiple cycle videos can be performed from the detection result screen shown in FIG. 6C, for example. Therefore, when the detection result screen is displayed (S31), the control unit 21 executes the processes from step S32 onward. In the screen of FIG. 6C, the information display field R2 is provided with check boxes for selecting each cycle video extracted from the video to be processed, and a comparison button for instructing execution of a comparison process of the selected multiple cycle videos. The control unit 21 accepts a selection of a cycle video from the user via the input unit 24 on the detection result screen (S32), and determines whether or not the comparison button has been operated (S33). If it is determined that the comparison button has not been operated (S33: NO), the control unit 21 continues to accept the selection of the cycle videos, and if it is determined that the comparison button has been operated (S33: YES), it instructs the server 10 to compare the selected cycle videos (S34). Note that the control unit 21 accepts the selection of multiple cycle videos, and transmits information indicating the selected cycle videos (e.g., file names, URLs, etc.) to the server 10 to request execution of a comparison process of the cycle videos. On the screen of FIG. 6C, the cycle videos Video 1 to Video 3 are selected.

[0036] The control unit 11 of the server 10 reads out the cycle video requested for comparison from the video folder 12c (S35), and generates a comparison screen for comparing the cycle videos to be compared (S36). For example, the control unit 11 generates a comparison screen as shown in FIG. 8A, and transmits the generated comparison screen to the user terminal 20 (S37). The comparison screen also has a video display field R1, an information display field R2, and a time display field R3. The video display field R1 of the screen in FIG. 8A displays the cycle videos of video 1 and video 2 out of video 1 to video 3, which are the selected cycle videos. Note that the cycle video being displayed in the video display field R1 is configured to be changeable to any cycle video from the cycle videos selected in step S32. Specifically, the video display field R1 is provided with switching buttons B1 and B2 for the cycle video being displayed, and the switching buttons B1 and B2 are provided with pull-down menus for selecting one of the selectable cycle videos. Additionally, the cycle video displayed to the right of the video display field R1 is provided with a switch button B3 that allows the cycle video to be switched in the order displayed in the information display field R2 of Fig. 6C, for example. With this configuration, any two cycle videos can be displayed in the video display field R1, allowing the two cycle videos to be compared. Note that the video display field R1 may be configured to display all selected cycle videos.

[0037] The information display field R2 in the screen of FIG. 8A displays a stacked graph D that shows the time required for each section video (each movement) for each selected cycle video (the movement time from the start time to the end time of the movement recorded in the section video) stacked up. The stacked graph D shows a bar graph that accumulates the time required for each section video (section 1, section 2, ...) for each cycle video in order from the bottom up. Note that the stacked graph D displays, from left to right, a bar graph of the cycle video displayed on the left side of the video display field R1, a bar graph of the cycle video displayed on the right side, and bar graphs of other cycle videos. In addition, in the stacked graph D for each cycle video, the time required for the same movement (section) may be displayed in the same color (display mode), in which case it is easy to grasp the length of time required for each movement in each cycle video. Each section in the stacked graph D may be configured to display the duration of the corresponding section of the cycle video when a specific operation (e.g., selection with the cursor) is performed, and may be configured to start playback of a segmented video of the corresponding cycle video in the video display field R1 when a specific operation (e.g., left clicking of the mouse) is performed. In the example of Fig. 8A, section 6 of video 2 is selected, and the duration of the selected section 6 is displayed.

[0038] The time display field R3 in the screen of FIG. 8A shows the playback time of the cycle video on the horizontal axis, and shows the start time and end time of each section video (each section) in the two cycle videos displayed in the video display field R1. In the example of FIG. 8A, the start times of the two cycle videos are aligned and the start time and end time of each section video are shown, and a mark C1 indicating the playback position of the two cycle videos in the video display field R1 is added. The comparison screen of this embodiment is configured to display (play) the multiple cycle videos displayed in the video display field R1 in synchronization (linkage) with each section video (each section) into which each cycle video is divided. For example, in the example of FIG. 8A, the required time for section 1 of video 1 is longer than the required time for section 1 of video 2, so when two cycle videos are played, after the playback of section 1 of video 2 ends, the playback of section 2 of video 1 waits until the playback of section 1 of video 1 ends, and when the playback of section 1 of video 1 ends, the playback of section 2 of the two cycle videos begins. The time display field R3 in Figure 8B shows the state at the time when playback of section 1 of the two cycle videos has ended, and mark C1 indicates the end time of section 1 of each cycle video, after which playback of section 2 of each cycle video begins.

[0039] In addition, when a predetermined operation (e.g., a left click of a mouse) is performed, each section displayed in the time display field R3 is configured to match the start time of the operated (selected) section video (section) to display the start time and end time of each section video. In the example of FIG. 8C, section 4 is selected, and the start time and end time of each section of the two cycle videos are displayed to match the start time of section 4. This makes it easy to compare the length of time required for the selected sections. At this time, the section videos of the selected movements (sections) in the two cycle videos may be displayed synchronously in the video display field R1, and may be loop-played (repeatedly played). This makes it possible to compare the movements of the worker in the two cycle videos for each section. In addition, when an instruction to switch the comparison target is given by using the switch buttons B1 to B3 while two cycle videos are displayed in the video display field R1, the comparison screen is configured to start playing the two cycle videos that were instructed to be switched from the same section (movement) as the section (movement) currently being displayed.

[0040] On the comparison screen configured as described above, when the cycle video to be displayed is changed using the switching buttons B1 to B3 in the video display field R1, the display contents of the information display field R2 and the time display field R3 are updated based on the changed cycle video. Also, when a command is given to play each section of each cycle video shown in the stacked graph D displayed in the information display field R2, or each section of each cycle video displayed in the time display field R3, the section video of that section is displayed in the video display field R1, and the display contents of the information display field R2 and the time display field R3 are updated based on the display content of the video display field R1.

[0041] The control unit 21 of the user terminal 20 receives the comparison screen transmitted by the server 10 and displays it on the display unit 25 (S38). As a result, the screen shown in FIG. 8A is displayed on the user terminal 20, and the user can compare the section videos (each movement) of the selected multiple cycle videos. The information display field R2 of the comparison screen is configured to display a "Find Similar Cycles" button as shown in FIG. 8A when a predetermined operation (e.g., selection with the cursor) is performed on any of the cycle videos in which the stacked graph D is displayed. When the user wants to check another cycle video (similar cycle video) to be compared among the cycle videos in which the stacked graph D is displayed in the information display field R2, the user selects (specifies) the cycle video and operates the "Find Similar Cycles" button. When the "Find Similar Cycles" button is operated, the server 10 searches for cycle videos to be compared with the selected cycle video, and the search results are displayed on the user terminal 20.

[0042] Furthermore, the time display field R3 of the comparison screen is configured to display a "Find similar movements" button as shown in FIG. 8D when a predetermined operation (e.g., selection with the cursor) is performed on any section of the cycle video being displayed. When the user wishes to check a section of another cycle video (similar segment video) to be compared among the sections of the cycle video displayed in the time display field R3, the user selects (specifies) that section and operates the "Find similar movements" button. When the "Find similar movements" button is operated, the server 10 searches for sections of cycle videos to be compared to the selected section, and the search results are displayed on the user terminal 20.

[0043] Therefore, control unit 21 determines whether or not an arbitrary cycle video has been selected in information display field R2 of the comparison screen and the "Find similar cycles" button has been operated (S39), and if it determines that this has not been done (S39: NO), it determines whether or not an arbitrary section of an arbitrary cycle video has been selected in time display field R3 and the "Find similar movements" button has been operated (S45). Note that if control unit 21 determines that the "Find similar movements" button has not been operated (S45: NO), it returns to step S39.

[0044] When it is determined that the "Search for Similar Cycles" button has been operated (S39: YES), the control unit 21 instructs the server 10 to search for cycle videos similar to the selected cycle video (S40). For example, the control unit 21 transmits information (e.g., file name, URL, etc.) indicating the selected cycle video to the server 10 and requests the execution of a search process for similar cycle videos. The control unit 11 of the server 10 searches for other cycle videos with a high similarity score (similarity) with the reference video based on the cycle video for which a search for similar cycle videos has been requested (hereinafter, may be referred to as a reference video) (S41). For example, the control unit 11 searches for cycle videos with cycle times similar to the reference video from other cycle videos of the same work as the reference video. A cycle video with a similar cycle time may be a cycle video whose difference in cycle time is less than a predetermined time. Furthermore, when extracting each cycle video in the cycle detection process, the control unit 11 calculates a similarity score between each section video in each cycle video and each section video of the training cycle video. Therefore, the control unit 11 may search for other cycle videos in which the similarity scores calculated for each section video of the reference video are similar to the similarity scores calculated for each section video. Furthermore, the control unit 11 may calculate the similarity between the reference video and other cycle videos, and search for cycle videos in which the similarity is high. The similarity may be calculated using, for example, a correlation coefficient or cosine similarity, or the similarity between two cycle videos may be estimated using a learning model constructed by machine learning. For example, a learning model made of CNN and trained to output the similarity between two cycle videos when two cycle videos are input may be used.

[0045] The control unit 11 also searches for cycle videos with high attention from among other cycle videos of the same work as the reference video (S42). As described later, various information such as comments and improvement plans can be added to each cycle video and each section video of each cycle video, and the control unit 11 determines whether or not a comment has been added, whether or not an improvement plan has been added, and whether or not the improvement plan (improvement process) has been completed for each other cycle video, and calculates the attention level based on the determination results. Work with a large number of cycle videos with comments and other information added is considered to be work with high attention. Therefore, cycle videos with calculated attention levels equal to or greater than a predetermined value are searched for as cycle videos with high attention levels. Specifically, the control unit 11 calculates the attention levels of other cycle videos from the determination results based on the contents stored in the ToDo list DB 12b.

[0046] The control unit 11 generates a comparison screen as shown in FIG. 8A based on the other cycle videos with high similarity scores searched for in step S41 and the other cycle videos with high attention levels searched for in step S42, and transmits the generated comparison screen to the user terminal 20 (S43). Here, the control unit 11 extracts a predetermined number of cycle videos in descending order of similarity score, extracts a predetermined number of cycle videos in descending order of attention levels, and generates a comparison screen that displays the extracted cycle videos. In this comparison screen, the reference video is displayed on the left side of the video display field R1, and one of the other cycle videos found (hereinafter referred to as the comparison video) is displayed on the right side. In the information display field R2, a stacked graph D of the reference video, the comparison video, and the other cycle videos is displayed from the left, and an indicator showing the start time and end time of each segment video (section) of the reference video and the comparison video is displayed in the time display field R3. The other cycle videos displayed on the comparison screen may be cycle videos with high similarity scores and cycle videos with high attention levels, as well as cycle videos with high similarity scores that are highly attention-grabbing. When the control unit 11 transmits the comparison screen to the user terminal 20, it associates the video URL (cycle video URL) of the reference video in the comparison screen with the current date and time, information about the user (executor) who requested the cycle video comparison, and information about other cycle videos presented on the comparison screen in the comparison analysis column of the video DB 12a. This enables the server 10 to collect information about each cycle video presented as a comparison target with the cycle video selected by the user.

[0047] The control unit 21 of the user terminal 20 receives the comparison screen sent by the server 10 and displays it on the display unit 25 (S44). This allows the user to check information on other cycle videos to be compared with the reference video for which a search for a similar cycle video has been requested. After processing step S44, the control unit 11 proceeds to step S45. If it is determined that the "Search for similar motion" button has been operated (S45: YES), it instructs the server 10 to search for a section video (motion) similar to the selected section video (S46). For example, the control unit 21 transmits information indicating the selected section video (e.g., file name, URL, etc.) to the server 10 and requests the execution of a search process for a similar motion. The control unit 11 of the server 10 searches for section videos of other cycle videos with a high similarity score (similarity) to the reference section video based on the section video for which a search for a similar motion (section video) has been requested (hereinafter, may be referred to as the reference section video) (S47). For example, the control unit 11 searches for section videos with the same motion as the reference section video in other cycle videos, and searches for section videos with similar motion times to the reference section video. The section videos with similar operation times can be section videos with a difference in operation time less than a predetermined time. Furthermore, the control unit 11 calculates a similarity score between each section video of each cycle video and each section video of the training cycle video in the cycle detection process. Therefore, the control unit 11 may search for section videos of other cycle videos for which a similarity score similar to the similarity score calculated for the reference section video is calculated. Furthermore, the control unit 11 may calculate the similarity between the reference section video and the section videos of other cycle videos, and search for section videos with high similarity. Here again, the similarity can be, for example, a correlation coefficient or cosine similarity, and the similarity between two section videos may be estimated using a learning model constructed by machine learning.

[0048] The control unit 11 also searches for section videos with high attention from section videos of other cycle videos with the same movement as the reference section video (S48). Here, the control unit 11 also calculates the attention level from the results of judging whether or not a comment is added to the section videos of the other cycle videos, whether or not an improvement plan is added, and whether or not the implementation of the improvement plan has been completed. Since a movement with a large number of section videos with information such as comments added thereto is considered to be a movement with high attention, section videos with the calculated attention level equal to or higher than a predetermined value are searched for as section videos with high attention. The control unit 11 generates a comparison screen based on the section videos of other cycle videos with high similarity scores searched for in step S47 and the section videos of other cycle videos with high attention levels searched for in step S48, and transmits the generated comparison screen to the user terminal 20 (S49). Here, the control unit 11 extracts a predetermined number of section videos in descending order of similarity score, extracts a predetermined number of section videos in descending order of attention level, and generates a comparison screen that displays cycle videos including the extracted section videos. Again, the control unit 11 stores the current date and time, information about the user (executor) who requested the comparison of the section videos, and information about the other cycle videos presented on the comparison screen in the comparison analysis column of the video DB 12a in association with the video URL (cycle video URL) of the cycle video including the reference section video on the comparison screen. This enables the server 10 to collect information about each section video presented as a comparison target for the section video selected by the user.

[0049] In this comparison screen, a cycle video including the reference segment video is displayed on the left side of a video display field R1, and one of the cycle videos including the other segment videos found through the search (comparison video) is displayed on the right side. In addition, in information display field R2, from left to right, a stacked graph D of the cycle video including the reference segment video, the comparison video, and other cycle videos is displayed, and in time display field R3, an indicator is displayed showing the start time and end time of each segment video (section) of the cycle video including the reference segment video and the comparison video. The other cycle videos displayed on the comparison screen may be cycle videos including segment videos with high similarity scores and cycle videos including segment videos with high attention, as well as cycle videos including segment videos with high similarity scores that include segment videos with high attention.

[0050] The control unit 21 of the user terminal 20 receives the comparison screen sent by the server 10 and displays it on the display unit 25 (S50). This allows the user to check information on other cycle videos that are section videos of the same movements as the reference section video for which a search for similar movements has been requested and that should be compared. By the above-mentioned process, other cycle videos with high similarity scores and other cycle videos with high attention can be extracted for an arbitrary cycle video, and each section video can be presented in a synchronized manner. In addition, for an arbitrary cycle video, cycle videos including section videos with high similarity scores and section videos with high attention can be extracted, and the section videos can be presented in a synchronized manner. Thus, the user can compare the work of each worker and the movements included in the work using multiple cycle videos and multiple section videos.

[0051] Next, a process of adding a comment to an arbitrary cycle video or section video and a process of registering the comment in a ToDo list in the information processing system of this embodiment will be described. Fig. 9 is a flowchart showing an example of the procedure of the comment adding process, and Figs. 10 to 13 are explanatory diagrams showing example screens. The user of user terminal 20 in the process of Fig. 9 is a person in charge who analyzes the state of work based on the cycle video and registers comments, etc., such as the worker himself or the worker's superior.

[0052] In this embodiment, for example, a comment can be added (appended) to the cycle video being displayed from the detection result screen shown in FIG. 6C, and a comment can be added to the division video of the cycle video being displayed from the comparison screen shown in FIG. 8A. Therefore, when the control unit 21 is displaying a screen such as the detection result screen or the comparison screen (S61), it executes the processes from step S62 onwards. The detection result screen is configured to display an "add comment" button when a predetermined operation (e.g., selection with the cursor) is performed on any of the cycle videos displayed in the time display field R3 as shown in FIG. 10A. When a cycle video to which a user wants to add a comment is found among the cycle videos displayed in the time display field R3 of the detection result screen, the user selects the cycle video and operates the "add comment" button. Also, the comparison screen is configured to display an "add comment" button when a predetermined operation (e.g., selection with the cursor) is performed on any of the sections (division videos) of the cycle video displayed in the time display field R3 as shown in FIG. 10B. When a division video to which a user wants to add a comment is found among the division videos of the cycle videos displayed in the time display field R3 of the comparison screen, the user selects the division video and operates the "add comment" button.

[0053] Therefore, the control unit 21 judges whether or not an arbitrary cycle video has been selected on the detection result screen and the "Add comment" button has been operated, or whether or not an arbitrary division video has been selected on the comparison screen and the "Add comment" button has been operated (S62). If the control unit 21 judges that the "Add comment" button has not been operated (S62: NO), it waits until it has been operated, and if it judges that it has been operated (S62: YES), it displays a comment input screen as shown in Fig. 10C on the display unit 25 (S63). The screen in Fig. 10C is a screen when an instruction to add a comment to the cycle video of video 2 is given via the screen in Fig. 10A, and has a comment input field, a save button, and a cancel button.

[0054] The control unit 21 accepts the input of a comment on the comment input screen (S64), displays the input comment in the input field, and determines whether the save button has been operated (S65). If it is determined that the save button has not been operated (S65: NO), the control unit 21 continues to accept the input of a comment. If it is determined that the save button has been operated (S65: YES), the control unit 21 instructs the server 10 to save the input comment (S66). The control unit 21 transmits information indicating the cycle video or section video selected as the target for adding a comment (e.g., file name, URL, etc.), the comment, and information on the person who input the comment to the server 10, and requests that the comment be saved.

[0055] When the control unit 11 of the server 10 acquires the URL of a cycle video or a section video from the user terminal 20, it stores the acquired URL (video URL), information on the person who input the comment, and the comment in ToDo list DB 12b in association with each other (S67). When the control unit 11 acquires identification information such as the file name of the cycle video or section video, it identifies the URL of the cycle video or section video from the acquired identification information and stores the identified URL in ToDo list DB 12b. The control unit 11 generates a comment list screen that displays a list of the comments stored in ToDo list DB 12b and transmits it to the user terminal 20 (S68).

[0056] The control unit 21 of the user terminal 20 acquires the comment list screen transmitted by the server 10 and displays it on the display unit 25 (S69). Fig. 11A shows an example of the comment list screen, and the screen in Fig. 11A displays the comments stored in the ToDo list DB 12b and information on the cycle video and section video to which each comment has been added, and is provided with an "Add to ToDo List" button associated with each comment (each cycle video and section video) for instructing to add the cycle video and section video to the ToDo list. The control unit 21 determines whether or not the "Add to ToDo List" button corresponding to any comment has been operated on the comment list screen (S70), and if it determines that it has not been operated (S70: NO), it waits until it is operated, and if it determines that it has been operated (S70: YES), it requests the server 10 to provide a ToDo list screen (S71).

[0057] When a ToDo list screen is requested, the control unit 11 of the server 10 generates a ToDo list screen as shown in FIG. 12A based on the stored contents of the ToDo list DB 12b and transmits it to the user terminal 20 (S72). The control unit 21 of the user terminal 20 acquires the ToDo list screen transmitted by the server 10 and displays it on the display unit 25 (S73). The screen of FIG. 12A displays the comment and inputter information input via the screens of FIG. 10A to FIG. 10C, and the video URL and video data as information of the selected cycle video or section video. In addition, when ToDo data for each video is stored in the ToDo list DB 12b, the ToDo list screen displays, as ToDo data, a comment on the improvement plan, a status indicating the implementation status of the improvement plan, a URL of a video taken after the implementation of the improvement plan (improved video URL), and contents improved by the implementation of the improvement plan. The comment on the improvement plan may include information of the inputter of the improvement plan, information of the person in charge of implementing the improvement plan, and a deadline for the improvement plan. The ToDo list screen allows the addition of items other than comments and improvement plans; for example, when a specified operation (e.g. right-clicking the mouse) is performed on the displayed item name (comment, video URL, etc.), an input screen for instructing the addition of an item name is displayed, and the screen is configured so that an item of any item name can be added.

[0058] The ToDo list screen shown in FIG. 12A is configured so that the information of each displayed item can be edited, and editing becomes possible when a predetermined operation (e.g., double-clicking the mouse) is performed on the display field of any item. In FIG. 12A, a predetermined operation is performed on the display field of the improvement plan for the comment "Put tools in their designated place after use", and this display field is editable (the cursor is displayed). Also, in FIG. 12A, editing of the status item is requested, and in this case, a status editing screen is displayed as shown in FIG. 12B. Therefore, when the status is edited via the status editing screen and the OK button is operated, the status information in the ToDo list screen is changed.

[0059] The control unit 21 accepts edits to the information (ToDo data) of each item being displayed on the ToDo list screen (S74), displays the edited information in the input field, and judges whether the update button has been operated (S75). If it is determined that the update button has not been operated (S75: NO), the control unit 21 continues to accept edits to the ToDo data. If it is determined that the update button has been operated (S75: YES), the control unit 21 instructs the server 10 to save each piece of edited information (edited ToDo data) (S76). The control unit 21 transmits each piece of information displayed on the ToDo list screen to the server 10, requesting that the ToDo data be saved.

[0060] The control unit 11 of the server 10 stores the ToDo data acquired from the user terminal 20 in the ToDo list DB 12b in association with the URL of the cycle video or the section video (S77). The above-mentioned process allows a comment to be associated with any cycle video or section video and stored, and in addition to the comment, information (data) related to the cycle video or section video, such as information related to an improvement plan, information to be kept, information to be shared with other users, and the like, can be associated and stored. When the above-mentioned process has been used to store information (comments, etc.) related to the cycle video or section video in the ToDo list DB 12b, the information associated with the cycle video or section video can be displayed on a comparison screen, for example, as shown in FIG. 11B. In the example of FIG. 11B, section 4 of the cycle video of video 2 is selected via the stacked graph D displayed in the information display field R2, and the comment added to this section (section video) is displayed. This allows the comments, etc. added to each cycle video and each section video to be confirmed when comparing and analyzing multiple videos, and can be referred to when considering an improvement policy.

[0061] On the screen in Fig. 11B, a "Check ToDo List" button is displayed in the display field for comments added to the video, and when the "Check ToDo List" button is operated, the control unit 21 requests a ToDo list screen from the server 10 and can obtain and display a ToDo list screen such as that shown in Fig. 12A from the server 10. This makes it possible to check various information (ToDo data) added to the cycle video or the section video from the comparison screen.

[0062] Furthermore, the ToDo list screen allows for the translation of information written in natural language for each item, and when a specific operation (e.g., right-clicking the mouse) is performed on any item as shown in FIG. 13A, a translation menu is displayed, along with a button for selecting the language to be translated. When the button for selecting the language is operated, the information for the selected item is translated into the selected language. When a translation into English is requested for the comment item in FIG. 13A, the control unit 21 adds and displays an English translation of the comment as shown in FIG. 13B. With this configuration, in a company that has factories (work spaces) in multiple countries, the person in charge in the other countries can grasp information such as comments registered at each factory, and each piece of information can be shared between the people in the multiple countries.

[0063] As described above, in this embodiment, it is possible to execute a plurality of types of processes, such as a process of uploading a video to the server 10, a cycle detection process of dividing an uploaded video into cycle videos, a comparison and analysis process of comparing divided cycle videos or section videos, a process of adding comments to cycle videos or section videos, and a process of registering the implementation status of an improvement plan for an operation or motion captured in a cycle video or section video in the ToDo list DB 12b. In addition, by storing each piece of information in the video DB 12a and the ToDo list DB 12b, the implementation status of each process is linked to the cycle video or section video. In factories and companies that introduce the information processing system of this embodiment, there is a demand to monitor the usage status of the system, the implementation status of an improvement plan in each factory or by each worker, etc. Therefore, the information processing system of this embodiment has a function of presenting the implementation status (number of times of execution) of each of the above-mentioned processes for each factory (department, work space) or for each person in charge.

[0064] Next, a process for monitoring the usage status of the system and the implementation status of an improvement plan in the information processing system of this embodiment will be described. Fig. 14 is a flow chart showing an example of a monitoring process procedure, and Fig. 15 is an explanatory diagram showing an example screen. The user of the user terminal 20 in the process of Fig. 14 is a person in charge of monitoring the usage status of the system and the implementation status of an improvement plan in each factory or for each worker, such as the worker's superior, the factory manager, or a person in charge at the head office who supervises each factory.

[0065] When execution of the monitor function is instructed by a predetermined operation via the input unit 24, the control unit 21 of the user terminal 20 displays a monitor screen as shown in Fig. 15A (S81). The screen in Fig. 15A has input fields for the period to be monitored, the affiliation (such as a factory), and the person in charge (such as a worker), and each input field is provided with a pull-down menu for selecting a selectable period, affiliation, and person in charge. Note that the period to be monitored can be selected in units of one week, one month, one year, etc., the affiliation can be selected as one or multiple affiliations or all, and the person in charge can be selected as one or multiple affiliations or all. The screen of FIG. 15A also has checkboxes for selecting, as monitor items, the number of videos uploaded to server 10 (number of uploaded videos), the number of videos that have undergone cycle detection processing (number of cycle detection videos), the number of cycle videos or section videos that have been comparatively analyzed (number of comparative analysis videos), the number of cycle videos or section videos registered in ToDo list DB12b (number of videos in the ToDo list), the number of cycle videos or section videos for which implementation of an improvement plan has been completed (number of improved videos), and the number of videos whose improved videos have been registered in ToDo list DB12b (number of registered improved videos).

[0066] The control unit 21 receives an input of the monitoring target via the input unit 24 on the monitor screen (S82), judges whether the OK button has been operated (S83), and continues to receive the input of the monitoring target when it is judged that the OK button has not been operated (S83: NO). When it is judged that the OK button has been operated (S83: YES), the control unit 21 transmits the input monitoring target to the server 10 (S84) and requests monitoring of the transmitted monitoring target. The control unit 11 of the server 10 counts the monitoring results for the monitoring target requested from the user terminal 20 based on the storage contents of the video DB 12a and the ToDo list DB 12b (S85). For example, when the monitoring target includes the number of uploaded videos, the control unit 11 counts the number of videos corresponding to the period, department, and person in charge of the monitoring target based on the storage contents of the video DB 12a. When the monitoring target includes the number of cycle detection videos, the control unit 11 counts the number of cycle videos corresponding to the period, department, and person in charge of the monitoring target based on the storage contents of the cycle detection column of the video DB 12a.

[0067] The control unit 11 transmits the counted monitor results to the user terminal 20 (S86), and the control unit 21 of the user terminal 20 displays the monitor results acquired from the server 10 on the monitor screen (S87). As a result, a monitor screen as shown in FIG. 15B is displayed on the display unit 25 of the user terminal 20. On the screen of FIG. 15B, the number of uploaded videos, the number of cycle detection videos, the number of comparison analysis videos, the number of videos in the ToDo list, and the number of videos with improvement completed are selected as monitor targets, and the monitor results showing the number of videos in each bar graph are displayed. As a result, the usage status of the information processing system of this embodiment and the implementation status of the improvement plan for the video with the improvement plan registered in the ToDo list can be monitored for each factory (work space) or for each person in charge. Therefore, it is possible to promote the use of the information processing system of this embodiment and the implementation of the improvement plan.

[0068] In this embodiment, a video captured by a camera is uploaded to the server 10, and various processes are performed by the server 10, but the present invention is not limited to this configuration. For example, the processes performed by the server 10 and the user terminal 20 may be performed by a single device. Even in such a configuration, the above-described processes of this embodiment can be performed, and similar effects can be obtained.

[0069] (Embodiment 2) In this embodiment, an information processing system is described in which each section (section video) of a cycle video displayed in the time display field R3 on the comparison screen is highlighted when there are other cycle videos to view (or many other cycle videos). With this configuration, when performing a comparative analysis with other cycle videos on the comparison screen, other cycle videos to view can be easily selected. The information processing system of this embodiment can be realized by devices similar to the devices 10, 20 of the information processing system of embodiment 1 shown in Figure 2, so a description of the configuration of the devices 10, 20 will be omitted.

[0070] Fig. 16 is a flow chart showing an example of the comparison process procedure of the second embodiment, and Fig. 17 is an explanatory diagram showing an example screen. The process shown in Fig. 16 is obtained by adding steps S91 to S94 between steps S36 and S37, and adding steps S95 to S97 instead of steps S39 to S48 in the process shown in Fig. 7. Explanations of the same steps as in Fig. 7 will be omitted.

[0071] In the information processing system of this embodiment, after the process of step S36, the control unit 11 of the server 10 calculates a similarity score (similarity) for each section video of a cycle video displayed in the time display field R3 of the comparison screen with section videos of the same movement in other cycle videos of the same work (S91). Then, based on the calculated similarity score, the control unit 11 searches for section videos of other cycle videos that have a high similarity score with each section video being displayed (S92). For example, the control unit 11 searches for section videos with similar movement times, section videos with a similar similarity score with the section video of a training cycle video calculated in the cycle detection process, section videos with a high similarity score with the section video of the displayed cycle video, etc.

[0072] The control unit 11 also searches for section videos with high attention from section videos of other cycle videos of the same work as the displayed cycle video (S93). Here, the control unit 11 calculates the attention level for the section videos of other cycle videos from the determination results of whether or not a comment has been added, whether or not an improvement plan has been added, and whether or not the implementation of the improvement plan has been completed, and searches for section videos with high attention. Then, for each section video being displayed, the control unit 11 identifies section videos to be highlighted based on the number of section videos of other cycle videos with high similarity scores and the number of section videos with high attention, and highlights the identified section videos (S94). For example, the control unit 11 identifies a section video as one to be highlighted when the number of section videos with high similarity scores, the number of section videos with high attention, or the total of both section videos is equal to or greater than a predetermined number. In the screen of FIG. 17A, the section videos being displayed in the time display field R3 are highlighted by hatching. Note that the section videos may be highlighted by adding a predetermined mark. Thereafter, the control unit 11 transmits a comparison screen on which some of the divided videos are highlighted to the user terminal 20 (S37), and the control unit 21 of the user terminal 20 displays the comparison screen on the display unit 25 (S38).

[0073] As shown in FIG. 17A, the comparison screen of this embodiment is configured to display a "Watch" button along with a message "Faster videos are available" when a predetermined operation (e.g., selection with the cursor) is performed on a section (section video) highlighted in the time display field R3. Thus, the control unit 21 determines whether the "Watch" button has been operated on the comparison screen (S95), and when it is determined that the button has been operated (S95: YES), it requests the server 10 for other videos to be compared (S96). When other videos are requested, the control unit 11 of the server 10 reads out from the video folder 12c cycle videos including the section videos with high similarity scores searched for in step S92 and cycle videos including the section videos with high attention levels searched for in step S93 (S97). The control unit 11 then generates a comparison screen displaying the read cycle videos and transmits the comparison screen to the user terminal 20 (S49). The control unit 21 of the user terminal 20 obtains comparison images from the server 10 and displays the comparison images as shown in FIG. 17B (S50). On the comparison screen here, the cycle video of video 2 is displayed as the reference video, and the cycle video of video 11 is displayed as the comparison video. Note that the cycle videos of videos 11 to 14 are cycle videos including the section videos searched for in steps S92 to S93.

[0074] By the above-mentioned process, in this embodiment, for each section video of the cycle video displayed in the time display field R3 of the comparison screen, if there is another video to be compared and analyzed, it is highlighted. Therefore, the user can easily select the video to be compared and analyzed. In addition, since other videos with high attention are presented for the selected video, it is possible to efficiently consider an improvement plan by viewing information such as comments given to the other videos. The configuration of this embodiment is applicable to the information processing system of the first embodiment, and the same effect can be obtained even when applied to the information processing system of the first embodiment. In addition, the modified examples described as appropriate in the above-mentioned first embodiment can be applied to this embodiment as well.

[0075] (Embodiment 3) The first and second embodiments described above are configured to compare cycle videos of the same tasks and to compare segmented videos contained in the cycle videos. In this embodiment, an information processing system is described that compares different tasks based on cycle videos of different tasks. The information processing system of this embodiment can be realized by devices similar to the devices 10, 20 of the information processing system of the first embodiment shown in FIG. 2, and therefore a description of the configuration of the devices 10, 20 is omitted.

[0076] In the manufacturing industry, a single product is completed by performing multiple types of work in a chronological order. In some cases, a line production system is adopted in which multiple workers are responsible for each work. In the line production system, workers' skills improve through repetition of work, making it possible to efficiently manufacture high-quality products. In addition, in the line production system, each work is required to be completed in the same amount of time, and if even one person is slow at work, this work becomes a bottleneck and reduces overall productivity. Therefore, in this embodiment, by comparing different works, it is possible to identify the work that will become a bottleneck.

[0077] Fig. 18 is a flow chart showing an example of a comparison process between different tasks, and Fig. 19 is an explanatory diagram showing an example screen. The user of the user terminal 20 in the process of Fig. 18 is a person in charge of analyzing the status of each task performed in chronological order, such as a factory manager or a person in charge at the head office who supervises each factory.

[0078] In the information processing system of this embodiment, when an instruction to perform an inter-task comparison is given by a predetermined operation via the input unit 24, the control unit 21 of the user terminal 20 displays an inter-task comparison screen as shown in FIG. 19A on the display unit 25 (S101). The screen of FIG. 19A has input fields for the period and affiliation (such as factory) to be compared, and each input field is provided with a pull-down menu for selecting a selectable period and affiliation. The period to be compared can be selected in units of one week, one month, one year, etc., and the affiliation can be selected from one or more affiliations (factories, etc.) that manufacture the same product, or all of them. The screen of FIG. 19A also has check boxes for selecting each work process (task) performed in the selected affiliation (factory or work space) to be compared.

[0079] The control unit 21 receives an input of a comparison target via the input unit 24 on the task comparison screen (S102), determines whether the OK button has been operated (S103), and continues to receive the input of the comparison target if it determines that the OK button has not been operated (S103: NO). If it determines that the OK button has been operated (S103: YES), the control unit 21 transmits the input comparison target to the server 10 (S104) and requests that a comparison process be performed on the transmitted comparison target. The control unit 11 of the server 10 extracts cycle videos that match the comparison target for the comparison process requested by the user terminal 20 (S105). For example, the control unit 11 identifies cycle videos of affiliations (locations, work spaces) and task processes that match the comparison target based on information on the shooting locations and shooting targets stored in the video DB 12a, and extracts the identified cycle videos.

[0080] The control unit 11 calculates (measures) the time required for each work process (work time, cycle time) based on the extracted cycle video of each work process, and calculates the minimum, first quartile, median, third quartile, and maximum value (values ​​indicating the variation in work time) for each work process based on the calculated work time (S106). Then, the control unit 11 generates a box-and-whisker plot (diagram) showing the variation in work time for each work process based on each calculated value, and generates a comparison result screen (S107). For example, the control unit 11 generates a comparison result screen that displays a box-and-whisker plot as shown in FIG. 19B. In the box-and-whisker plot shown in FIG. 19B, each work process is associated with each position on the horizontal axis, the vertical axis indicates the work time of each work process, and the variation in work time is expressed by the box and whiskers indicating the minimum, first quartile, median, third quartile, and maximum value of the work time, respectively. As shown in FIG. 19B, in the box plot of each operation, a standard time previously set for each operation may be indicated by a dashed line, and an average value of the operation time for each operation may be plotted by a black circle.

[0081] The control unit 11 transmits the generated comparison result screen to the user terminal 20 (S108), and the control unit 21 of the user terminal 20 displays the comparison result screen acquired from the server 10 on the display unit 25 (S109). As a result, the comparison result screen as shown in FIG. 19B is displayed on the user terminal 20. On the screen of FIG. 19B, the first to third processes are selected as the comparison targets, and a box plot showing the variation in the respective work times is displayed. This makes it possible to compare the work times in each work process required to complete one product, and to grasp the variation in each work time.

[0082] The screen of FIG. 19B is configured such that each point in the box-and-whisker plot is associated with a cycle video having a corresponding work time, and a selected cycle video is played back as shown in FIG. 19C by performing a predetermined operation (e.g., left-clicking the mouse) on any point in the box-and-whisker plot. The screen of FIG. 19C has a similar configuration to the comparison screen of FIG. 8A, and the selected cycle video is displayed in a video display field R1, a stacked graph D of the selected cycle video is displayed in an information display field R2, and the start time and end time of each segment video (section) of the selected cycle video are displayed in a time display field R3. As with the screens of FIG. 8A and FIG. 8D, the screen of FIG. 19C also allows a search for other cycle videos similar to the displayed cycle video, and allows a search for other segment videos similar to the displayed cycle video. Therefore, the factory manager, the person in charge at headquarters who supervises each factory, etc. can check the variation in work time for each task via the screen of Figure 19B, and then, for example, for tasks that take a long (slow) time, they can compare it with other cycle videos of the same task (other workers) via the screens of Figures 19C and 8A, etc., and consider improvement measures to make each task more efficient.

[0083] By the above-mentioned process, in this embodiment, it is possible to compare the work time (cycle time) between different tasks based on cycle videos of different tasks, and this can be used to identify bottlenecks in a series of work processes. The configuration of this embodiment is applicable to the information processing systems of the first and second embodiments, and similar effects can be obtained even when applied to the information processing systems of the first and second embodiments. Furthermore, in this embodiment, the modified examples described as appropriate in the above-mentioned first to second embodiments can also be applied.

[0084] In the above-mentioned first to third embodiments, the video of a worker performing a task is the processing target, but the processing target is not limited to the worker. For example, based on a video of a robot configured to perform a predetermined task, the task performed by the robot and the motions included in the task may be compared with other robots or workers.

[0085] The matters described in the above-mentioned embodiments can be combined with each other. In addition, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations regardless of the citation format. Furthermore, the claims use a format in which a claim cites two or more other claims (multiple claim format), but this is not limited to this. A multiple claim (multi-multi claim) that cites at least one multiple claim may also be used.

[0086] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0087] 10 Server 11 Control section 12 Storage section 13. Communications Department 20 User terminal 21 Control section 22 Memory section 23 Communications Department M Learning Model 12a Video DB 12b ToDo List DB

Claims

1. A video is taken by a photographing device attached to a subject, and multiple videos are obtained in which the subject performs the same task in different work spaces, The acquired multiple videos are input into a learning model that has been trained to classify a video of a work space in which a subject is working into each of the actions included in the work, and each of the multiple videos is classified into each of the actions included in the work. Based on the divided videos obtained by dividing the plurality of videos into groups for each action, the divided videos of each action in the plurality of videos, which are taken in different work spaces where the subject performs the same work, are output in a synchronized manner. A program that causes a computer to carry out processing.

2. The video includes a video captured by a camera with a fixed shooting position. The program according to claim 1.

3. Data relating to the video is associated with the video; Accepting designation of one video from among the plurality of videos; Extract other videos that have a high similarity to the specified video from among other videos of the same work as the specified video; outputting the designated moving image and the extracted other moving image by synchronizing the divided moving images of each moving image; When outputting the video, data associated with the video is output.

3. The program according to claim 1, which causes the computer to execute a process.

4. Data relating to the segmented video is associated with the segmented video, Accepting designation of one section video from among the sections of the video; Extracting a segmented video of another video having the same motion as the designated segmented video, the segmented video having a high similarity to the designated segmented video; outputting the designated section video and the extracted section video of the other video in synchronization with each other; When outputting the divided video, data associated with the divided video is output.

3. The program according to claim 1, which causes the computer to execute a process.

5. For each of the divided videos in the video, a similarity between the divided videos of the same action in other videos and an attention level to the divided videos of the same action in the other videos are obtained; highlighting a segment of the video in accordance with the acquired similarity and attention level; Accepting designation of one section video from among the highlighted section videos of the videos; Extracting a segment video from among other videos having the same motion as the designated segment video based on the similarity and attention level with the designated segment video; The designated section video and the extracted section video from the other video are output in synchronization with each other.

3. The program according to claim 1, which causes the computer to execute a process.

6. The data about the video is written in natural language; The data relating to the video is translated into a specified language and output. The program according to claim 3, which causes the computer to execute a process.

7. The video is associated with a plurality of types of processing execution statuses for the video; Counting the number of times each process is performed on the video of the object or the workspace for each of the objects or workspaces; Outputting the counted number of times each process is executed for each of the objects or each of the work spaces.

3. The program according to claim 1, which causes the computer to execute a process.

8. From a video of multiple tasks performed in chronological order, a cycle video of each task is extracted. measuring a work time required for each of the tasks based on a plurality of cycle videos of each of the tasks; Output a chart showing the variation in the measured work time for each task.

3. The program according to claim 1, which causes the computer to execute a process.

9. A video is taken by a photographing device attached to a subject, and a plurality of videos are obtained in which the subject takes pictures of different work spaces performing the same task, The acquired multiple videos are input into a learning model that has been trained to classify a video of a work space in which a subject is working into each of the actions included in the work, and each of the multiple videos is classified into each of the actions included in the work. Based on the divided videos obtained by dividing the plurality of videos into groups for each action, the divided videos of each action in the plurality of videos, which are taken in different work spaces where the subject performs the same work, are output in a synchronized manner. An information processing method in which processing is performed by a computer.

10. In an information processing device having a control unit, The control unit is Obtaining a plurality of videos taken by a photographing device attached to a subject, the videos capturing different work spaces in which the subject performs the same task; The acquired multiple videos are input into a learning model that has been trained to classify a video of a work space in which a subject is working into each of the actions included in the work, and each of the multiple videos is classified into each of the actions included in the work. Based on the divided videos obtained by dividing the plurality of videos into groups for each action, the divided videos of each action in the plurality of videos, which are taken in different work spaces where the subject performs the same work, are output in a synchronized manner. Information processing device.

11. A video is acquired by a photographing device attached to a subject, the video being a plurality of videos of work spaces in which the subject performs the same work and in which the arrangement of items related to the same work is different from each other, The acquired multiple videos are input into a learning model that has been trained to classify a video of a work space in which a subject is working into each of the actions included in the work, and each of the multiple videos is classified into each of the actions included in the work. Based on the divided videos obtained by dividing the plurality of videos into groups for each action, the divided videos of each action in the plurality of videos, which are taken in the work spaces in which the items are arranged in different states, are output in a synchronized manner. A program that causes a computer to carry out processing.

12. A video is taken by a camera attached to a subject, and multiple videos are obtained in which the subject takes a same task in the same work space, The acquired multiple videos are input into a learning model that has been trained to classify a video of a work space in which a subject is working into each of the actions included in the work, and each of the multiple videos is classified into each of the actions included in the work. Based on the divided videos obtained by dividing the plurality of videos into groups for each action, the divided videos of each action in the plurality of videos, which are taken in the same working space where the subject performs the same work, are output in a synchronized manner. A program that causes a computer to carry out processing.

Images