Program, information processing method and information processing device

Abstract

To provide a program, etc. enabling synchronized presentation of categorized videos of each action within a plurality of videos obtained by capturing work scenes.SOLUTION: A computer categorizes a video obtained by capturing a work space of a subject during work, by each action included in the work. Further, the computer outputs synchronized categorized videos for each action in a plurality of videos on the basis of the categorized videos obtained by categorizing the plurality of videos by action, including videos obtained by capturing different work spaces where the subject performs the same work or videos obtained by capturing work spaces where arrangement states of items related to the same work are different.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a program, an information processing method, and an information processing device. [Background technology]

[0002] Patent Document 1 discloses a technique for detecting partial videos corresponding to partial movements included in an overall movement from an overall video captured of the overall movements of a trainee performing training, and evaluating the overall movement and the partial movements by referring to the partial videos. The technique disclosed in Patent Document 1 evaluates the overall movement and the partial movements based on the presence or absence of partial videos corresponding to each partial movement, the order of the partial videos, the duration of the partial videos, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6757010 Summary of the Invention [Problem to be solved by the invention]

[0004] When evaluating the movements of each trainee, there is a desire to compare them with the movements of other trainees or the movements of the same trainee at different times. However, in Patent Document 1, evaluation is based on whether or not partial movements have been performed, whether or not the partial movements have been performed in a predetermined order, and whether or not the partial movements have been performed for a predetermined period of time, and does not take into consideration comparison with the movements of other trainees or the movements of the same trainee at different times.

[0005] The present disclosure aims to provide a program or the like that can present a work situation captured in multiple videos by synchronizing segmented videos of each action in the videos. [Means for solving the problem]

[0006] A program according to one aspect of the present disclosure causes a computer to execute a process of dividing a video taken of a workspace in which a subject is working into each action included in the work, and synchronizing and outputting divided videos of each action in a plurality of videos based on the divided videos obtained by dividing multiple videos, including videos taken of different workspaces in which the subject is performing the same work, or videos taken of workspaces in which the arrangement of items related to the same work is different. [Effects of the Invention]

[0007] In one aspect of the present disclosure, a work situation captured in a plurality of videos can be presented by synchronizing segmented videos of each action in the videos. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 10 is an explanatory diagram showing an example of a moving image to be processed. [Figure 2] FIG. 1 is a block diagram illustrating an example of the configuration of an information processing system. [Figure 3] FIG. 1 is an explanatory diagram illustrating an example of the configuration of a learning model. [Figure 4] 10 is an explanatory diagram showing an example of the record layout of a video DB and a ToDo list DB. FIG. [Figure 5] 10 is a flowchart illustrating an example of an upload process procedure and a cycle detection process procedure. [Figure 6] FIG. 10 is an explanatory diagram showing an example of a screen. [Figure 7] 10 is a flowchart showing an example of a comparison process procedure for cycle videos. [Figure 8] FIG. 10 is an explanatory diagram showing an example of a screen. [Figure 9] 10 is a flowchart illustrating an example of a procedure for adding a comment. [Figure 10] FIG. 10 is an explanatory diagram showing an example of a screen. [Figure 11] FIG. 10 is an explanatory diagram showing an example of a screen. [Figure 12] FIG. 10 is an explanatory diagram showing an example of a screen. [Figure 13] FIG. 10 is an explanatory diagram showing an example of a screen. [Figure 14] 10 is a flowchart illustrating an example of a monitoring process procedure. [Figure 15] FIG. 10 is an explanatory diagram showing an example of a screen. [Figure 16] 10 is a flowchart showing an example of a comparison processing procedure according to the second embodiment. [Figure 17] FIG. 10 is an explanatory diagram showing an example of a screen. [Figure 18] 10 is a flowchart illustrating an example of a comparison process procedure between different tasks. [Figure 19] FIG. 10 is an explanatory diagram showing an example of a screen. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, a program, an information processing method, and an information processing device according to the present disclosure will be described in detail with reference to the drawings illustrating embodiments thereof.

[0010] (Embodiment 1) In this embodiment, an information processing system is described that compares and presents multiple videos of a worker (subject) performing a task. FIG. 1 is an explanatory diagram showing an example of a video to be processed. The video to be processed in this embodiment is video data containing multiple images (still images), for example, 30 or 15 frames per second, and is a captured image of a worker performing a task, a workspace where the worker is working, etc., as shown in FIG. 1A. The videos to be compared in this embodiment include, for a single task (same task), videos of the same worker in the same workspace, videos of different workers in the same workspace, and videos of workers (the same worker and different workers) in different workspaces. Note that the different workspaces may be different workspaces in the same location (factory) or in different locations. Furthermore, as long as the tasks are the same, each video can be compared regardless of whether the workspaces have different arrangements of the items to be worked on and the items used for the task (items related to the task) or whether the worker is left-handed or right-handed.

[0011] The videos compared in this embodiment include videos captured with a camera with a fixed shooting position and videos captured with a camera attached to a worker. The upper left and upper right of FIG. 1A show examples of videos captured with a camera with a fixed shooting position. The upper left shows a video of the worker and the workspace from above and in front of the worker, and the upper right shows a video of the worker and the workspace from above and to the left of the worker. The lower left and lower right of FIG. 1A show examples of videos captured with a wearable camera attached to a worker, capturing the worker's hands and the workspace. Wearable cameras are, for example, ear-hook type, headband type (head strap type), or eyeglass type cameras, and are worn on the worker's head (face). The wearable camera may be attached to the temples of the worker's eyeglasses, the brim of a hat, or a helmet, or may be attached to the worker's clothing or body using a clip, chest harness, or the like. Therefore, in the system of this embodiment, for the same task, videos taken in different locations and different work spaces, videos taken from different shooting directions and angles of view relative to the work space, etc. can be compared.

[0012] Furthermore, the video to be processed in this embodiment may not only be a video (hereinafter referred to as a cycle video) of a worker performing a series of tasks, but also a video including multiple cycle videos, as shown in FIG. 1B. The upper side of FIG. 1B shows a video of a worker repeatedly performing task 1 (assembling case A), and the lower side of FIG. 1B shows a video of a worker performing task 1 (assembling case A), task 2 (assembling part A), etc. In this way, the video to be processed may be a video including multiple cycle videos of the same task, or may be a video including multiple cycle videos of different tasks. Furthermore, the worker performing each task may be the same worker or different workers. Different tasks are, for example, multiple tasks required to assemble a single product. Furthermore, the cycle video may include videos segmented into multiple actions included in each task (hereinafter referred to as segmented videos). In the example shown at the top of Figure 1B, the video to be processed can be divided into cycle videos of a worker performing task 1 (assembling case A), and each cycle video can be divided into segment videos of a worker performing actions such as action 1 (removing the case), action 2 (tightening the screws), etc.

[0013] FIG. 2 is a block diagram showing an example of the configuration of an information processing system. The information processing system of this embodiment includes a server 10 and a user terminal 20, which are communicatively connected via a network N. The network N may be the Internet or a public communication line, or a local area network (LAN) constructed within the facility where the server 10 is installed. The server 10 is an information processing device capable of various information processing and transmitting and receiving information, and is configured, for example, by a server computer, a personal computer, a workstation, etc. The user terminal 20 is an information processing device capable of various information processing and transmitting and receiving information, and is configured, for example, by a personal computer, a tablet terminal, a smartphone, etc. The user terminal 20 is a terminal used by a user who uses the server 10. The users in this embodiment include personnel who perform various processes in the analysis processing of the work status of each worker using the server 10, such as the worker to be photographed, the worker's boss or supervisor, a person in charge of each work site (such as a factory) (e.g., a factory manager), and a manager who manages each work site (e.g., a person in charge at the headquarters).

[0014] The server 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, etc., and these units are interconnected via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), or an AI chip (AI semiconductor). The control unit 11 executes the information processing and control processing to be performed by the server 10 by appropriately executing a program 12P stored in the storage unit 12. Note that if the control unit 11 has multiple processes, the control unit 11 may execute each process using a different processor.

[0015] The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), etc. The storage unit 12 stores a program 12P (program product) executed by the control unit 11 and various data. The storage unit 12 also temporarily stores data generated when the control unit 11 executes the program 12P. The storage unit 12 also stores a learning model M that has been trained through machine learning. The learning model M is expected to be used as a program module constituting artificial intelligence software. The learning model M performs a predetermined calculation on input values ​​and outputs the calculation result. The storage unit 12 stores information that defines the learning model M, such as information on the layers included in the learning model M, information on the nodes that constitute each layer, and weights (coupling coefficients) between nodes. Note that a learning model M is prepared for each task performed by a worker. The storage unit 12 also stores a video DB 12a, a to-do list DB 12b, and a video folder 12c. At least one of the video DB 12a, the ToDo list DB 12b, and the video folder 12c may be stored in another storage device connected to the server 10, or in another storage device with which the server 10 can communicate.

[0016] The communication unit 13 is a communication module for performing processes related to wired or wireless communication, and transmits and receives information to other devices via the network N. The input unit 14 accepts operation input by the user and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 11. A part of the input unit 14 and the display unit 15 may be a touch panel configured as an integrated unit.

[0017] The reading unit 16 reads information stored in a portable storage medium 10a, which may include a CD (Compact Disc), a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, etc. The program 12P and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 10a via the reading unit 16 and stored in the storage unit 12. The program 12P and various data may be written to the storage unit 12 during the manufacturing stage of the server 10, or may be downloaded by the control unit 11 from another device via the communication unit 13 and stored in the storage unit 12.

[0018] The server 10 is not limited to a single computer, but may be a multi-computer including multiple computers, or may be a virtual machine virtually constructed by software within a single device. When the server 10 is configured as a server computer, the server 10 may be a local server or a cloud server connected via the Internet or the like. The following description assumes that the server 10 is a single computer. The program 12P may be deployed on a single computer or at a single site, or may be distributed across multiple sites and executed on multiple computers interconnected via a network. Furthermore, the input unit 14 and the display unit 15 are not essential to the server 10. The server 10 may be configured to accept operations via a connected computer (e.g., a user terminal 20) or to output information to be displayed to an external display device (e.g., a user terminal 20).

[0019] The user terminal 20 includes a control unit 21, a memory unit 22, a communication unit 23, an input unit 24, a display unit 25, a reading unit 26, etc., and these units are interconnected via a bus. The control unit 21, the memory unit 22, the communication unit 23, the input unit 24, the display unit 25, and the reading unit 26 of the user terminal 20 have the same configurations as the control unit 11, the memory unit 12, the communication unit 13, the input unit 14, the display unit 15, and the reading unit 16 of the server 10, and therefore a description thereof will be omitted. Note that if the user terminal 20 is a terminal of a user that uploads videos of a worker (work space) captured with a camera to the server 10, the memory unit 22 stores a video file 22a acquired from the camera in addition to the program 22P (program product) executed by the control unit 21.

[0020] FIG. 3 is an explanatory diagram showing an example of the configuration of learning model M. FIG. 3A is an explanatory diagram of the learning process of learning model M, and FIG. 3B is an explanatory diagram of the estimation process using learning model M. Learning model M is configured using algorithms such as recurrent neural networks (RNNs), long short-term memory (LSTMs), convolution neural networks (CNNs), and transformers, and may also be configured by combining multiple algorithms. Learning model M is a model that has been pre-trained using videos of humans performing a variety of actions (videos including human movements), and is generated by loading one training cycling video. In the learning process, learning model M extracts features from the input training cycling video. For example, learning model M detects moving objects in the training cycling video and extracts features from the captured area of ​​the moving objects. Then, when the learning model M, which has already loaded the training cycle video, receives the video to be estimated during the estimation process, it divides the loaded training cycle video into multiple segmented videos based on image features, extracts cycle videos from the video to be estimated that have a high similarity score to the training cycle video based on the division results, and divides (segments) each extracted cycle video into multiple segmented videos (the same number as the number of segmented videos in the training cycle video) based on the segmented videos in the training cycle video. Note that the number of segmented videos into which the cycle video is divided may be preset in the learning model M, for example, or may be configurable to any number. Thus, the learning model M receives the video to be estimated as input, performs calculations to extract cycle videos from the input video, and divides each extracted cycle video into segmented videos, and outputs the results of the calculations. Specifically, the learning model M has an input layer into which the video to be estimated is input, an intermediate layer that extracts features from the input video and performs various calculations, and an output layer that outputs the start and end times of each cycle video extracted from the video based on the calculation results of the intermediate layer, as well as the start and end times of each segment video in each cycle video.The start and end times of the cycle video and the start and end times of the segmented videos are represented by the elapsed time from the start of playback of the video. By using this learning model M, when the server 10 inputs a video to be estimated into the learning model M, it can extract cycle videos from the video to be estimated and divide each cycle video into segmented videos based on the output information from the learning model M (the start and end times of the cycle video and the start and end times of the segmented videos). Furthermore, in the estimation process, when dividing the cycle videos extracted from the video to be estimated into segmented videos, the learning model M divides the segmented videos into segmented videos with high similarity scores to each segmented video in the training cycle video, and also outputs the similarity score for each segmented video with each segmented video in the training cycle video.

[0021] Learning of the learning model M (loading of training cycle videos) may be performed by another learning device. The trained learning model M generated by learning performed by another learning device is downloaded from the learning device to the server 10, for example, via the network N or a portable storage medium 10a, and stored in the storage unit 12. The learning model M is not limited to a configuration in which a video of a worker is input. For example, the server 10 may be configured to estimate the worker's skeleton based on each frame (each image) included in the video, generate skeletal data, and input the time-series skeletal data to the learning model M. In this case, the control unit 11 of the server 10 estimates the worker's skeleton in each frame and extracts the worker's joint positions using a technology for extracting the joint positions of a person in an image, such as OpenPose. The control unit 11 may acquire time-series skeletal data by performing a process to extract joint positions for each frame included in the video, and use the obtained time-series skeletal data as input to the learning model M.

[0022] FIG. 4 is an explanatory diagram showing an example of the record layout of the video DB 12a and the ToDo list DB 12b. FIG. 4A shows the video DB 12a, which stores information about videos uploaded to the server 10. The video DB 12a shown in FIG. 4A includes a video URL (Uniform Resource Locator) column, a shooting location column, a shooting target column, a registrant column, a cycle detection column, a comparison analysis column, and the like, and stores information about videos uploaded to the server 10 in association with the video URL. The video URL stores information for reading video data stored in the video folder 12c. Note that instead of the video URL, a folder name and file name indicating the storage location of the video data may be stored. The shooting location column stores information about the video shooting location, such as information indicating the location of the factory or the location of the workspace within the factory. The shooting target column stores information about the worker who was the subject of the video, such as the worker's affiliation, name, employee number, and the like. Note that if a video includes multiple cycle videos capturing multiple workers, information about the multiple workers is stored. The registrant column stores information about the user (registrant) who uploaded the video to the server 10, such as identification information assigned to the registrant (such as an employee number or user ID). The cycle detection column stores information about the process of extracting cycle videos from the target video using the learning model M, such as the date and time the process was performed, information about the executor, and the URL of each extracted cycle video. The executor is the user who instructed the server 10 to perform the cycle video extraction process. The cycle videos extracted from the target video may be stored in the video folder 12c. The comparison analysis column stores information about the process of displaying and analyzing multiple cycle videos side by side, such as the date and time the process was performed, information about the executor, and the URL of the cycle video used for comparison. The executor here is the user who instructed the server 10 to perform the process of comparing multiple cycle videos. The comparison target stores information about the cycle video displayed as the comparison target, such as the URL of the cycle video. The stored contents of the video DB 12a are not limited to the example shown in FIG. 4A; for example, the date and time the video was shot or registered may also be stored.

[0023] FIG. 4B shows the ToDo list DB 12b, which stores information about cycle videos and section videos with comments added among videos uploaded to the server 10, as well as information about cycle videos and section videos registered in the ToDo list. The ToDo list DB 12b shown in FIG. 4B includes a video URL column, a comment column, an improvement plan column, a status column, an improved video URL column, an improvement content column, and so on, and stores information about cycle videos and section videos registered in the ToDo list (ToDo data) in association with video URLs. The video URL stores information for reading video data for cycle videos or section videos stored in the video folder 12c. Again, instead of a video URL, a folder name and file name indicating the storage location of the video data may be stored. The comment column stores information about comments entered for cycle videos and section videos, including information about the comment inputter and the content of the comment. The improvement plan column stores information about the improvement plan entered for the cycle video or section video to which a comment has been added, including information about the person who entered the information, the content of the improvement plan, the person responsible for implementing the improvement plan, and the deadline. The status column stores information indicating the implementation status of the improvement plan, storing "Completed" if the implementation has been completed and "Not yet" if it has not been completed. The improved video URL column stores the URL of the cycle video or section video filmed after the implementation of the improvement plan has been completed. The improvement content column stores information indicating the improvement status resulting from the implementation of the improvement plan, including the time required before improvement, the time required after improvement, and the improvement rate. The time required before improvement is the time required (playback time) for the task or action in the cycle video or section video whose URL is registered in the video URL column. The time required after improvement is the time required (playback time) for the task or action in the cycle video or section video whose URL is registered in the improved video URL column. The improvement rate is a ratio calculated, for example, by dividing the time required after improvement by the time required before improvement. The contents stored in the ToDo list DB 12b are not limited to the example shown in FIG. 4B, and the items in the ToDo list DB 12b can be changed for each work site (for example, factory, work space) or each company, for example.For example, the configuration may be such that not only the improvement plan but also messages from the worker's superior or supervisor, information to be shared with other users, information for promoting improvements in work efficiency, etc. Also, the input date and time of each of the comments, improvement plan, status, improved video URL, and improvement content may be stored.

[0024] The following describes the process of uploading a video captured by a camera to the server 10 and the cycle detection process of extracting a cycle video from the video uploaded to the server 10 in the information processing system of this embodiment. Fig. 5 is a flowchart showing an example of the upload process procedure and the cycle detection process procedure, and Fig. 6 is an explanatory diagram showing an example screen.

[0025] FIG. 5A shows the upload process. In the process of FIG. 5A, the user of the user terminal 20 is the worker himself / herself, the worker's supervisor, or another person in charge of uploading videos to the server 10. A user who wants to upload videos copies the videos stored in the camera to the storage unit 22 of the user terminal 20. Alternatively, the user connects a camera to the user terminal 20 so that the videos stored in the camera can be acquired by the user terminal 20. The user performs a predetermined operation via the input unit 24 of the user terminal 20, and the control unit 21 of the user terminal 20 performs processing in accordance with the operation, thereby displaying an upload screen on the display unit 25 and uploading the video to the server 10 via the upload screen. For example, the control unit 21 accesses the server 10 and acquires the upload screen from the server 10. Furthermore, if an app (application program) for using the server 10 is stored in the storage unit 22, the control unit 21 may display the upload screen by activating the app.

[0026] The control unit 21 of the user terminal 20 displays an upload screen as shown in Fig. 6A on the display unit 25 (S11). The upload screen displays a list of file names of videos (video files 22a) stored in the storage unit 22 and videos that can be acquired from a camera connected to the user terminal 20, and has check boxes for selecting each video and an upload button for instructing uploading of the selected video.

[0027] The control unit 21 receives a video selection from the user on the upload screen via the input unit 24 (S12), and determines whether the upload button has been operated (S13). If it determines that the upload button has not been operated (S13: NO), the control unit 21 continues to receive the video selection. If it determines that the upload button has been operated (S13: YES), the control unit 21 reads out the video file 22a of the selected video from the storage unit 22 or the camera and transmits (uploads) it to the server 10 (S14). When uploading a video to the server 10, the control unit 21 receives, for example, via the input unit 24, information about the shooting location (e.g., a factory, a work space within the factory), information about the worker being filmed, information about the uploading user (registered user), and the like, and transmits these to the server 10 together with the video file.

[0028] The control unit 11 of the server 10 acquires the video file uploaded from the user terminal 20 and stores it in the storage unit 12 (S15). For example, the control unit 11 stores the acquired video file in a video folder 12c of the storage unit 12 and generates a URL for reading out this video file. The control unit 11 then associates the generated URL with the information on the shooting location and the worker who was the subject of the video acquired from the user terminal 20, and the information on the registered person, and stores them in the video DB 12a. This allows the server 10 to collect videos shot in each factory.

[0029] FIG. 5B illustrates the cycle detection process. In the process of FIG. 5B, the user of user terminal 20 is the worker himself / herself, the worker's supervisor, or another person in charge of performing the cycle detection process, which extracts cycle videos from uploaded videos. A user who wishes to perform the cycle detection process performs a predetermined operation via input unit 24 of user terminal 20, and control unit 21 of user terminal 20 performs processing in accordance with the operation, thereby displaying a cycle detection screen on display unit 25 and requesting server 10 to perform the cycle detection process. Control unit 21 of user terminal 20 then displays a cycle detection screen as shown in FIG. 6B on display unit 25 (S21). The cycle detection screen displays a list of file names of videos (video files) uploaded to server 10, and includes check boxes for selecting each video and a cycle detection button for instructing execution of cycle detection process for the selected video. Note that for videos for which cycle detection process has already been performed, "Cycle Detected" may be displayed after the file name.

[0030] The control unit 21 receives a video selection from the user via the input unit 24 on the cycle detection screen (S22), and determines whether the cycle detection button has been operated (S23). If it determines that the cycle detection button has not been operated (S23: NO), the control unit 21 continues to receive video selections, and if it determines that the cycle detection button has been operated (S23: YES), it instructs the server 10 to execute cycle detection processing on the selected video (S24). The control unit 21 transmits information indicating the selected video (for example, file name, URL, etc.) to the server 10 to request that the cycle detection processing be executed.

[0031] The control unit 11 of the server 10 reads the video for which the cycle detection process has been requested from the video folder 12c and executes the cycle detection process (S25). Here, the control unit 11 inputs the selected video as a video to be estimated into the learning model M, and based on the output information from the learning model M, extracts cycle videos from the input video and divides each cycle video into segmented videos. The control unit 11 stores each extracted cycle video in the video folder 12c, generates a URL for reading each cycle video (video file), and stores the execution date and time of the cycle detection process, information about the user (executor) who requested the execution of the cycle detection process, and the URL of each cycle video in association with each other in the cycle detection column of the video DB 12a. This allows the server 10 to register information about the cycle video extracted from each video.

[0032] The control unit 11 then generates a detection result screen displaying the results of the cycle detection processing as shown in FIG. 6C (S26) and transmits the generated screen to the user terminal 20 (S27). The screen shown in FIG. 6C has a video display field R1 displaying the video to be processed, an information display field R2 displaying information about cycle videos extracted from this video, and a time display field R3 displaying the start time and end time of each cycle video relative to the playback time of the video to be processed. The information display field R2 displays "Video 11" as the name (e.g., file name) of the video to be processed, and displays Video 1, Video 2, ... as the names of the cycle videos extracted from this video. The time display field R3 displays rectangles indicating the start time and end information of each cycle video relative to an indicator showing the playback time of the video to be processed.

[0033] The control unit 21 of the user terminal 20 receives the detection result screen sent by the server 10 and displays it on the display unit 25 (S28). As a result, the screen shown in Fig. 6C is displayed on the user terminal 20, and the user can check each cycle video included in the video for which the cycle detection process was requested.

[0034] Next, a process for comparing and presenting multiple cycle videos of the same work in the information processing system of this embodiment will be described. Fig. 7 is a flowchart showing an example of the cycle video comparison process procedure, and Fig. 8 is an explanatory diagram showing an example screen. The user of user terminal 20 in the process of Fig. 7 is a person in charge of checking and analyzing the work status using the cycle videos, such as the worker himself or the worker's supervisor.

[0035] In this embodiment, multiple cycle videos can be compared from the detection result screen shown in FIG. 6C, for example. Therefore, when the detection result screen is displayed (S31), the control unit 21 executes the processes from step S32 onward. On the screen of FIG. 6C, the information display field R2 includes check boxes for selecting each cycle video extracted from the video to be processed and a compare button for instructing the execution of a comparison process for the selected multiple cycle videos. The control unit 21 accepts a cycle video selection from the user via the input unit 24 on the detection result screen (S32) and determines whether the compare button has been operated (S33). If the control unit 21 determines that the compare button has not been operated (S33: NO), it continues accepting the selection of the cycle videos. If the control unit 21 determines that the compare button has been operated (S33: YES), it instructs the server 10 to compare the selected cycle videos (S34). Note that the control unit 21 accepts the selection of multiple cycle videos and transmits information indicating the selected cycle videos (e.g., file names, URLs, etc.) to the server 10 to request the execution of a comparison process for the cycle videos. On the screen of FIG. 6C, the cycle videos Video 1 to Video 3 are selected.

[0036] The control unit 11 of the server 10 reads the cycle video for which the comparison process was requested from the video folder 12c (S35) and generates a comparison screen for comparing the cycle videos to be compared (S36). For example, the control unit 11 generates a comparison screen as shown in FIG. 8A and transmits the generated comparison screen to the user terminal 20 (S37). The comparison screen also has a video display field R1, an information display field R2, and a time display field R3. The video display field R1 of the screen in FIG. 8A displays the cycle videos Video 1 and Video 2 out of the selected cycle videos Video 1 to Video 3. Note that the cycle video displayed in the video display field R1 can be changed to any of the cycle videos selected in step S32. Specifically, the video display field R1 has buttons B1 and B2 for switching between the displayed cycle videos, and the buttons B1 and B2 have pull-down menus for selecting one of the available cycle videos. Additionally, the cycle video displayed to the right of the video display field R1 has a switch button B3 that allows you to switch between cycle videos in the order displayed in the information display field R2 of FIG. 6C, for example. This configuration allows any two cycle videos to be displayed in the video display field R1, allowing the two cycle videos to be compared. The video display field R1 may also be configured to display all selected cycle videos.

[0037] The information display field R2 on the screen in FIG. 8A displays a stacked graph D that shows the duration of each section video (each movement) (the duration of the movement from the start to the end of the movement recorded in the section video) for each selected cycle video. The stacked graph D displays a bar graph that accumulates the duration of each section video (section 1, section 2, etc.) for each cycle video, starting from the bottom. Note that the stacked graph D displays, from left to right, a bar graph for the cycle video displayed on the left side of the video display field R1, a bar graph for the cycle video displayed on the right side, and bar graphs for other cycle videos. Furthermore, in the stacked graph D for each cycle video, the duration of the same movement (section) may be displayed in the same color (display style), which makes it easy to grasp the length of time required for each movement in each cycle video. Each section in the stacked graph D may be configured to display the duration of the corresponding section of the cycle video when a predetermined operation (e.g., selection with a cursor) is performed, and may also be configured to start playback of a segmented video of the corresponding cycle video in the video display field R1 when a predetermined operation (e.g., left-clicking with a mouse) is performed. In the example of Figure 8A, section 6 of video 2 is selected, and the duration of the selected section 6 is displayed.

[0038] In the screen of FIG. 8A, the time display field R3 shows the playback time of the cycle video on the horizontal axis, and indicates the start and end times of each segment video (each section) of the two cycle videos displayed in the video display field R1. In the example of FIG. 8A, the start times of the two cycle videos are aligned, and the start and end times of each segment video are displayed. Marks C1 are added to indicate the playback positions of the two cycle videos in the video display field R1. The comparison screen of this embodiment is configured to display (play) the multiple cycle videos displayed in the video display field R1 in synchronization (linked) with each segment video (each section) into which each cycle video is divided. For example, in the example of FIG. 8A, the duration of section 1 of video 1 is longer than the duration of section 1 of video 2. Therefore, when two cycle videos are played, after video 2 finishes playing section 1, it waits until section 1 of video 1 finishes playing. Once section 1 of video 1 finishes playing, section 2 of the two cycle videos begins playing. The time display field R3 in Figure 8B shows the state at the time when playback of section 1 of the two cycle videos has ended, and mark C1 indicates the end point of section 1 of each cycle video, after which playback of section 2 of each cycle video begins.

[0039] Furthermore, when a specific operation (e.g., left-clicking a mouse) is performed, the start and end times of each section video (section) displayed in the time display field R3 are displayed by matching the start time of the selected section video (section). In the example of FIG. 8C, section 4 is selected, and the start and end times of each section in the two cycle videos are displayed by matching the start time of section 4. This allows for easy comparison of the duration of the selected sections. At this time, the section videos of the selected movements (sections) in the two cycle videos may be displayed synchronously in the video display field R1, or may be looped (played repeatedly). This allows for comparison of the worker's movements in the two cycle videos for each section. Furthermore, when an instruction to switch the comparison target is given using the switch buttons B1 to B3 while two cycle videos are displayed in the video display field R1, the comparison screen is configured to start playback of the two cycle videos from the same section (movement) as the currently displayed section (movement).

[0040] On the comparison screen configured as described above, when the cycle video to be displayed is changed using the switching buttons B1 to B3 in the video display field R1, the display contents of the information display field R2 and the time display field R3 are also updated based on the changed cycle video. Furthermore, when a command is issued to play a section of each cycle video shown in the stacked graph D displayed in the information display field R2 or a section of each cycle video displayed in the time display field R3, the section video for that section is displayed in the video display field R1, and the display contents of the information display field R2 and the time display field R3 are also updated based on the display contents of the video display field R1.

[0041] The control unit 21 of the user terminal 20 receives the comparison screen sent by the server 10 and displays it on the display unit 25 (S38). As a result, the screen shown in FIG. 8A is displayed on the user terminal 20, allowing the user to compare the segment videos (each movement) of the multiple selected cycle videos. The information display field R2 of the comparison screen is configured to display a "Find Similar Cycle Videos" button, as shown in FIG. 8A, when a predetermined operation (e.g., selection with the cursor) is performed on one of the cycle videos displaying the stacked graph D. When the user wants to check other cycle videos (similar cycle videos) to compare with the cycle videos displaying the stacked graph D in the information display field R2, the user selects (specifies) the cycle video and operates the "Find Similar Cycle Videos" button. When the "Find Similar Cycle Videos" button is operated, the server 10 searches for cycle videos to compare with the selected cycle video, and the search results are displayed on the user terminal 20.

[0042] Furthermore, the time display field R3 of the comparison screen is configured to display a "Find similar movements" button as shown in FIG. 8D when a predetermined operation (e.g., selection with the cursor) is performed on any section of the displayed cycle video. When the user wishes to check a section of another cycle video (similar segment video) to compare with a section of the cycle video displayed in the time display field R3, the user selects (specifies) that section and operates the "Find similar movements" button. When the "Find similar movements" button is operated, the server 10 searches for sections of cycle videos to be compared with the selected section, and the search results are displayed on the user terminal 20.

[0043] Therefore, control unit 21 determines whether or not an arbitrary cycle video has been selected in information display field R2 of the comparison screen and the "Find similar cycles" button has been operated (S39), and if it determines that this has not been done (S39: NO), it determines whether or not an arbitrary section of an arbitrary cycle video has been selected in time display field R3 and the "Find similar movements" button has been operated (S45). Note that if control unit 21 determines that the "Find similar movements" button has not been operated (S45: NO), it returns to step S39.

[0044] If the control unit 21 determines that the "Find Similar Cycles" button has been operated (S39: YES), it instructs the server 10 to search for cycle videos similar to the selected cycle video (S40). For example, the control unit 21 transmits information indicating the selected cycle video (e.g., file name, URL, etc.) to the server 10, requesting execution of a search process for similar cycle videos. Based on the cycle video for which the search for similar cycle videos was requested (hereinafter, sometimes referred to as the reference video), the control unit 11 of the server 10 searches for other cycle videos with a high similarity score (degree of similarity) to the reference video (S41). For example, the control unit 11 searches for cycle videos with a similar cycle time to the reference video from among other cycle videos of the same work as the reference video. A cycle video with a similar cycle time can be one whose difference in cycle time is less than a predetermined time. Furthermore, when extracting each cycle video in the cycle detection process, the control unit 11 calculates a similarity score for each segment video in each cycle video with respect to each segment video of the training cycle video. Therefore, the control unit 11 may search for other cycle videos whose similarity scores are similar to those calculated for each section video of the reference video. Furthermore, the control unit 11 may calculate the similarity between the reference video and other cycle videos and search for cycle videos with high similarity. Similarity may be measured using, for example, a correlation coefficient or cosine similarity. Alternatively, the similarity between two cycle videos may be estimated using a learning model constructed by machine learning. For example, a learning model constructed by CNN may be used, which is trained to output the similarity between two cycle videos when two cycle videos are input.

[0045] The control unit 11 also searches for cycle videos with high attention levels from among other cycle videos of the same work as the reference video (S42). As will be described later, various information such as comments and improvement plans can be added to each cycle video and each section video of each cycle video. The control unit 11 determines whether or not other cycle videos have comments, whether or not an improvement plan has been added, and whether or not the improvement plan (improvement process) has been completed, and calculates the attention level based on the determination results. Work with a large number of cycle videos with comments and other information added is considered to be work with high attention levels. Therefore, cycle videos with calculated attention levels equal to or greater than a predetermined value are searched for as cycle videos with high attention levels. Specifically, the control unit 11 calculates the attention levels for other cycle videos from the determination results based on the contents stored in the ToDo list DB 12b.

[0046] The control unit 11 generates a comparison screen as shown in FIG. 8A based on the other cycle videos with high similarity scores found in step S41 and the other cycle videos with high attention levels found in step S42, and transmits the generated comparison screen to the user terminal 20 (S43). Here, the control unit 11 extracts a predetermined number of cycle videos in descending order of similarity score, extracts a predetermined number of cycle videos in descending order of attention levels, and generates a comparison screen displaying the extracted cycle videos. Note that on this comparison screen, the reference video is displayed on the left side of the video display field R1, and one of the other cycle videos found (hereinafter referred to as the comparison video) is displayed on the right side. The information display field R2 displays, from left to right, a stacked graph D of the reference video, comparison video, and other cycle videos. The time display field R3 displays indicators indicating the start and end times of each segment video (section) of the reference video and comparison video. The other cycle videos displayed on the comparison screen may be not only cycle videos with high similarity scores and cycle videos with high attention levels, but also cycle videos with high similarity scores that have attracted attention. When the comparison screen is sent to the user terminal 20, the control unit 11 associates the current date and time, information about the user (executor) who requested the cycle video comparison, and information about other cycle videos presented on the comparison screen with the video URL (cycle video URL) of the reference video on the comparison screen, and stores them in the comparison analysis column of the video DB 12a. This allows the server 10 to collect information about each cycle video presented as a comparison target for the cycle video selected by the user.

[0047] The control unit 21 of the user terminal 20 receives the comparison screen sent by the server 10 and displays it on the display unit 25 (S44). This allows the user to check information about other cycle videos to be compared with the reference video for which a search for similar cycle videos has been requested. After processing step S44, the control unit 11 proceeds to step S45. If it determines that the "Search for Similar Actions" button has been operated (S45: YES), it instructs the server 10 to search for section videos (actions) similar to the selected section video (S46). For example, the control unit 21 sends information indicating the selected section video (e.g., file name, URL, etc.) to the server 10 to request execution of a search process for similar actions. Based on the section video for which a search for similar actions (section videos) has been requested (hereinafter sometimes referred to as the reference section video), the control unit 11 of the server 10 searches for section videos of other cycle videos with a high similarity score (degree of similarity) to the reference section video (S47). For example, the control unit 11 searches for section videos with the same actions as the reference section video among other cycle videos and with similar action times to the reference section video. Section videos with similar movement times can be those in which the difference in movement time is less than a predetermined time. Furthermore, during the cycle detection process, the control unit 11 calculates a similarity score for each section video in each cycle video with respect to each section video of the training cycle video. Therefore, the control unit 11 may search for section videos of other cycle videos for which a similarity score similar to the similarity score calculated for the reference section video has been calculated. Furthermore, the control unit 11 may calculate the similarity between the reference section video and the section videos of other cycle videos, and search for section videos with high similarity. Here, similarity can be measured using, for example, a correlation coefficient or cosine similarity, or the similarity between two section videos can be estimated using a learning model constructed by machine learning.

[0048] The control unit 11 also searches for segment videos with high attention levels from segment videos of other cycle videos that perform the same movements as the reference segment video (S48). Here, the control unit 11 also calculates the attention level for the segment videos of other cycle videos based on the results of determining whether comments have been added, whether an improvement plan has been added, and whether the implementation of the improvement plan has been completed. Since movements with a large number of segment videos with added information such as comments are considered to be movements with high attention levels, segment videos with calculated attention levels equal to or greater than a predetermined value are searched for as high-attention segment videos. The control unit 11 generates a comparison screen based on the segment videos of other cycle videos with high similarity scores searched for in step S47 and the segment videos of other cycle videos with high attention levels searched for in step S48, and transmits the generated comparison screen to the user terminal 20 (S49). Here, the control unit 11 extracts a predetermined number of segment videos in descending order of similarity score, extracts a predetermined number of segment videos in descending order of attention levels, and generates a comparison screen displaying cycle videos including the extracted segment videos. Here too, control unit 11 associates the current date and time, information about the user (executor) who requested the comparison of the segment videos, and information about other cycle videos presented on the comparison screen with the video URL (cycle video URL) of the cycle video including the reference segment video on the comparison screen, and stores this information in the comparison analysis column of video DB 12a. This allows server 10 to collect information about each segment video presented as a comparison target for the segment video selected by the user.

[0049] On the comparison screen here, a cycle video including the reference segment video is displayed on the left side of video display field R1, and one of the cycle videos including the other segment videos found through the search (comparison video) is displayed on the right side. Furthermore, information display field R2 displays, from left to right, a stacked graph D of the cycle video including the reference segment video, the comparison video, and the other cycle videos, and time display field R3 displays indicators showing the start and end times of each segment video (section) of the cycle video including the reference segment video and the comparison video. The other cycle videos displayed on the comparison screen may include segment videos with high similarity scores and segment videos with high attention, as well as segment videos with high similarity scores that include segment videos with high attention.

[0050] The control unit 21 of the user terminal 20 receives the comparison screen sent by the server 10 and displays it on the display unit 25 (S50). This allows the user to check information about other cycle videos that have the same movements as the reference section video for which a search for similar movements has been requested and that should be compared. The above-described process makes it possible to extract, for a given cycle video, other cycle videos with high similarity scores and other cycle videos with high attention ratings, and to present these section videos in a synchronized manner. Furthermore, for a given cycle video, it is possible to extract cycle videos that include section videos with high similarity scores and section videos with high attention ratings, and to present these section videos in a synchronized manner. This allows the user to compare the tasks and movements of each worker using multiple cycle videos and multiple section videos.

[0051] Next, a process for adding a comment to an arbitrary cycle video or section video and a process for registering the comment in a ToDo list in the information processing system of this embodiment will be described. Fig. 9 is a flowchart showing an example of the comment adding process, and Figs. 10 to 13 are explanatory diagrams showing example screens. The user of user terminal 20 in the process of Fig. 9 is the worker himself or the worker's supervisor, or another person in charge who analyzes the state of work using the cycle video and registers comments, etc.

[0052] In this embodiment, for example, a comment can be added (appended) to a displayed cycle video from the detection result screen shown in FIG. 6C , and a comment can be added to a segment video of a displayed cycle video from the comparison screen shown in FIG. 8A . Therefore, when the control unit 21 is displaying a screen such as the detection result screen or the comparison screen (S61), it executes the processes from step S62 onward. As shown in FIG. 10A , the detection result screen is configured to display an “Add comment” button when a predetermined operation (e.g., selection with the cursor) is performed on any of the cycle videos displayed in the time display field R3. When a user wants to add a comment to a cycle video displayed in the time display field R3 of the detection result screen, the user selects the cycle video and operates the “Add comment” button. Furthermore, as shown in FIG. 10B , the comparison screen is configured to display an “Add comment” button when a predetermined operation (e.g., selection with the cursor) is performed on any section (segment video) of a cycle video displayed in the time display field R3 of the comparison screen. When a user wants to add a comment to a segment video of a cycle video displayed in the time display field R3 of the comparison screen, the user selects the segment video and operates the “Add comment” button.

[0053] Therefore, control unit 21 determines whether any cycle video has been selected on the detection result screen and the "Add comment" button has been operated, or whether any division video has been selected on the comparison screen and the "Add comment" button has been operated (S62). If control unit 21 determines that the "Add comment" button has not been operated (S62: NO), it waits until it has been operated, and if it determines that it has been operated (S62: YES), it displays a comment input screen such as that shown in Fig. 10C on display unit 25 (S63). The screen in Fig. 10C is the screen displayed when an instruction to add a comment to the cycle video of video 2 is given via the screen in Fig. 10A, and has a comment input field, a save button, and a cancel button.

[0054] The control unit 21 accepts the input of a comment on the comment input screen (S64), displays the input comment in the input field, and determines whether the save button has been operated (S65). If it determines that the save button has not been operated (S65: NO), the control unit 21 continues to accept the input of a comment. If it determines that the save button has been operated (S65: YES), the control unit 21 instructs the server 10 to save the input comment (S66). The control unit 21 transmits information indicating the cycle video or section video selected as the target for adding a comment (for example, file name, URL, etc.), as well as information about the comment and the person who input the comment, to the server 10, requesting that the comment be saved.

[0055] When control unit 11 of server 10 acquires the URL of a cycle video or section video from user terminal 20, it stores the acquired URL (video URL), information about the commenter, and the comment in ToDo list DB 12b in association with each other (S67). When control unit 11 acquires identification information such as the file name of the cycle video or section video, it identifies the URL of the cycle video or section video from the acquired identification information and stores the identified URL in ToDo list DB 12b. Control unit 11 generates a comment list screen that displays a list of comments stored in ToDo list DB 12b and transmits it to user terminal 20 (S68).

[0056] The control unit 21 of the user terminal 20 acquires the comment list screen sent by the server 10 and displays it on the display unit 25 (S69). Fig. 11A shows an example of the comment list screen. The screen of Fig. 11A displays comments stored in the ToDo list DB 12b and information about the cycle videos and section videos to which each comment is added. An "Add to ToDo List" button is provided in association with each comment (each cycle video and section video) for instructing the user to add that cycle video or section video to the ToDo list. The control unit 21 determines whether the "Add to ToDo List" button corresponding to any comment on the comment list screen has been operated (S70). If it determines that the button has not been operated (S70: NO), the control unit 21 waits until it is operated, and if it determines that the button has been operated (S70: YES), the control unit 21 requests the server 10 to display the ToDo list screen (S71).

[0057] When a ToDo list screen is requested, the control unit 11 of the server 10 generates a ToDo list screen as shown in FIG. 12A based on the contents stored in the ToDo list DB 12b and transmits it to the user terminal 20 (S72). The control unit 21 of the user terminal 20 acquires the ToDo list screen transmitted by the server 10 and displays it on the display unit 25 (S73). The screen of FIG. 12A displays the comments and information of the person who inputted the comments input via the screens of FIGS. 10A to 10C, as well as the video URL and video data as information of the selected cycle video or section video. Furthermore, when ToDo data for each video is stored in the ToDo list DB 12b, the ToDo list screen displays, as ToDo data, comments regarding the improvement plan, a status indicating the implementation status of the improvement plan, the URL of a video taken after the improvement plan was implemented (improved video URL), the content improved by implementing the improvement plan, and the like. The comments regarding the improvement plan may include information about the person who input the improvement plan, information about the person implementing the improvement plan, and the deadline for the improvement plan. The ToDo list screen allows the addition of items other than comments and improvement plans. For example, when a specified operation (such as right-clicking the mouse) is performed on the displayed item name (comment, video URL, etc.), an input screen is displayed to instruct the addition of an item name, and an item with any name can be added.

[0058] The ToDo list screen shown in FIG. 12A allows editing of the information of each displayed item, and is configured so that editing becomes possible when a predetermined operation (e.g., double-clicking the mouse) is performed on the display field of any item. In FIG. 12A, a predetermined operation is performed on the display field of the improvement plan for the comment "Put used tools in their designated locations," making this display field editable (the cursor is displayed). Also, in FIG. 12A, editing of the status item is requested, and in this case, the status editing screen is displayed as shown in FIG. 12B. Therefore, when the status is edited via the status editing screen and the OK button is operated, the status information in the ToDo list screen is changed.

[0059] The control unit 21 accepts edits to the information (ToDo data) of each item being displayed on the ToDo list screen (S74), displays the edited information in the input field, and determines whether the update button has been operated (S75). If it determines that the update button has not been operated (S75: NO), the control unit 21 continues to accept edits to the ToDo data. If it determines that the update button has been operated (S75: YES), the control unit 21 instructs the server 10 to save each piece of edited information (edited ToDo data) (S76). The control unit 21 transmits each piece of information displayed on the ToDo list screen to the server 10, requesting that the ToDo data be saved.

[0060] The control unit 11 of the server 10 stores the ToDo data acquired from the user terminal 20 in the ToDo list DB 12b, associating it with the URL of the cycle video or section video (S77). The above-described process allows comments to be associated with any cycle video or section video and stored. In addition to comments, other information (data) related to the cycle video or section video, such as information related to improvement plans, information to be kept, and information to be shared with other users, can also be associated and stored. When information related to the cycle video or section video (such as comments) is stored in the ToDo list DB 12b through the above-described process, the information associated with the cycle video or section video can be displayed on a comparison screen, as shown in FIG. 11B. In the example of FIG. 11B, section 4 of the cycle video in video 2 is selected via stacked graph D displayed in information display field R2, and comments added to this section (section video) are displayed. This allows users to check the comments and other information added to each cycle video and each section video when comparing and analyzing multiple videos, and can be referenced when considering improvement strategies.

[0061] On the screen of Fig. 11B, a "Check ToDo List" button is displayed in the display field for comments added to the video, and when the "Check ToDo List" button is operated, the control unit 21 requests a ToDo list screen from the server 10, and can acquire and display a ToDo list screen such as that shown in Fig. 12A from the server 10. This allows various information (ToDo data) added to the cycle video or segment video to be checked from the comparison screen.

[0062] Furthermore, the ToDo list screen allows for the translation of information written in natural language for each item. When a predetermined operation (e.g., right-clicking the mouse) is performed on any item, as shown in FIG. 13A, a translation menu is displayed, along with a button for selecting the language to translate into. When the language selection button is operated, the information for the selected item is translated into the selected language. When a translation into English is requested for the comment item in FIG. 13A, the control unit 21 adds and displays an English translation of the comment, as shown in FIG. 13B. With this configuration, in a company with factories (work spaces) in multiple countries, staff in other countries can understand information such as comments registered at each factory, and this information can be shared among staff in multiple countries.

[0063] As described above, this embodiment can perform multiple types of processes, such as uploading videos to the server 10, dividing the uploaded videos into cycle videos, detecting cycles, comparing and analyzing the divided cycle videos or segment videos, adding comments to the cycle videos or segment videos, and registering the implementation status of an improvement plan for the work or motion captured in the cycle videos or segment videos in the ToDo list DB 12b. Furthermore, by storing each piece of information in the video DB 12a and the ToDo list DB 12b, the implementation status of each process is linked to the cycle video or segment video. Factories and companies that adopt the information processing system of this embodiment often desire to monitor the system's usage status, the implementation status of improvement plans within each factory, or for each worker. Therefore, the information processing system of this embodiment has a function for displaying the implementation status (number of executions) of each of the above-mentioned processes for each factory (department, workspace) or for each person in charge.

[0064] Next, a process for monitoring the usage status of the system and the implementation status of an improvement plan in the information processing system of this embodiment will be described. Fig. 14 is a flowchart showing an example of the monitoring process procedure, and Fig. 15 is an explanatory diagram showing an example screen. The user of user terminal 20 in the process of Fig. 14 is a person in charge of monitoring the usage status of the system and the implementation status of an improvement plan in each factory or for each worker, such as the worker's supervisor, the factory manager, or a person in charge at headquarters who supervises each factory.

[0065] When execution of the monitoring function is instructed by a predetermined operation via the input unit 24, the control unit 21 of the user terminal 20 displays a monitor screen such as that shown in Fig. 15A (S81). The screen of Fig. 15A has input fields for the period to be monitored, the affiliation (e.g., factory) and the person in charge (e.g., worker), and each input field is provided with a pull-down menu from which the user can select the period, affiliation, and person in charge. The period to be monitored can be selected in units of one week, one month, one year, etc., the affiliation can be selected as one or more affiliations or all, and the person in charge can be selected as one or more affiliations or all. The screen of Figure 15A also has check boxes for selecting the number of videos uploaded to server 10 (number of uploaded videos), the number of videos that have undergone cycle detection processing (number of cycle detection videos), the number of cycle videos or section videos that have been comparatively analyzed (number of comparative analysis videos), the number of cycle videos or section videos registered in ToDo list DB12b (number of videos in ToDo list), the number of cycle videos or section videos for which implementation of an improvement plan has been completed (number of videos with improvement completed), and the number of videos whose improved videos have been registered in ToDo list DB12b (number of registered videos after improvement).

[0066] The control unit 21 receives input of the monitoring target via the input unit 24 on the monitor screen (S82), determines whether the OK button has been operated (S83), and continues receiving input of the monitoring target. If it determines that the OK button has not been operated (S83: NO), the control unit 21 transmits the input monitoring target to the server 10 (S84) and requests monitoring of the transmitted monitoring target. The control unit 11 of the server 10 counts the monitoring results for the monitoring target requested from the user terminal 20 based on the contents stored in the video DB 12a and the ToDo list DB 12b (S85). For example, if the monitoring target includes the number of uploaded videos, the control unit 11 counts the number of videos corresponding to the period, department, and person in charge of the monitoring target based on the contents stored in the video DB 12a. If the monitoring target includes the number of cycle detection videos, the control unit 11 counts the number of cycle videos corresponding to the period, department, and person in charge of the monitoring target based on the contents stored in the cycle detection column of the video DB 12a.

[0067] The control unit 11 transmits the counted monitoring results to the user terminal 20 (S86), and the control unit 21 of the user terminal 20 displays the monitoring results acquired from the server 10 on the monitor screen (S87). As a result, a monitor screen such as that shown in FIG. 15B is displayed on the display unit 25 of the user terminal 20. On the screen of FIG. 15B, the number of uploaded videos, the number of cycle detection videos, the number of comparison and analysis videos, the number of videos in the ToDo list, and the number of videos for which improvements have been completed are selected as monitoring targets, and the monitoring results are displayed, showing the number of videos in each bar graph. This allows the usage status of the information processing system of this embodiment and the implementation status of improvement plans for videos with improvement plans registered in the ToDo list to be monitored for each factory (work space) or for each person in charge. This makes it possible to promote the use of the information processing system of this embodiment and the implementation of improvement plans.

[0068] In this embodiment, a video captured by a camera is uploaded to the server 10 and various processes are performed by the server 10, but the present invention is not limited to this configuration. For example, the processes performed by the server 10 and the user terminal 20 may be performed by a single device. Even in such a configuration, the above-described processes of this embodiment can be performed, and similar effects can be obtained.

[0069] (Embodiment 2) In this embodiment, an information processing system is described in which each section (section video) of a cycle video displayed in the time display field R3 on the comparison screen is highlighted when there are (or many) other cycle videos to view. With this configuration, other cycle videos to view can be easily selected when performing comparative analysis with other cycle videos on the comparison screen. The information processing system of this embodiment can be realized using devices similar to the devices 10 and 20 of the information processing system of embodiment 1 shown in Figure 2, so a description of the configuration of the devices 10 and 20 will be omitted.

[0070] Fig. 16 is a flowchart showing an example of the comparison processing procedure of the second embodiment, and Fig. 17 is an explanatory diagram showing an example screen. The processing shown in Fig. 16 is the same as the processing shown in Fig. 7 except that steps S91 to S94 are added between steps S36 and S37, and steps S95 to S97 are added instead of steps S39 to S48. Explanations of the same steps as in Fig. 7 will be omitted.

[0071] In the information processing system of this embodiment, after processing step S36, the control unit 11 of the server 10 calculates a similarity score (degree of similarity) for each segment video of a cycle video displayed in the time display field R3 of the comparison screen with segment videos of the same movement in other cycle videos of the same task (S91). Then, based on the calculated similarity scores, the control unit 11 searches for segment videos of other cycle videos that have a high similarity score with each segment video currently being displayed (S92). For example, the control unit 11 searches for segment videos with similar movement times, segment videos with a similar similarity score to the segment video of the training cycle video calculated in the cycle detection process, or segment videos with a high similarity score with the segment video of the currently being displayed cycle video.

[0072] The control unit 11 also searches for segment videos with high attention ratings among other segment videos of cycle videos performing the same tasks as the currently displayed cycle video (S93). Here, the control unit 11 calculates the attention ratings of the segment videos of other cycle videos based on the results of determining whether comments have been added, whether an improvement plan has been added, and whether the implementation of the improvement plan has been completed, and searches for segment videos with high attention ratings. The control unit 11 then identifies segment videos to be highlighted for each currently displayed segment video based on the number of segment videos of other cycle videos with high similarity scores and the number of segment videos with high attention ratings, and highlights the identified segment videos (S94). For example, the control unit 11 identifies a segment video as one to be highlighted if the number of segment videos with high similarity scores, the number of segment videos with high attention ratings, or the total of both segment videos is equal to or greater than a predetermined number. In the screen of FIG. 17A, the segment videos currently displayed in the time display field R3 are highlighted by hatching. Note that highlighting may also be achieved by adding a predetermined mark. Thereafter, the control unit 11 transmits a comparison screen in which some of the segmented videos are highlighted to the user terminal 20 (S37), and the control unit 21 of the user terminal 20 displays the comparison screen on the display unit 25 (S38).

[0073] As shown in FIG. 17A, the comparison screen of this embodiment is configured to display a "Watch" button along with a message saying "Faster videos are available" when a predetermined operation (e.g., selection with the cursor) is performed on a section (segment video) highlighted in the time display field R3. Therefore, the control unit 21 determines whether the "Watch" button has been operated on the comparison screen (S95). If it determines that the button has been operated (S95: YES), it requests other videos to be compared from the server 10 (S96). If other videos are requested, the control unit 11 of the server 10 reads from the video folder 12c cycle videos including the segment videos with high similarity scores found in step S92 and cycle videos including the segment videos with high attention ratings found in step S93 (S97). The control unit 11 then generates a comparison screen displaying the read cycle videos and transmits it to the user terminal 20 (S49). The control unit 21 of the user terminal 20 obtains comparison images from the server 10 and displays the comparison images as shown in FIG. 17B (S50). On the comparison screen here, the cycle video of video 2 is displayed as the reference video, and the cycle video of video 11 is displayed as the comparison video. Note that the cycle videos of videos 11 to 14 are cycle videos that include the section videos searched for in steps S92 and S93.

[0074] According to the above-described process, in this embodiment, for each segment video of a cycle video displayed in the time display field R3 of the comparison screen, if there are other videos to be compared and analyzed, they are highlighted. This allows the user to easily select a video to be compared and analyzed. Furthermore, since other videos that are attracting attention for the selected video are presented, the user can efficiently consider an improvement plan by viewing information such as comments added to the other videos. The configuration of this embodiment is applicable to the information processing system of embodiment 1, and similar effects can be obtained even when applied to the information processing system of embodiment 1. Furthermore, the modified examples described in embodiment 1 above can also be applied to this embodiment.

[0075] (Embodiment 3) The first and second embodiments described above are configured to compare cycle videos of the same tasks, and to compare segmented videos included in the cycle videos. In this embodiment, an information processing system will be described that compares different tasks based on cycle videos of different tasks. The information processing system of this embodiment can be realized using devices similar to the devices 10 and 20 of the information processing system of the first embodiment shown in FIG. 2, and therefore a description of the configuration of the devices 10 and 20 will be omitted.

[0076] In the manufacturing industry, a single product is completed by performing multiple types of work in chronological order. In some cases, a line production system is employed, in which multiple workers are each responsible for a different work. With a line production system, workers' skills improve through repetition of work, making it possible to efficiently manufacture high-quality products. Furthermore, a line production system requires that each work be completed at the same time. If even one worker is slow, that work becomes a bottleneck, reducing overall productivity. Therefore, in this embodiment, by comparing different work tasks, it is possible to identify the work that is the bottleneck.

[0077] Fig. 18 is a flowchart showing an example of the comparison processing procedure between different tasks, and Fig. 19 is an explanatory diagram showing an example screen. The user of user terminal 20 in the processing of Fig. 18 is a person in charge of analyzing the status of each task performed in chronological order, such as a factory manager or a person in charge at headquarters who supervises each factory.

[0078] In the information processing system of this embodiment, when an instruction to perform an inter-task comparison is received through a predetermined operation via the input unit 24, the control unit 21 of the user terminal 20 displays an inter-task comparison screen as shown in FIG. 19A on the display unit 25 (S101). The screen of FIG. 19A has input fields for the period and affiliation (e.g., factory) to be compared, and each input field has a pull-down menu for selecting a selectable period and affiliation. The comparison period can be selected in units of one week, one month, one year, etc., and the affiliation can be one, multiple affiliations (e.g., factories) that manufacture the same product, or all affiliations. The screen of FIG. 19A also has check boxes for selecting each work process (task) performed at the selected affiliation (factory or workspace) to be compared.

[0079] The control unit 21 accepts input of a comparison target via the input unit 24 on the task comparison screen (S102), determines whether the OK button has been pressed (S103), and continues accepting input of a comparison target if it determines that the OK button has not been pressed (S103: NO). If it determines that the OK button has been pressed (S103: YES), the control unit 21 transmits the input comparison target to the server 10 (S104) and requests that a comparison process be performed on the transmitted comparison target. For the comparison process requested by the user terminal 20, the control unit 11 of the server 10 extracts cycle videos that match the comparison target (S105). For example, the control unit 11 identifies cycle videos of the category (location, work space) and task process that match the comparison target based on information on the shooting location and shooting target stored in the video DB 12a, and extracts the identified cycle videos.

[0080] The control unit 11 calculates (measures) the required time (task time, cycle time) for each task based on the extracted cycle video of each task, and calculates the minimum, first quartile, median, third quartile, and maximum values ​​(values ​​indicating the variation in task time) for each task based on the calculated task time (S106). The control unit 11 then generates a box-and-whisker plot (diagram) showing the variation in task time for each task based on the calculated values, and generates a comparison result screen (S107). For example, the control unit 11 generates a comparison result screen displaying a box-and-whisker plot as shown in FIG. 19B. In the box-and-whisker plot shown in FIG. 19B, each task corresponds to a position on the horizontal axis, the vertical axis indicates the task time for each task, and the variation in task time is expressed by the box and whiskers indicating the minimum, first quartile, median, third quartile, and maximum value of task time, respectively. As shown in FIG. 19B, in the box-and-whisker plot of each work process, a standard time previously set for each work process may be indicated by a broken line, and the average work time for each work process may be plotted by a black circle.

[0081] The control unit 11 transmits the generated comparison result screen to the user terminal 20 (S108), and the control unit 21 of the user terminal 20 displays the comparison result screen acquired from the server 10 on the display unit 25 (S109). As a result, the comparison result screen as shown in FIG. 19B is displayed on the user terminal 20. On the screen of FIG. 19B, the first to third processes are selected as the comparison targets, and a box plot showing the variation in the respective work times is displayed. This makes it possible to compare the work times for each work process required to complete one product, and to grasp the variation in each work time.

[0082] The screen of FIG. 19B is configured such that each point in the box-and-whisker plot is associated with a cycle video with a corresponding task time. Performing a specific operation (e.g., left-clicking the mouse) on any point in the box-and-whisker plot causes the selected cycle video to play, as shown in FIG. 19C. The screen of FIG. 19C has a similar configuration to the comparison screen of FIG. 8A. The selected cycle video is displayed in the video display field R1, a stacked graph D of the selected cycle video is displayed in the information display field R2, and the start and end times of each segment video (section) of the selected cycle video are displayed in the time display field R3. As with the screens of FIGS. 8A and 8D, the screen of FIG. 19C also allows users to search for other cycle videos similar to the currently displayed segment video. Therefore, the factory manager, the person in charge at headquarters who supervises each factory, etc. can check the variation in work time for each task via the screen of Figure 19B, and then, for example, for tasks that take a long (slow) time, they can compare it with other cycle videos of the same task (other workers) via screens such as Figure 19C and Figure 8A, and consider improvement measures to improve the efficiency of each task.

[0083] By the above-described processing, in this embodiment, it is possible to compare the work time (cycle time) between different tasks based on cycle videos of different tasks, and this can be used to identify bottlenecks in a series of work processes. The configuration of this embodiment can be applied to the information processing systems of embodiments 1 and 2, and similar effects can be obtained even when applied to the information processing systems of embodiments 1 and 2. Furthermore, the modified examples described in the above-mentioned embodiments 1 to 2 can also be applied to this embodiment as appropriate.

[0084] In the above-described first to third embodiments, the processing target is a video of a worker performing a task, but the processing target is not limited to a worker. For example, a configuration may be adopted in which, based on a video of a robot configured to perform a predetermined task, the task performed by the robot and the actions included in the task are compared with those of other robots or workers.

[0085] The features described in the above-mentioned embodiments can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations, regardless of the reference format. Furthermore, the claims use a format in which a claim references two or more other claims (multiple claim format), but this is not limited to this. A multiple claim (multi-multi claim) that references at least one other multiple claim may also be used.

[0086] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0087] 10 Servers 11 Control section 12 Storage section 13 Communications Department 20 User terminal 21 Control section 22 Memory section 23 Communications Department M Learning Model 12a Video DB 12b ToDo list DB

Claims

1. A video of the workspace where the subject is working is divided into actions included in the work, Based on the segmented videos obtained by segmenting a plurality of videos, including videos taken in different work spaces where the subject performs the same work or videos taken in work spaces where the arrangement of items related to the same work is different, for each action, segmented videos of each action in the plurality of videos are output in synchronization. A program that causes a computer to perform a process.

2. The video includes a video taken by a camera device at a fixed position and a video taken by a camera device attached to the subject. The program according to claim 1.

3. Data relating to the video is linked to the video, Accepting designation of one video from the plurality of videos; Extracting other videos that have a high similarity to the specified video from other videos of the same work as the specified video; The designated moving image and the extracted other moving image are output in synchronization with each other, When outputting the video, output data associated with the video.

3. The program according to claim 1, which causes the computer to execute processing.

4. Data relating to the segmented video is linked to the segmented video, Accepting designation of one section video from among the section videos of the video; extracting a segmented video of another video having the same action as the designated segmented video, the segmented video having a high similarity to the designated segmented video; The designated section video and the extracted section video of the other video are output in synchronization with each other; When outputting the divided video, data associated with the divided video is output.

3. The program according to claim 1, which causes the computer to execute processing.

5. For each segment video in the video, a similarity to a segment video of the same action in another video and an attention level to the segment video of the same action in the other video are obtained; highlighting the segmented moving images among the moving images according to the acquired similarity and attention level; Accepting the designation of one section moving image from among the highlighted section moving images; extracting a segment video from other videos having the same action as the designated segment video based on the similarity and attention level with the designated segment video; The designated section video and the extracted section video from the other video are output in synchronization with each other.

3. The program according to claim 1, which causes the computer to execute processing.

6. the data about the video is written in natural language; Translate the data related to the video into a specified language and output it. The program according to claim 3, which causes the computer to execute processing.

7. The video is associated with execution statuses of a plurality of types of processing for the video; Counting the number of times each process is executed on the video of the object or the workspace for each of the objects or workspaces; The number of times each process is executed is output for each of the objects or the workspaces.

3. The program according to claim 1, which causes the computer to execute processing.

8. From a video of multiple types of work performed in chronological order, we extract cycle videos of each work, measuring a work time required for each of the work tasks based on a plurality of cycle videos of each of the work tasks; For each task, a chart showing the variation in the measured work time for each task is output.

3. The program according to claim 1, which causes the computer to execute processing.

9. A video of the workspace where the subject is working is divided into actions included in the work, Based on the segmented videos obtained by segmenting a plurality of videos, including videos taken in different work spaces where the subject performs the same work or videos taken in work spaces where the arrangement of items related to the same work is different, for each action, segmented videos of each action in the plurality of videos are output in synchronization. An information processing method in which processing is performed by a computer.

10. In an information processing device having a control unit, The control unit A video of the workspace where the subject is working is divided into actions included in the work, Based on the segmented videos obtained by segmenting a plurality of videos, including videos taken in different work spaces where the subject performs the same work or videos taken in work spaces where the arrangement of items related to the same work is different, for each action, segmented videos of each action in the plurality of videos are output in synchronization. Information processing device.

Images