Information processing system, information processing method, and program

Abstract

To accurately detect each work even when there are a plurality of work processes SOLUTION: An information processing apparatus comprising an acquisition unit configured to acquire an image captured by an image capturing apparatus, a detection unit configured to detect a first work using a first detection model and detect a second work using a second detection model with respect to the image acquired by the acquisition unit, and a control unit configured to perform control to switch a detection model used for work detection from the first detection model to the second detection model in response to detection of the first work by the detection unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] The work performed by a worker is analyzed using video to recognize what kind of work the worker is performing.

[0003] Patent Document 1 discloses a technology for recognizing standard tasks that have been predetermined as targets for monitoring from footage of a worker, with the aim of recognizing with high accuracy a series of actions that a worker takes while working. Specifically, it discloses a technology for acquiring multiple frame images contained in the video and determining the worker's work actions from characteristic changes in each frame image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-87312 DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0005] Patent Document 1 does not disclose a mechanism for accurately detecting each operation when there are multiple operation processes.

[0006] Therefore, the present invention aims to provide a mechanism for accurately detecting each operation even when there are multiple operation processes. [Means for solving the problem]

[0007] an acquisition means for acquiring an image captured by an imaging device; a detection means for detecting a first operation using a first detection model and detecting a second operation using a second detection model in the image acquired by the acquisition means; a control means for controlling the detection model used for detection to be switched from the first detection model to the second detection model in response to the detection of the first operation by the detection means; An information processing device comprising: [Effects of the Invention]

[0008] According to the present invention, even when there are multiple work processes, each work can be detected with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] A diagram showing an example of a system configuration [Figure 2] A diagram showing an example of a hardware configuration [Figure 3] Flowchart showing processing details [Figure 4] A diagram explaining the feature extraction process [Figure 5] Diagram explaining work hours and specified times [Figure 6] Diagram explaining the work process and detection model [Figure 7] A diagram showing an example of a work confirmation screen DETAILED DESCRIPTION OF THE INVENTION

[0010] FIG. 1 is a diagram showing an example of a system configuration according to the present invention.

[0011] As shown in FIG. 1, an information processing device 100 and an imaging device 101 are connected so as to be able to communicate with each other.

[0012] The information processing device 100 is a device that executes the processing shown in the flowchart of FIG. The imaging device 101 has a function for acquiring and analyzing video data captured by the imaging device 101 and a function for performing various processes according to the present invention.

[0013] The imaging device 101 is installed in a factory or the like and is a device for capturing images of workers working in the factory. The captured video and images are transmitted to the information processing device 100.

[0014] 1 shows only two devices, the information processing device 100 and the imaging device 101, but a configuration may be adopted in which data captured by the imaging device 101 and models for detecting various operations are stored in an external server device. In that case, the server device is connected to the information processing device 100 and the imaging device 101 so as to be able to communicate with each other.

[0015] In addition, although the present embodiment will be described with reference to work performed by workers in a factory, the scope of application is not limited to this, and the invention can be applied to any work that involves physical movement, such as nursing care work, cooking, sports movements, medical procedures such as surgery, etc. Furthermore, the invention is not limited to human movements and can also be applied to robot movements.

[0016] FIG. 2 is a block diagram showing an example of the hardware configuration of an information processing device that can be used as the server device 101 or the client terminal 102 of the present invention.

[0017] As shown in FIG. 2, the information processing device is connected to a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a storage device 204, an input controller 205, an audio controller 206, a video controller 207, a memory controller 208, and a communication I / F controller 209 via a system bus 200.

[0018] The CPU 201 controls all devices and controllers connected to the system bus 200 .

[0019] ROM202 or external memory 213 stores the BIOS (Basic Input / Output System) and OS (Operating System), which are control programs executed by CPU201, computer-readable and executable programs for realizing this information processing method, and various necessary data (including data tables).

[0020] The RAM 203 functions as a main memory, a work area, etc. for the CPU 201. The CPU 201 loads programs and the like required for executing processing from the ROM 202 or the external memory 213 into the RAM 203, and executes the loaded programs to realize various operations.

[0021] The input controller 205 controls input from input devices such as a keyboard 210 and a pointing device such as a mouse (not shown). If the input device is a touch panel, the user can issue various instructions by pressing (touching with a finger or the like) icons, cursors, or buttons displayed on the touch panel.

[0022] The touch panel may also be a touch panel capable of detecting positions touched by multiple fingers, such as a multi-touch screen.

[0023] The video controller 207 controls the display on an external output device such as a display 212. The display also includes the display of a notebook computer integrated with the main body. Note that the external output device is not limited to a display, and may be, for example, a projector. In addition, for devices capable of receiving the above-mentioned touch operation, an input device is also provided.

[0024] The video controller 207 can control a video memory (VRAM) for display control, and can use part of the RAM 203 as a video memory area, or can provide a separate dedicated video memory.

[0025] The memory controller 208 controls access to the external memory 213. The external memory may be an external storage device (hard disk) that stores a boot program, various applications, font data, user files, edited files, and various data, a flexible disk (FD), or a CompactFlash (registered trademark) memory connected to a PCMCIA card slot via an adapter.

[0026] The communication I / F controller 209 connects and communicates with external devices via a network, and executes communication control processing on the network. For example, communication using TCP / IP, telephone lines such as ISDN, and 4G and 5G mobile phone lines are possible.

[0027] The CPU 201 enables display on the display 212 by, for example, executing a process of expanding (rasterizing) an outline font into a display information area in the RAM 203. The CPU 201 also enables user instructions using a mouse cursor (not shown) on the display 212.

[0028] Next, the processing of the present invention executed by the information processing device 100 will be described with reference to the flowchart of Fig. 3. The processing shown in the flowchart of Fig. 3 is processing in which the CPU 201 of the information processing device 100 reads and executes a predetermined control program.

[0029] In step S101, frames of a moving image captured by the imaging device 101 are acquired and added to a queue.

[0030] In step S102, it is determined whether the number of frames acquired in step S101 has reached the upper limit of the queue. If the upper limit has been reached (step S102: YES), the process proceeds to step S103. If the upper limit has not been reached (step S102: NO), the process returns to step S101 to acquire frames again.

[0031] The upper limit of the queue may be specified as a number of frames, such as "30 frames," or as a time period of video data, such as the number of frames equivalent to "1 second of video."

[0032] In step S103, features are extracted from the data stored in the queue.

[0033] In this way, by storing a certain amount of frame data in a queue and extracting features from the stored data in step S103, it becomes possible to extract features as moving images rather than still images, making it possible to detect and identify work content, etc. with high accuracy.

[0034] In step S104, it is determined whether the currently active detection model (the model used for work detection) is a model for detecting work that has started. If it is a model for detecting work that has started (S104: YES), the process proceeds to step S105. If it is not a model for detecting work that has started (S104: NO), the process proceeds to step S109.

[0035] In step S105, the feature amount extracted in step S103 is analyzed to determine whether the start work has started or is continuing.

[0036] The feature extraction process will be described with reference to FIG.

[0037] 4 shows an example in which one second of video data is accumulated in steps S101 and S102, and features are extracted from the one second of video data. RGB features and optical flow features are extracted from the video data, and tasks are detected based on the extracted features. The process of detecting tasks from the features uses a trained model that has been trained using video of workers performing tasks related to the process to be detected as training data.

[0038] The RGB feature is a feature based on the RGB (Red, Green, Blue) values ​​of each pixel, and the optical flow feature is a feature based on changes in the image between frames (movement of an object). Note that although an example using the RGB feature and the optical flow feature has been described in this embodiment, other feature may also be used.

[0039] If the start or continuation of the start task is detected (S105: YES), the process proceeds to step S106. If not detected (S105: NO), the process returns to step S101, and the next video to be analyzed is acquired.

[0040] If the start of the work is not detected for a long time, an alert may be output in the same manner as in step S115 described later. The output of the alert will be described in detail in step S115.

[0041] In step S106, it is determined whether the starting work has continued for a predetermined time.

[0042] If it is determined that the predetermined time has elapsed (S106: YES), the process proceeds to step S107. If the duration of the started task does not reach the predetermined time (S106: NO), the process returns to step S101, and the next video to be analyzed is acquired.

[0043] The predetermined time used in this step is a time set based on past performance, specifically, the time obtained by subtracting the standard deviation from the average time required for past started tasks. By determining that a task has started when it has continued for the predetermined time, it is possible to prevent false positives, such as when a momentary action similar to the started task is performed. Furthermore, by using the time obtained by subtracting the standard deviation from the average required time, it is possible to switch the detection model to a model that detects other tasks at the appropriate time (preventing delays in the model switching), thereby preventing missed detections of the start of other tasks.

[0044] Furthermore, even if there is a very short period of time (which is to be set in advance) during which the starting work cannot be detected before the "predetermined time" has elapsed since the start of the starting work, it will be assumed that a detection error occurred during that period, or that the worker performed a different action for just an instant, and the starting work will be considered to have continued.

[0045] In this embodiment, the "predetermined time" is the time obtained by subtracting the standard deviation from the average required time. However, it is also possible to use, for example, "the time obtained by subtracting 2 times the standard deviation from the average required time," or the time obtained by multiplying the average required time by a preset coefficient (e.g., 0.8 times), or the time obtained by subtracting a preset time from the average required time.

[0046] In step S107, the fact that the starting work has started is recorded.

[0047] In step S108, the active detection model is switched from the start operation detection model to the operation detection model.

[0048] In step S109, it is determined whether the start of work in the next process has been detected. For example, if the current process (the most recently detected process) is the start of work, it is determined whether work in the second process has started. Similarly, if the current process is work related to the second process, it is determined whether work in the third process has started.

[0049] If the start of the next process is detected (S109: YES), the process proceeds to step S110. If the start of the next process is not detected (S109: NO), the process proceeds to step S115.

[0050] In step S110, the time taken for the work of the previous process (the work related to the process previous to the process whose start of work was detected in S109) (the time from when the start of work is detected in step S105 or S109 to when the start of work of the next process is detected in S109) is recorded.

[0051] Figure 5 shows an example of a data table in which task times are recorded. Figure 5 shows an example in which task times for starting tasks are recorded, with the number of tasks and task time recorded. For example, it shows that the first task took 5 seconds to start, and the second task took 3 seconds to start. The average time from the first to Nth tasks is 5.75 seconds, with a standard deviation of 0.89 seconds.

[0052] Work time can be recorded for each worker by correlating it with information (ID) that identifies the worker, and the average time and standard deviation can be calculated for each worker. It is also possible to record the start and end times of work, allowing for different average times and standard deviations to be calculated for day shifts and night shifts. By calculating the average work time and standard deviation according to the worker and work environment in this way, it becomes possible to output alerts at more appropriate times and change the detection model.

[0053] When the work time is recorded, the average work time is updated taking the newly recorded work time into account. The standard deviation is also updated. This updating process is also performed in the process of step S113.

[0054] In step S111, it is determined whether the current process (the process for which the start of work was detected in S109) is the final process. If it is the final process (S111: YES), the process proceeds to step S112. If it is not the final process (S111: NO), the process of this flowchart ends.

[0055] In step S112, it is determined whether the end of the work related to the final process has been detected or whether the elapsed time since the start of the final process (the elapsed time since the start was detected in S109) has exceeded a predetermined time. If the end of the work has been detected or the predetermined time has exceeded (S112: YES), the process proceeds to step S113. If the end of the work has not been detected and the predetermined time has not exceeded (S112: NO), the process of this flowchart ends.

[0056] The "predetermined time" used here is a time set based on past performance, specifically the average time required for the last process in the past plus the standard deviation. This makes it possible to switch to a mode that detects the start of a task even when the task is taking longer than past performance (i.e., when the last process task would normally have been completed but the completion of the task could not be detected for some reason (such as the worker not completing the last process task or a detection error)).

[0057] In this embodiment, the "predetermined time" is the average required time plus the standard deviation, but it is also possible to use, for example, "the average required time plus 2 times the standard deviation," or the average required time multiplied by a preset coefficient (1.2 times, etc.), or the average required time plus a preset time.

[0058] In step S113, the time required for the final process (specifically, the time from when the start of the process is detected in S109 until the determination in S112 is YES) is recorded.

[0059] In step S114, the active detection model is switched to a model that detects the start of work, and the process of this flowchart ends.

[0060] Next, a case where the determination in step S109 is NO (when the start of work in the next process is not detected) will be described.

[0061] In step S115, it is determined whether a state in which the work of the next process has not started has continued for a predetermined time (whether a predetermined time has passed since the start of the work of the previous process was detected).

[0062] If it has continued for the predetermined time (S115: YES), the process proceeds to step S115. If it has not continued for the predetermined time (S115: NO), the process of this flowchart ends.

[0063] The "predetermined time" here is set based on past performance, as in S112, and specifically, is the average time required for the work in the previous process plus the standard deviation.

[0064] In this embodiment, the "predetermined time" is the average required time plus the standard deviation, but it is also possible to use, for example, "the average required time plus 2 times the standard deviation," or the average required time multiplied by a preset coefficient (1.2 times, etc.), or the average required time plus a preset time.

[0065] In step S116, an alert is output based on the fact that work for the next process has not started within a predetermined time. Specific examples of the alert include displaying information on a monitor or the like installed at the work site urging the worker to start the next process, or outputting audio from a speaker urging the worker to start the next process. It is also possible to configure the system so that the manager is notified of information based on the fact that the next process has not started (this may be displayed on a monitor as described above, or output as audio). A combination of these is also possible.

[0066] When the process of this flowchart is completed, the process is executed again from step S101.

[0067] (Second Example) In the first embodiment, two types of operation detection models were used: a "start operation detection model" and a "model for detecting operations in other processes." However, in the second embodiment, a different model is used for each operation process.

[0068] As shown in FIG. 6, a detection model is associated with each work process, and whether work is being performed is detected using the detection model associated with each process.

[0069] In the second embodiment, if the determination in step S109 is YES, the model is switched to a model that detects the next process (this may be before or after the processing in S110). Also, if the determination in S114 is YES, the model is switched to a model that detects the next process (this may be before or after the processing in S115).

[0070] In this way, by sequentially switching to a model that detects the work of the next process and using a model specialized for detecting a specific work, it is possible to detect work with high accuracy.

[0071] FIG. 7 is a diagram showing an example of a task confirmation screen. This screen plays back video captured by the imaging device 101, and displays a graph 701 showing the task time for each step, an indicator 702 indicating the playback position, a list 703 of task steps, and a display area 705 for displaying the currently played video. The example in FIG. 7 shows an example of video displaying a task consisting of three steps, Tasks 1 to 3, and the screen shows the video of Task 2 being performed as indicated by the indicator 702. As indicated by 704, the task list 703 may also display a clear indication of which task is being performed. Graph 701 displays the time required for each task based on the time required for each step recorded in steps S110 and S113.

[0072] Although the embodiments have been described above, the present invention can be embodied as, for example, a system, an apparatus, a method, a program, a recording medium, etc. Specifically, the present invention may be applied to a system made up of multiple devices, or may be applied to an apparatus made up of a single device.

[0073] Furthermore, the program of the present invention is a program that enables a computer to execute the processing method of the flowchart shown in Fig. 3, and the storage medium of the present invention stores a program that enables a computer to execute the processing method of Fig. 3. Note that the program of the present invention may be a program for each processing method of each device in Fig. 3.

[0074] As described above, it goes without saying that the object of the present invention can also be achieved by supplying a recording medium on which a program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0075] In this case, the program itself read from the recording medium will realize the novel functions of the present invention, and the recording medium on which the program is recorded will constitute the present invention.

[0076] Examples of recording media for supplying the program include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, and silicon disks.

[0077] Furthermore, it goes without saying that not only are the functions of the above-mentioned embodiments realized by the computer executing a program it has read, but also cases are included in which an OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program, and the functions of the above-mentioned embodiments are realized through that processing.

[0078] Furthermore, it goes without saying that this also includes cases where a program read from a recording medium is written into a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to the computer, and then a CPU or the like provided on the function expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

[0079] Furthermore, the present invention may be applied to a system consisting of multiple devices, or to a device consisting of a single device. It goes without saying that the present invention can also be applied to a case where the present invention is achieved by supplying a program to a system or device. In this case, the system or device can enjoy the effects of the present invention by reading a recording medium containing a program for achieving the present invention into the system or device.

[0080] Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that the present invention also includes configurations that combine the above-mentioned embodiments and their modified examples. [Explanation of symbols]

[0081] 100 Information processing device 101 Imaging device

Claims

1. an acquisition means for acquiring an image captured by an imaging device; a detection means for detecting a first operation using a first detection model and detecting a second operation using a second detection model in the image acquired by the acquisition means; a control means for controlling the switching of a detection model used for detecting an operation from the first detection model to a second detection model in response to the detection of a first operation by the detection means; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the control means further controls the detection model used to detect the work to be switched from the second detection model to the first detection model when a predetermined time has elapsed since the work related to the final process was detected using the second detection model.

3. 2. The information processing apparatus according to claim 1, wherein the predetermined time is determined based on an average time required for work related to a final process.

4. 4. The information processing apparatus according to claim 3, wherein the predetermined time is calculated by adding an average required time to a standard deviation.

5. the detection means detects work using a detection model associated with each work process; the control means controls the detection model used for detecting the work to be switched to the detection model associated with the (n+1)th work in response to the detection means detecting the work related to the nth work; The information processing device according to claim 1, characterized in that, in response to the detection of work relating to the final process by the detection model associated with the final process, the detection model used to detect the work is controlled to be switched to the detection model associated with the work relating to the starting process.

6. an acquisition step in which an acquisition means of the information processing device acquires an image captured by the imaging device; a detection step in which a detection means of the information processing device detects a first task using a first detection model and detects a second task using a second detection model for the image acquired in the acquisition step; a control step of controlling the control means of the information processing device to switch the detection model used for detecting the operation from the first detection model to a second detection model in response to the first operation being detected by the detection step; An information processing method comprising:

7. A program for causing a computer to function as each of the means according to any one of claims 1 to 5.

Images