Information processing device, change method, and change program

Abstract

An information processing device (100) comprises: an acquisition unit (120) that acquires an input video which is obtained by filming a subject who is performing work, and a reference video which is obtained by filming a person who is performing correct work; a warping path generation unit (130) that generates a warping path on the basis of the input video and the reference video; a slope calculation unit (140) that calculates a plurality of local slopes of the warping path by using the input video and the reference video; a setting unit (150) that, in cases in which the average value of the degree of similarity between individual frames of the input video belonging to a section for which the slope is less than a predetermined first threshold value and individual frames of the reference video corresponding thereto in the warping path is less than a predetermined second threshold value, sets the section as a section of interest; and a change unit (160) that changes the warping path of the section of interest to a straight line.

Description

Information processing device, modification method, and modification program

[0001] This disclosure relates to an information processing device, a modification method, and a modification program.

[0002] Warping paths are known. Technologies related to warping paths have been proposed (see Patent Document 1). The sequential data analysis device of Patent Document 1 calculates a warping path using input sequential data, which is video footage of work performed in a factory, and reference sequential data. For example, the reference sequential data is video footage of a skilled worker's work.

[0003] Patent No. 6704543

[0004] The most comfortable posture for each worker varies. For example, the posture of the worker in question in Patent Document 1's input sequence data differs from that of a skilled worker. Furthermore, the physique of the worker in question in Patent Document 1's input sequence data differs from that of a skilled worker. Such individual differences may be reflected in the warping path. Incidentally, it is possible to determine whether the work time is long or not using the slope of the warping path. If individual differences are reflected in the warping path, there is a possibility that the work time may be incorrectly determined to be long.

[0005] The purpose of this disclosure is to correct individual differences reflected in warping paths.

[0006] An information processing device according to one aspect of the present disclosure is provided. The information processing device includes: an acquisition unit that acquires input video, which is video obtained by filming a person performing a task, and reference video, which is video obtained by filming a person performing the task correctly; a warping path generation unit that generates a warping path based on the input video and the reference video; a slope calculation unit that calculates a plurality of local slopes of the warping path using the input video and the reference video; a setting unit that sets an interval as a target interval if the average similarity between each frame of the input video belonging to an interval in which the slope is less than a predetermined first threshold and each frame of the reference video corresponding to the warping path is less than a predetermined second threshold; and a modification unit that changes the warping path in the target interval to a straight line.

[0007] According to this disclosure, individual differences reflected in the warping path can be corrected.

[0008] This is a diagram showing the hardware of the information processing device. This is a block diagram showing the functions of the information processing device. This is a diagram showing a specific example of a warping path. This is a diagram showing an example of calculating the slope of a warping path. This is a diagram for explaining the setting process. This is a diagram showing an example of a warping path after modification. This is a diagram showing a specific example of the judgment process. This is a flowchart showing an example of a process executed by the information processing device (part 1). This is a flowchart showing an example of a process executed by the information processing device (part 2).

[0009] The embodiments will be described below with reference to the drawings.

[0010] Embodiment. Figure 1 shows the hardware of the information processing device. The information processing device 100 is a device that executes a change method. The information processing device 100 is also called a computer. The information processing device 100 has a processor 101, a volatile storage device 102, and a non-volatile storage device 103.

[0011] The processor 101 controls the entire information processing device 100. For example, the processor 101 may be a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), etc. The processor 101 may also be a multiprocessor. The information processing device 100 may also have processing circuits.

[0012] The volatile storage device 102 is the main memory of the information processing device 100. For example, the volatile storage device 102 is RAM (Random Access Memory). The non-volatile storage device 103 is the auxiliary storage device of the information processing device 100. For example, the non-volatile storage device 103 is an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0013] Next, the functions of the information processing device 100 will be explained. Figure 2 is a block diagram showing the functions of the information processing device. The information processing device 100 includes a storage unit 110, an acquisition unit 120, a warping path generation unit 130, a slope calculation unit 140, a setting unit 150, a modification unit 160, a determination unit 170, and an output unit 180.

[0014] The storage unit 110 may be implemented as a storage area reserved in a volatile storage device 102 or a non-volatile storage device 103. The storage unit 110 may also be called a memory. Some or all of the acquisition unit 120, warping path generation unit 130, slope calculation unit 140, setting unit 150, modification unit 160, determination unit 170, and output unit 180 may be implemented by processing circuits. In addition, some or all of the acquisition unit 120, warping path generation unit 130, slope calculation unit 140, setting unit 150, modification unit 160, determination unit 170, and output unit 180 may be implemented as modules of a program executed by the processor 101. For example, the program executed by the processor 101 is also called a modification program or modification program product. For example, the modification program is recorded on a recording medium.

[0015] The memory unit 110 stores various types of information.

[0016] The acquisition unit 120 acquires video footage obtained by photographing a person performing a task. For example, the acquisition unit 120 acquires the video footage from the storage unit 110. Alternatively, the acquisition unit 120 may acquire the video footage from an external device. The external device is a device located outside the information processing device 100. For example, the external device could be a cloud server or external memory. The diagram of the external device is omitted. This video footage is called the input video. For example, the subject is a worker performing a task in a factory. Alternatively, the subject is a junior worker. The acquisition unit 120 acquires video footage obtained by photographing a person performing a task correctly from the storage unit 110 or an external device. For example, the person is a skilled worker. This video footage is called the reference video.

[0017] The warping path generation unit 130 generates a warping path based on the input video and the reference video. For example, as described in Patent Document 1, the warping path generation unit 130 generates a warping path. Specifically, the warping path generation unit 130 generates a warping path using the input video, the reference video, and DTW (Dynamic Time Warping). A specific example of a warping path is shown below.

[0018] Figure 3 shows a specific example of a warping path. The vertical axis represents the reference video axis. The horizontal axis represents the input video axis. Figure 3 shows the warping path 200.

[0019] The tilt calculation unit 140 calculates multiple local tilts of the warping path using the input video and the reference video. Specifically, the tilt calculation unit 140 calculates each tilt θ in the same way as described in Patent Document 1. The calculation method will be explained in detail using the figures.

[0020] Figure 4 shows an example of calculating the slope of the warping path. The vertical direction indicates the frame number of the reference video. The horizontal direction indicates the frame number of the input video. Cell m of the warping path is one of 0 to M-1. M is the length of the warping path. p[m] is the frame number of the reference video corresponding to the m-th cell of the warping path. q[m] is the frame number of the input video corresponding to the m-th cell of the warping path. d is set to 1. d may also be an integer of 2 or greater.

[0021] If d ≤ m < M-1-d, the slope calculation unit 140 calculates the slope θ using equation (1).

[0022] θ[m] = tan －１ (q[m+d]-q[m-d], p[m+d]-p[m-d])...(1)

[0023] If m < d, the slope calculation unit 140 calculates the slope θ using equation (2).

[0024] θ[m] = tan －１ (q[m+d]-q[0], p[m+d]-p[0])...(2)

[0025] When m ≥ M - 1 - d, the slope calculation unit 140 calculates the slope θ using Equation (3).

[0026] θ[m] = tan －１ (q[M - 1] - q[m - d], p[M - 1] - p[m - d]) … (3)

[0027] The setting unit 150 identifies a section where the slope θ is less than a predetermined threshold value. This threshold value is also referred to as the first threshold value. The setting unit 150 sets a section where the average value of the similarity between each frame of the input video belonging to the section where the slope θ is less than the first threshold value and each frame of the reference video corresponding via the warping path is less than a predetermined threshold value as the target section. This threshold value is also referred to as the second threshold value. Note that, for example, the similarity is calculated using a conventional technique.

[0028] The setting unit 150 sets a section that is the section before the target section and to which a plurality of frames belong as the first consideration section. The number of the plurality of frames may be predetermined. The setting unit 150 includes the first consideration section in the target section when the average value of the similarity between each frame of the input video belonging to the first consideration section and each frame of the reference video corresponding via the warping path is less than a predetermined threshold value. Note that this threshold value is the second threshold value.

[0029] The setting unit 150 sets a section that is the section after the target section and to which a plurality of frames belong as the second consideration section. The number of the plurality of frames may be predetermined. The setting unit 150 includes the second consideration section in the target section when the average value of the similarity between each frame of the input video belonging to the second consideration section and each frame of the reference video corresponding via the warping path is less than a predetermined threshold value. Note that this threshold value is the second threshold value.

[0030] The setting process will be described with reference to the drawings. FIG. 5 is a diagram for explaining the setting process. The section where the slope θ is less than the first threshold value is the section from section t １ to t ２ The setting unit 150 is the section from section t １ to t ２If the average value of the similarity between each frame of the input video belonging to [interval] and each frame of the reference video corresponding via the warping path is less than the second threshold value, interval t １ ~t ２ is set as the target interval. For example, if the average value of the similarity between each frame of the input video belonging to interval t １ ~t ２ is less than the second threshold value, with the plurality of frames of the input video being the fifth frame and the sixth frame, when the setting unit 150 uses the average value of the similarity between the fifth frame of the input video and the fifth frame of the reference video and the similarity between the sixth frame of the input video and the sixth frame of the reference video, interval t １ ~t ２ is set as the target interval.

[0031] The setting unit 150 sets the interval t ３ ~t １ , which is the interval before the target interval, as the first consideration interval. The average value of the similarity between each frame of the input video belonging to the first consideration interval and each frame of the reference video corresponding via the warping path is less than the second threshold value. The setting unit 150 includes the first consideration interval (that is, interval t ３ ~t １ ) in the target interval.

[0032] The setting unit 150 sets the interval t ５ ~t ３ , which is the interval before the target interval, as the first consideration interval. The average value of the similarity between each frame of the input video belonging to the first consideration interval and each frame of the reference video corresponding via the warping path is greater than or equal to the second threshold value. The setting unit 150 does not include the first consideration interval (that is, interval t ５ ~t ３ ) in the target interval.

[0033] The setting unit 150 sets the interval t ２ ~t ４ , which is the interval after the target interval, as the second consideration interval. The average value of the similarity between each frame of the input video belonging to the second consideration interval and each frame of the reference video corresponding via the warping path is less than the second threshold value. The setting unit 150 includes the second consideration interval (that is, interval t ２ ~t ４ ) in the target interval.

[0034] The setting unit 150 is the section t which is the section after the target section. ４ ~t ６ Set this as the second study interval. The average similarity between each frame of the input video belonging to the second study interval and each frame of the reference video corresponding by the warping path shall be greater than or equal to the second threshold. The setting unit 150 sets the second study interval (i.e., interval t) ４ ~t ６ ) is not included in the target section. As a result, section t ３ ~t ４ However, this will be the affected section.

[0035] The modification unit 160 changes the warping path in the target section to a straight line. Specifically, the modification unit 160 changes the warping path in the target section so that the beginning of the warping path in the target section and the end of the warping path in the target section are connected by a straight line. An example of the warping path after modification is shown.

[0036] Figure 6 shows an example of the warping path after modification. The modified section 160 is the target section t ３ ~t ４ Change the warping path to a straight line.

[0037] The determination unit 170 determines whether the slope of the warping path in the target section is less than a predetermined threshold. If the slope is less than the threshold, the determination unit 170 determines that the work time of the subject shown in the input video is long. This threshold is also called the third threshold. A specific example of the determination process is shown below.

[0038] Figure 7 shows a specific example of the determination process. The determination unit 170 determines whether the slope α of the warping path in the target section is less than a third threshold. Here, Figure 7 shows a reference line. The slope of the reference line is 45 degrees. If the work time of the subject shown in the input video and the work time of the worker shown in the reference video are the same, the slope of the warping path will be 45 degrees. Therefore, for example, the third threshold is 45 degrees. By comparing the warping path and the reference line, work delays can be detected. For example, if the slope α is 45 degrees or less, the determination unit 170 determines that the work time of the subject shown in the input video is long.

[0039] If the output unit 180 determines that the subject's work time shown in the input video is long, it outputs information indicating that the subject's work time is long. For example, the output unit 180 outputs this information to the display of the information processing device 100.

[0040] Next, the processes performed by the information processing device 100 will be explained using a flowchart. Figure 8 is a flowchart of an example (part 1) of the processes performed by the information processing device. (Step S11) The acquisition unit 120 acquires the input video and the reference video from the storage unit 110. (Step S12) The warping path generation unit 130 generates a warping path based on the input video and the reference video. (Step S13) The slope calculation unit 140 calculates multiple local slopes of the warping path using the input video and the reference video.

[0041] (Step S14) The setting unit 150 identifies an interval in which the slope is less than a first threshold. (Step S15) The setting unit 150 determines whether the average similarity between each frame of the input video belonging to the interval in which the slope is less than a first threshold and each frame of the reference video corresponding to it in the warping path is less than a second threshold. If the average is less than a second threshold, the process proceeds to step S16. If the average is greater than or equal to a second threshold, the process proceeds to step S26. (Step S16) The setting unit 150 sets the interval in which the slope is less than a first threshold as the target interval.

[0042] (Step S17) The setting unit 150 sets the section preceding the target section and to which multiple frames belong as the first consideration section. (Step S18) The setting unit 150 determines whether the average similarity between each frame of the input video belonging to the first consideration section and each frame of the reference video corresponding in the warping path is less than the second threshold. If the average is less than the second threshold, the process proceeds to step S19. If the average is greater than or equal to the second threshold, the process proceeds to step S21. (Step S19) The setting unit 150 includes the first consideration section in the target section. Then, the process proceeds to step S17. Steps S17 to S19 are repeated until the average is greater than or equal to the second threshold. This expands the target section.

[0043] Figure 9 is a flowchart showing an example (part 2) of the processing performed by the information processing device. (Step S21) The setting unit 150 sets the section after the target section and to which multiple frames belong as the second examination section. (Step S22) The setting unit 150 determines whether the average similarity between each frame of the input video belonging to the second examination section and each frame of the reference video corresponding in the warping path is less than the second threshold. If the average is less than the second threshold, the process proceeds to step S23. If the average is greater than or equal to the second threshold, the process proceeds to step S24. (Step S23) The setting unit 150 includes the second examination section in the target section. Then, the process proceeds to step S21. Steps S21 to S23 are repeated until the average is greater than or equal to the second threshold. This expands the target section.

[0044] (Step S24) The modification unit 160 changes the warping path of the target section to a straight line. (Step S25) The determination unit 170 determines whether the slope of the warping path of the target section is less than a third threshold. If the slope is less than the third threshold, the process proceeds to step S26. If the slope is greater than or equal to the third threshold, the process ends. (Step S26) The determination unit 170 determines that the work time of the subject shown in the input video is long. Specifically, if step S26 is executed after step S15, the determination unit 170 determines that the work time of the subject is long in the section where the slope is less than the first threshold. If step S26 is executed after step S25, the determination unit 170 determines that the work time of the subject is long in the target section.

[0045] (Step S27) The output unit 180 outputs information indicating that the subject's work time is long. The output unit 180 may also output information indicating a section or target section in which the slope is less than the first threshold. For example, the output unit 180 outputs information indicating the start time and end time of the section. Alternatively, for example, the output unit 180 outputs information indicating the start time and end time of the target section. This allows, for example, the manager to identify sections with long work times.

[0046] According to this embodiment, the information processing device 100 sets a target section. The target section is a section in which individual differences are evident. Therefore, the information processing device 100 changes the warping path of the target section to a straight line. This allows the information processing device 100 to correct the individual differences reflected in the warping path.

[0047] The information processing device 100 does not have to perform steps S17-19 and S21-23. However, by performing steps S17-19 and S21-23, the information processing device 100 can determine whether or not there are sections before and after the target section set in step S16 in which individual differences are present. In other words, by performing steps S17-19 and S21-23, the information processing device 100 can improve the accuracy of detecting sections in which individual differences are present. Alternatively, the information processing device 100 may perform only steps S17-19 or only steps S21-23.

[0048] By executing step S25, the information processing device 100 can detect whether the work time of the subject shown in the input video is long or not.

[0049] 100 Information processing device, 101 Processor, 102 Volatile memory device, 103 Non-volatile memory device, 110 Storage unit, 120 Acquisition unit, 130 Warping path generation unit, 140 Slope calculation unit, 150 Setting unit, 160 Modification unit, 170 Determination unit, 180 Output unit, 200 Warping path.

Claims

1. An information processing device comprising: an acquisition unit that acquires input video, which is video obtained by filming a person performing a task, and reference video, which is video obtained by filming a person performing the task correctly; a warping path generation unit that generates a warping path based on the input video and the reference video; a slope calculation unit that calculates a plurality of local slopes of the warping path using the input video and the reference video; a setting unit that sets an interval as a target interval if the average similarity between each frame of the input video belonging to an interval in which the slope is less than a predetermined first threshold and each frame of the reference video corresponding to the warping path is less than a predetermined second threshold; and a modification unit that changes the warping path in the target interval to a straight line.

2. The setting unit sets a section preceding the target section and to which a plurality of frames belong as a first study section, and if the average value of the similarity between each frame of the input video belonging to the first study section and each frame of the reference video corresponding in the warping path is less than the second threshold, the information processing device according to claim 1.

3. The setting unit sets a second examination interval which is an interval after the target interval and to which a plurality of frames belong, and if the average value of the similarity between each frame of the input video belonging to the second examination interval and each frame of the reference video corresponding in the warping path is less than the second threshold, the information processing apparatus according to claim 1 or 2.

4. An information processing device according to any one of claims 1 to 3, further comprising: a determination unit that determines that the worker's working time is long if the slope of the warping path in the target section is less than a predetermined third threshold; and an output unit that outputs information indicating that the worker's working time is long, or information indicating the target section, when it is determined that the worker's working time is long.

5. The information processing apparatus according to claim 1, further comprising: a determination unit that determines that the subject's working time is long if the average value is greater than or equal to the second threshold; and an output unit that outputs information indicating that the subject's working time is long, or information indicating an interval in which the slope is less than the first threshold, when it is determined that the subject's working time is long.

6. A modification method comprising: an information processing device acquiring input video, which is video obtained by filming a person performing a task, and reference video, which is video obtained by filming a person performing the task correctly; generating a warping path based on the input video and the reference video; calculating a plurality of local slopes of the warping path using the input video and the reference video; setting the interval as a target interval if the average similarity between each frame of the input video belonging to an interval in which the slope is less than a predetermined first threshold and each frame of the reference video corresponding to the warping path is less than a predetermined second threshold; and changing the warping path in the target interval to a straight line.

7. A modification program that causes an information processing device to execute the following processes: acquire input video, which is video obtained by filming a person performing a task, and reference video, which is video obtained by filming a person performing a task correctly; generate a warping path based on the input video and the reference video; calculate a plurality of local slopes of the warping path using the input video and the reference video; set the interval as a target interval if the average similarity between each frame of the input video belonging to an interval in which the slope is less than a predetermined first threshold and each frame of the reference video corresponding to the warping path is less than a predetermined second threshold; and change the warping path in the target interval to a straight line.

Images