Serial data analysis device, serial data analysis method and serial data analysis program

Abstract

Serial data analysis device (10), comprising: a computation unit (21) for extracting combinations, each comprising one data element of the input series data and one data element of the reference series data, wherein the data elements correspond to each other by comparing input series data, which are time series data obtained by observing a given event, with reference series data, which are standard time series data for the event, and for calculating a relative progress rate, which is a relative progress rate of the event in the input series data compared to the progress of the event in the reference series data, for each of the extracted combinations; and a decision unit (22) for determining, on the basis of the relative progress rate calculated by the computation unit (21), whether there is an anomaly in the progress of the event in the input series data; where if the decision unit (22) determines that a first anomaly exists, i.e., that data for a section corresponding to a given first section in the reference series data are missing in the input series data, and a second anomaly exists, i.e., that data for a given second section included in the input series data are non-standard data, the decision unit (22) corrects the input series data by deleting the data for the second section from the input series data and inserting the deleted data into the input series data as the data for the section corresponding to the first section, where the non-standard data are series data that do not appear in the reference series data, wherein the computation unit (21) computes a first similarity, which is a similarity between the input series data before correction by the decision unit (22) and the reference series data, and a second similarity, which is a similarity between the input series data after correction by the decision unit (22) and the reference series data, and wherein the decision unit (22) compares the first similarity with the second similarity and determines that a third anomaly exists, i.e. ordinal positions in the event are exchanged in the input series data if the second similarity is higher than the first similarity.

Description

Technical area

[0001] The present invention relates to a series data analysis apparatus, a series data analysis method and a series data analysis program. Background to the state of the art

[0002] Patent Literature 1 describes a method of calculating, through DTW, a distance between standard time series data and time series data as an anomaly detection object and examining whether the time series data as the anomaly detection object is abnormal or normal. "DTW" stands for Dynamic Time Warping.

[0003] Patent Literature 2 describes a method of learning a normal model for waveforms based on extreme values ​​occurring in a time series waveform of sensor data and times of occurrence of the extreme values, comparing the normal model with sensor data as a diagnosis object, and determining whether the sensor data as the diagnosis object is abnormal or normal.

[0004] Patent Literature 3 describes a method in which a reference point value is obtained from reference point data and a characteristic coefficient extracted from a time series of the reference point data, a target value is obtained from target data and a characteristic coefficient extracted from a time series of the target data, the reference point value is compared with the target value, and it is determined whether the target data is abnormal or normal.

[0005] Document US 2017 / 0 103 672 A1 discloses a device for virtual training guidance, by means of which reference movements are compared with recorded movements and deviations between the movement sequences are determined. Reference listPatent documents Patent Literature 1: JP 2010-271997 A Patent Literature 2: JP 2017-033470 A Patent Literature 3: JP 2016-038856 A Summary of the inventionTechnical problem

[0006] A factory production facility needs to detect abnormal worker actions for quality control, such as "using a dropped part," "forgetting to tighten a screw," and "working in a different order." However, under these circumstances, abnormal actions are detected by visual inspection of surveillance video, and the workload for a monitoring officer is high. Therefore, a method for automatically detecting abnormal work scenes from work videos using video analysis is needed.

[0007] A given pattern involving a series of work actions appears repeatedly in a work video in a factory. Therefore, an abnormal action of a worker can be automatically detected by comparing a reference work video with a monitoring video and detecting a difference. If the analysis of a work video is interpreted as the analysis of series data, the detection objects referred to as scenes such as "using a dropped part," "forgetting to tighten a screw," and "working in a different order" can be interpreted as "insertion of non-standard series data," "missing series data," and "replacing parts of series data," respectively. This leads to the need for a series data analysis method capable of detecting such anomalies in series data on a per-frame basis.However, a conventional method is unable to detect an anomaly in serial data on a per-image basis.

[0008] The present invention has as its object to detect an anomaly in serial data on a per data item basis. Solution to the problem

[0009] A series data analysis apparatus according to one aspect of the present invention comprises a calculation unit for extracting combinations each comprising one piece of data of the input series data and one piece of data of the reference series data, the data items corresponding to each other by comparing input series data, which is time series data obtained by observing a given event, with reference series data, which is standard time series data for the event, and for calculating a relative progress speed, which is a relative progress speed of the event in the input series data to the progress of the event in the reference series data, for each of the extracted combinations; and a decision unit for determining, based on the relative progress speed calculated by the calculation unit, whether there is an anomaly in the progress of the event in the input series data; wherein if the decision unit determines that a first anomaly exists, i.e., that data for a section corresponding to a given first section in the reference series data is missing in the input series data, and a second anomaly exists, i.e., data for a given second section included in the input series data is non-standard data, the decision unit corrects the input series data by deleting from the input series data the data for the second section and inserting into the input series data the deleted data as the data for the section corresponding to the first section, where the non-standard data are series data that do not appear in the reference series data, wherein the calculation unit calculates a first similarity which is a similarity between the input series data before correction by the decision unit and the reference series data, and a second similarity which is a similarity between the input series data after correction by the decision unit and the reference series data, and wherein the decision unit compares the first similarity with the second similarity and determines that a third anomaly exists, i.e., ordinal positions in the event in the input series data are exchanged, if the second similarity is higher than the first similarity. Advantageous effects of the invention

[0010] According to the present invention, a relative progress rate is calculated for each of the combinations, ie a relative rate of progress of an event in input series data to the progress of the event in reference series data, each combination comprising one data item of the input series data and one data item of the reference series data, the data items corresponding to each other.

[0011] Whether an anomaly in the progress of the event in the input series data exists is determined based on the relative progress speed. Therefore, the present invention is capable of detecting an anomaly in input series data on a per-data-item basis. Short description of the drawings Fig. 1 is a block diagram illustrating a configuration of a series data analysis apparatus according to Embodiment 1. Fig. 2 is a flowchart illustrating an operation of the series data analysis device according to Embodiment 1. Fig. 3 is a flowchart illustrating a procedure for a curvature path calculation process according to Embodiment 1. Fig. 4 is a diagram illustrating an example of a curvature path calculation process according to Embodiment 1. Fig. 5 is a flowchart illustrating a procedure for a relative progress speed calculation process according to Embodiment 1. Fig. 6 is a diagram illustrating an example of a slope of a curved path calculated by the relative progress speed calculation process according to Embodiment 1. Fig. 7 is a diagram illustrating an example of a relative progress speed calculated by the relative progress speed calculation process according to Embodiment 1. Fig. 8 is a flowchart illustrating a procedure for a row data shortage section detection process according to Embodiment 1. Fig. 9 is a diagram illustrating an example of a row data shortage section for a row data shortage section detection process according to Embodiment 1. Fig. 10 is a flowchart illustrating a procedure for a non-standard row data insertion section detection process according to Embodiment 1. Fig. 11 is a diagram illustrating an example of a non-standard row data insertion section detected by the non-standard row data insertion section detection process according to Embodiment 1. Fig. 12 is a flowchart illustrating a flow for a series data exchange section detection process according to Embodiment 1. Fig. 13 is a diagram illustrating an example of a row data exchange section acquired by the row data exchange section acquisition process according to Embodiment 1. Fig. 14 is a view illustrating an example of anomaly information recorded by an anomaly information recording process according to Embodiment 1. Fig. 15 is a view illustrating an example of the anomaly information recorded by the anomaly information recording process according to Embodiment 1. Fig. 16 is a block diagram illustrating a configuration of a series data analysis apparatus according to a modification of Embodiment 1. Description of embodiments

[0012] An embodiment of the present invention will be explained below with reference to the drawings. In the drawings, like or corresponding parts are designated by like reference numerals. A description of like or corresponding parts will be omitted or simplified as appropriate in the description of the embodiment. It should be noted that the present invention is not limited to the embodiment described below, and various changes can be made as necessary. For example, the embodiment described below may be partially implemented. Embodiment 1.

[0013] The present embodiment will be described with reference to the Fig. 1 to 15 described. *** Description of the configuration ***

[0014] A configuration of a series data analysis apparatus 10 according to the present embodiment will be described with reference to Fig. 1 explained.

[0015] The series data analysis device 10 is a computer. The series data analysis device 10 includes a processor 11 and other hardware, such as a memory 12, a reference series data storage device 13, an input series data storage device 14, and an anomaly information storage device 15. The processor 11 is connected to the other hardware via signal lines and controls the other hardware.

[0016] The series data analysis device 10 includes, as functional elements, a calculation unit 21 and a decision unit 22. The calculation unit 21 includes a curvature path calculation unit 23 and a relative progress speed calculation unit 24. The decision unit 22 includes a data shortage detection unit 25, a non-standard data insertion detection unit 26, a data exchange inference unit 27, and an anomaly information recording unit 28. Functions of the calculation unit 21 and the decision unit 22 are implemented by software.

[0017] Processor 11 is a device that executes a serial data analysis program. The serial data analysis program is a program that implements the functions of calculation unit 21 and decision unit 22. Processor 11 is, for example, a CPU, a GPU, a DSP, or a combination of some or all of these. "CPU" stands for central processing unit. "GPU" stands for graphics processing unit. "DSP" stands for digital signal processor.

[0018] Memory 12 is a device that stores the row data analysis program and curvature path information 33, relative advance speed information 34, row data shortage section information 35, non-standard row data insertion section information 36, row data exchange information 37, and other information to be generated by the row data analysis program. Memory 12 is, for example, a RAM, a flash memory, or a combination thereof. "RAM" stands for Random Access Memory.

[0019] The reference series data storage device 13 is a device that stores reference series data 31 to be input into the series data analysis program. The reference series data storage device 13 is, for example, an HDD, a flash memory, or a combination thereof. "HDD" stands for hard disk drive.

[0020] The input series data storage device 14 is a device that stores input series data 32 to be input into the series data analysis program. The input series data storage device 14 is, for example, an HDD, a flash memory, or a combination thereof.

[0021] The anomaly information storage device 15 is a device that stores anomaly information 38 output from the series data analysis program. The anomaly information storage device 15 is, for example, an HDD, a flash memory, or a combination thereof.

[0022] The reference series data storage device 13, the input series data storage device 14, and the anomaly information storage device 15 may be provided outside the series data analysis device 10. The reference series data storage device 13, the input series data storage device 14, and the anomaly information storage device 15 may be mounted as respective independent hardware or as integrated hardware.

[0023] The serial data analysis device 10 may include, as hardware, a communication device, an input device, and a display.

[0024] The communication device includes a receiver that receives the reference series data 31, the input series data 32, and other data to be input to the series data analysis program, and a transmitter that transmits the anomaly information 38 and other information output by the series data analysis program. The communication device is, for example, a communication chip or an NIC. "NIC" stands for Network Interface Card.

[0025] The input device is a device operated by a user to input a parameter, such as a threshold, into the serial data analysis program. The input device is, for example, a mouse, a keyboard, a touch pad, or a combination of some or all of these.

[0026] The display is a device that displays the anomaly information 38 and other information output by the series data analysis program on a screen. The display is, for example, an LCD. "LCD" stands for Liquid Crystal Display.

[0027] The serial data analysis program is read from memory 12 into processor 11 and executed by processor 11. Not only the serial data analysis program, but also an operating system (OS) is stored in memory 12. "OS" stands for operating system. Processor 11 executes the serial data analysis program while executing the OS. Note that part or all of the serial data program may be embedded in the OS.

[0028] The series data analysis program and the operating system may be stored in an auxiliary storage device. The auxiliary storage device is, for example, an HDD, a flash memory, or a combination thereof. When the series data analysis program and the operating system are stored in the auxiliary storage device, the series data analysis program and the operating system are loaded into the memory 12 and executed by the processor 11. An auxiliary storage device may be used as at least one of the reference series data storage device 13, the input series data storage device 14, and the anomaly information storage device 15.

[0029] The serial data analysis device 10 may include a plurality of processors that replace the processor 11. The plurality of processors share the execution of the serial data analysis program. Each processor is, for example, a CPU, a GPU, a DSP, or a combination of some or all of these.

[0030] A signal value and a variable value used, processed, or output by the series data analysis program are stored in the memory 12, the auxiliary storage device, or a register or cache memory in the processor 11.

[0031] The series data analysis program is a program that causes a computer to execute, as a calculation process and a decision process, processes performed by the calculation unit 21 and the decision unit 22. The series data analysis program may be provided in a state of being recorded on a computer-readable medium, provided in a state of being stored on a recording medium, or provided as a program product.

[0032] The serial data analysis device 10 may consist of one computer or a plurality of computers. When the serial data analysis device 10 consists of a plurality of computers, the functions of the calculation unit 21 and the decision unit 22 may be implemented in a state where they are distributed among the computers. *** Description of the business ***

[0033] The operation of the series data analysis apparatus 10 according to the present embodiment will now be described with reference to Fig. 2 to 15. The operation of the serial data analysis device 10 corresponds to a serial data analysis method according to the present embodiment.

[0034] Fig. 13 illustrates a method for a series data analysis process to be performed by the series data analysis device 10.

[0035] The series data analysis process is broadly divided into four: a series data reading process in step S100; a curvature path calculation process in step S200; an anomaly extraction process in step S300; and an anomaly information writing process in step S400.

[0036] In step S100, the calculation unit 21 receives the input series data 32 and the reference series data 31. The input series data 32 is time series data obtained by observing a given event X. The event X can be any event. In the present embodiment, the event X is an event that can be shot and recorded as a video, and more specifically, a work in a factory. That is, input series data 32 is data of a video in the present embodiment, and more specifically, data of a video obtained by recording a work performed in the factory. The reference series data 31 is standard time series data for an event X. The reference series data 31 cannot be data that has the same form as the input series data 32.In the present embodiment, the reference series data 31 is data having the same form as the input series data 32, that is, data of a video, and more specifically, data of a video obtained by shooting a properly performed work.

[0037] More specifically, a reference row data reading process in step S110 and an input row data reading process in step S120 are performed in the row data reading process in step S100.

[0038] In the reference series data reading process in step S110, a process of writing the reference series data 31 read from the reference series data storage device 13 into the memory 12 is performed by the calculation unit 21. In the input series data reading process in step S120, a process of writing the input series data 32 read from the input series data storage device 14 into the memory 12 is performed by the curvature path calculation unit 23 of the calculation unit 21.

[0039] In step S200, the calculation unit 21 extracts combinations each including one piece of data from the input series data 32 and one piece of data from the reference series data 31, the data pieces corresponding to each other, by comparing the input series data 32 with the reference series data 31. As described above, the input series data 32 is data of a video, and each frame of the video is treated as one piece of data from the input series data 32. The reference series data 31 is also data of a video, and each frame of the video is treated as one piece of data from the reference series data 31.

[0040] More specifically, a similarity map calculation process in step S210, a score map calculation process in step S220, and a curvature path calculation process in step S230 are performed in the curvature path calculation process in step S200.

[0041] The similarity map calculation process in step S210, the score map calculation process in step S220, and the curvature path calculation process in step S230 are described below.

[0042] In step S300, the calculation unit 21 calculates a relative progress speed V for each of the combinations extracted in step S200. The relative progress speed V is a relative speed of the progress of the event X in the input series data 32 to the progress of the event X in the reference series data 31. The decision unit 22 determines whether an anomaly exists in the progress of the event X in the input series data 32 based on the relative progress speed calculated by the calculation unit 21.

[0043] Specifically, a relative progress speed calculation process in step S310, a row data shortage section detection process in step S320, a non-standard row data insertion section detection process in step S330, and a row data replacement section inference process in step S340 are performed in the anomaly extraction process in step S300.

[0044] The relative progress speed calculation process in step S310, the row data shortage section detection process in step S320, the non-standard row data insertion section detection process in step S330, and the row data replacement section inference process in step S340 will be described below.

[0045] In the abnormality information writing process in step S400, a process of writing the abnormality information 38 extracted in the abnormality extraction process in step S300 into the abnormality information storage device 15 is performed by the abnormality information recording unit 28 of the decision unit 22.

[0046] Fig. 14 illustrates a method for the curvature path calculation process in step S200.

[0047] In step S200, the curvature path calculation unit 23 of the calculation unit 21 calculates the curvature path information 33 indicating corresponding images of the two pieces of row data using DP comparison for the reference row data 31 and input row data 32 read from the reference row data storage device 13 and the input row data storage device 14.

[0048] Specifically, as described above, the similarity map calculation process in step S210, the score map calculation process in step S220, and the curvature path calculation process in step S230 are performed in the curvature path calculation process in step S200.

[0049] In step S210, the curvature path calculation unit 23 calculates a similarity SIM[i,j] between each combination of images for the input series data 32 and the reference series data 31 and writes the similarity SIM[i,j] into the memory 12. The similarity SIM[i,j] is determined by the expression below. SIM[i,j]=A[i]⋅B[j] / (‖A[i]‖‖[B[j]‖)

[0050] Where i = {0, ..., Q-1} and j = {0, ..., P-1}, Q is the length of the input series data, and P is the length of the reference series data. A and B correspond to the input series data 32 and the reference series data 31, respectively, and are represented by the following expression. A={A[0],A[1],…,A[i],…,A[Q−1]} B={B[0],B[1],…,B[j],…,B[P−1]}

[0051] Where A[i]={A[i,0],A[i,1],…,A[i,D−1]}and B[j]={B[j,0],B[j,1],…,} B[j,D-1]} are satisfied. D represents the number of data dimensions.

[0052] In step S220, the curvature path calculation unit 23 initializes an evaluation map DTW by the expression below. DTW[*,0]=SIM[*,0] DTW[0,*]=SIM[0,*]

[0053] Thereafter, the curvature path calculation unit 23 calculates the evaluation map DTW from a similarity map SIM by the expression below. DTW[k,h]=SIM[k−1,h]+max{DTW[k−1,h],DTW[k,h−1],DTW[k−1,h−1]}

[0054] Where k={1,…,Q−1}and h={1,…,P−1}are satisfied.

[0055] In step S230, the curvature path calculation unit 23 searches for a curvature path in the score map DTW. The curvature path calculation unit 23 searches for the curvature path by determining a cell for DTW[Q-1,P-1] as the origin point and repeating the movement past an adjacent cell selected by the Max function in the expression used to calculate a value of the cell.

[0056] As described above, in the present embodiment, the calculation unit 21 calculates the curvature path between the input series data 32 and the reference series data 31. Then, the calculation unit 21 extracts each element of the calculated curvature path as a combination of the data element of the input series data 32 and a data element of the reference series data 31.

[0057] An example of the curvature path is shown in Fig. 4. In the example, a first element of the curvature path is extracted as a combination of a first image of the input row data 32 and a first image of the reference row data 31. A second element of the curvature path is extracted as a combination of a second image of the input row data 32 and the second image of the reference row data 31. A third element of the curvature path is extracted as a combination of a third image of the input row data 32 and a second image of the reference row data 31. A fourth element of the curvature path is extracted as a combination of a fourth image of the input row data 32 and the second image of the reference row data 31. A fifth or subsequent element of the curvature path is extracted as a combination of any image of the input row data 32 and any image of the reference row data 31.

[0058] Fig. 15 illustrates a method for the relative progress speed calculation process in step S310.

[0059] In step S310, the relative progress speed calculation unit 24 of the calculation unit 21 receives the curvature path information 33 from the curvature path calculation unit 23, examines a relative progress speed between the reference series data 31 and the input series data 32 on an image basis, and outputs the relative progress speed as the relative progress speed information 34.

[0060] More specifically, in step S311, a process of determining a gradient θ of the curved path and in step S312, a process of determining the relative progress speed V from the gradient θ of the curved path in the relative progress speed calculation process in step S310 are performed.

[0061] In step S311, the relative progress speed calculation unit 24 determines the local gradient θ for the curvature path calculated in step S230 by the expression below. For me <M−1−d: θ[m]=tan−1(q[m+d]−q[m−d],p[m+d]−p[m−d]) For <d: θ[m]=tan−1(q[m+d]−q[0],p[m+d]−p[0]) For m≥M−1−d: θ[m]=tan−1(q[M−1]−q[m−d],p[M−1]−p[m−d])

[0062] Where m = {0, ..., M-1} is satisfied and M is the length of a curvature path. p[m] and q[m] respectively indicate an image number in the reference row data 31 and an image number in the input row data 32 corresponding to an m-th cell in the curvature path.

[0063] In step S312, the relative progress speed calculation unit 24 determines the relative progress speed V from the slope θ of the curved path using the expression below. V[m]=(θ[m]−π / 4) / [π / 4]

[0064] As described above, in the present embodiment, the calculation unit 21 calculates the relative progress speed V for each extracted combination of a data item of the input series data 32 and a data item of the reference series data 31.

[0065] An example of the gradient θ of the curvature path is shown in Fig. 6. Although d = 1 in the example, d can be set to an integer greater than 1.

[0066] An example of the relative progress rate V is shown in Fig. 7. In the example, the relative progress speed V is determined in accordance with the examples in the Fig. 4 and Fig. 6. For this reason, for the first element of the curvature path θ[0] = tan - 1 (q[1]-q[0],p[1]-p[0]) = tan -1 (1,0) and V[0] = (θ[0]-π / 4) / (π / 4) = (tan -1(1,0)-π / 4) / (π / 4) = -1.0. For the second element of the curvature path, θ[1] = tan -1 (q[2]-q[0],p[2]-p[0]) = tan -1 (1,2) and V[1] = (θ[1]-π / 4) / (π / 4) = (tan -1 (1,2)-π / 4) / (π / 4) = -0.4. For the third element of the curvature path, θ[2] = tan -1 (q[3]-q[1],p[3]-p[1]) = tan -1 (1,2) and V[2] = (θ[2]-π / 4) / (π / 4) = (tan -1 (1,2)-π / 4) / (π / 4) = -0.4. For the fourth element of the curvature path, θ[3] = tan -1 (q[4]-q[2],p[4]-p[2]) = tan -1 (1,1) and V[3] = (θ[3]-π / 4) / (π / 4) = (tan -1 (1,1)-π / 4) / (π / 4) = 0. For the fifth or subsequent element of the curvature path, V[m] is also calculated.

[0067] Fig. 8 illustrates a method for the row data shortage section detection process in step S320.

[0068] In step S320, the data shortage detection unit 25 of the decision unit 22 checks, based on the relative progress speed information 34 received from the relative progress speed calculation unit 24, whether the input row data 32 is exactly the same as the reference row data 31 and whether no row data is missing. The data shortage detection unit 25 outputs the row data shortage section information 35 if there is a section where row data is missing. That is, the data shortage detection unit 25 detects a row data shortage section obtained when the input row data 32 is compared with the reference row data 31 based on the relative progress speed V calculated in step S310.

[0069] In step S321, the data shortage detection unit 25 initializes a "start image number" and a "counter" storing a start image of a detection target and the length of the section to -1 and 0, respectively.

[0070] Thereafter, the data deficiency detection unit 25 performs the following processes in order from a start cell to an end cell using an index r pointing to a cell that is a component of the curvature path.

[0071] If a relative progress speed V[r] is 1.0, the data shortage detection unit 25 increments the count by 1 in step S322. If the start frame number is set to -1, which is an initialization code at this time, the data shortage detection unit 25 assigns a value of index r to the start frame number in step S323. If the relative progress speed V[r] is not 1.0, the data shortage detection unit 25 examines whether the count is less than or greater than a threshold value T1. If a value of the count is greater than the threshold value T1, the data shortage detection unit 25 detects a section [the start frame number, r-1] as a row data shortage section in step S324. Next, the data shortage detection unit 25 initializes the count and the start frame number in step S325. Note that the threshold value T1 is appropriately adjusted.

[0072] As described above in the present embodiment, the decision unit 22 determines, based on the relative progress speed V, whether there is an abnormality in the progress of the event X in the input series data 32 and whether consecutive data items of the reference series data 31 correspond to corresponding data items of the input series data 32 in two or more of the combinations extracted by the calculation unit 21.

[0073] The decision unit 22 determines, based on a relative speed of progress of the event X in the data for a given section R1 included in the reference series data 31, whether an anomaly exists, ie, whether data for a section R2 corresponding to the section R1 is missing in the input series data 32.

[0074] An example of a series data deficiency section is shown in Fig. 9. In the present embodiment, the relative progress speed V remains 1.0, and the section R1 whose length exceeds a length determined by the threshold value T1 is detected as a row data shortage section. This enables automatic detection of a worker's abnormal action, such as "forgetting to tighten a screw."

[0075] Fig. 10 illustrates a method for the non-standard row data insertion section detection process in step S330.

[0076] In step S330, the non-standard data insertion detection unit 26 of the decision unit 22 examines whether row data that does not appear in the reference row data 31 is inserted into the input row data 32 based on the relative progress speed information 34 received from the relative progress speed calculation unit 24. The non-standard data insertion detection unit 26 outputs the non-standard row data insertion section information 36 when there is a row data section that does not appear for the reference row data 31. That is, the non-standard data insertion detection unit 26 detects a non-standard row data insertion section obtained when the input row data 32 is compared with the reference row data 31 based on the relative progress speed V determined in step S310.

[0077] In step S331, the non-standard data insertion acquisition unit 26 initializes the “start image number” and the “counter” storing a start image of the acquisition target section and the length of the section to -1 and 0, respectively.

[0078] Thereafter, the non-standard data insertion detection unit 26 performs the following processes in order from a start cell to an end cell using the index r pointing to a cell that is a component of the curvature path.

[0079] If the relative progress speed V[r] is -1.0, the non-standard data insertion detection unit 26 increments the count by 1 in step S332. If the start frame number is set to -1, which is an initialization code at this time, the non-standard data insertion detection unit 26 assigns a value of index r to the start frame number in step S333. If the relative progress speed V[r] is not -1.0, the non-standard data insertion detection unit 26 examines whether the count is greater than or less than a threshold value T2. If a value of the count is greater than the threshold value T2, the non-standard data insertion detection unit 26 detects a section [the start frame number, r-1] as a non-standard row data insertion section in step S334. Next, the non-standard data insertion detecting unit 26 initializes the count and the start image number in step S335.It should be noted that the threshold T2 is appropriately adjusted.

[0080] As described above in the present embodiment, the decision unit 22 determines, based on the relative progress speed V, whether there is an abnormality in the progress of the event X in the input series data 32 and whether consecutive data items of the input series data 32 correspond to corresponding data items of the reference series data 31 in two or more of the combinations extracted by the calculation unit 21.

[0081] The decision unit 22 determines whether an anomaly exists, ie, whether the data for the section R2 is non-standard data, based on a relative speed of progress of the event X in the data for a given section R2 included in the input series data 32.

[0082] An example of a non-standard row data insertion section is shown in Fig. 11. In the present embodiment, the relative progress speed V remains -1.0, and the section R2 whose length exceeds a length determined by the threshold T2 is detected as a non-standard row data insertion section. This enables automatic detection of an abnormal action by a worker, such as "using a dropped part."

[0083] Fig. 12 illustrates a method for the row data exchange section inference process in step S340.

[0084] In step S340, the data exchange inference unit 27 of the decision unit 22 checks whether the input row data 32 is the same in order as the reference row data 31 based on the row data shortage section information 35 and the non-standard row data insertion section information 36 received from the data shortage detection unit 25 and the non-standard data insertion detection unit 26, respectively. The data exchange inference unit 27 outputs the row data exchange section information 37 when an exchange of parts of row data is found. That is, the data exchange inference unit 27 examines, based on the row data shortage section information 35 and the non-standard row data insertion section information 36, whether there is an exchange of parts of row data in the input row data 32 compared with the reference row data 31.

[0085] First, the data exchange inference unit 27 checks whether the number of detected row data shortage sections and the number of detected non-standard row data insertion sections are both one. If at least one of the number of sections is not one, the data exchange inference unit 27 determines that there is no exchange of parts of the series data. If the number of sections is each one, the data exchange inference unit 27 examines whether a difference between a length L1 of a section in the reference series data 31 corresponding to a row data shortage section and a length L2 of a section in the input series data 32 corresponding to a section with a non-standard series data insertion section is smaller than a threshold value T3.If the length difference between the two sections is smaller than the threshold value T3, the data exchange inference unit 27 processes the input series data 32 in step S341 by clipping row data for the non-standard series data section and inserting the row data at a position where the series data deficiency section appears. For processed input series data, the similarity map calculation process in step S210, the score map calculation process in step S220, and the curvature path calculation process in step S230 are performed as for the original input series data 32. In step S342, the data exchange inference unit 27 examines an evaluation value of a curvature path and compares the evaluation value with an evaluation value of the curvature path of the input series data 32 before processing. An evaluation value refers to a value in a score map at a curvature path terminal cell position.If the evaluation value is greater than that before processing, the exchange inference unit 27 detects the two sections as serial data exchange sections in step S343. Note that the threshold T3 is adjusted appropriately.

[0086] As described above, in the present embodiment, the decision unit 22 can determine that there is an anomaly, that is, data for a section R2a corresponding to a given first section R1a in the reference series data 31 is missing from the input series data 32, and an anomaly, that is, data for a given second section R2b included in the input series data 32 is non-standard data. In this case, the decision unit 22 corrects the input series data 32 by deleting the data for the second section R2b from the input series data 32 and inserting the deleted data into the input series data 32 as data for the section R2a corresponding to the first section R1a.

[0087] The calculation unit 21 calculates a first similarity, which is a similarity between the input series data 32 before correction by the decision unit 22 and the reference series data 31, and a second similarity, which is a similarity between the input series data after correction by the decision unit 22 and the reference series data 31. The decision unit 22 compares the first similarity with the second similarity. If the second similarity is higher than the first similarity, the decision unit 22 determines that an anomaly exists, that is, ordinal positions are exchanged in event X in the input series data 32.

[0088] An example of series data exchange sections is shown in Fig. 13. The detection of serial data exchange sections as in the example enables the automatic detection of an abnormal action by a worker, such as "working in a different sequence."

[0089] In step S400, which is Fig. 2, the anomaly information recording unit 28 of the decision unit 22 generates the anomaly information 38 from the row data shortage section information 35, the non-standard row data insertion section information 36, and the row data exchange section information 37, which are respectively received from the data shortage detection unit 25, the non-standard data insertion detection unit 26, and the data exchange inference unit 27. The anomaly information recording unit 28 writes the anomaly information 38 into the anomaly information recording device 109. Examples of output to the anomaly information storage device 109 are shown in FIGS. Fig. 14 and Fig. 15 shown. *** Description of advantageous effects of the embodiment ***

[0090] In the present embodiment, a relative progress speed, that is, a relative speed of the progress of event X in the input series data 32 to the progress of event X in the reference series data 31, is calculated for each combination of a piece of data in the input series data 32 and a piece of data in the reference series data 31, where the data pieces correspond to each other. Then, based on the relative progress speed, it is determined whether there is an abnormality in the progress of the event in the input series data 32. For this reason, the present embodiment is capable of detecting an abnormality in the input series data 32 on a per-piece basis.

[0091] According to the present embodiment, it is possible to detect “insertion of non-standard row data,” “lack of row data,” and “replacement of parts of row data” in the input row data 32 on a per-image basis.

[0092] In the present embodiment, a relative image progress speed between the reference series data 31 and the input series data 32 is calculated from image correspondence information between the input series data 32 and the reference series data 31 obtained by DTW. More specifically, the image correspondence information is a curvature path. A "row data absence" section and a "non-standard row data insertion" section are detected as abnormal sections on a per-image basis. After that, the ordinal positions of pieces of row data for the two abnormal sections are swapped. A change in similarity to the reference series data 31 between the input series data 32 before the swap and the input series data after the swap is examined. If the swap has increased the similarity, it is determined that there is a swap in the input series data 32.The processing techniques described above enable the detection of an anomaly in the input series data 32 at a highly detailed level. *** Other configurations ***

[0093] In the present embodiment, the functions of the calculation unit 21 and the decision unit 22 are implemented by software. As a modification, the functions of the calculation unit 21 and the decision unit 22 are implemented by hardware. The modification will be described primarily with a focus on differences from the present embodiment.

[0094] A configuration of a series data analysis apparatus 10 according to a modification of the present embodiment will be described with reference to Fig. 16 explained.

[0095] The series data analysis device 10 includes hardware such as an electronic circuit 16, the reference series data storage device 13, the input series data storage device 14, and the anomaly information storage device 15.

[0096] The electronic circuit 16 is dedicated hardware that implements the functions of the calculation unit 21 and the decision unit 22. The electronic circuit 16 is, for example, a single circuit, a compound circuit, a programmed processor, a parallel-programmed processor, a logic IC, a GA, an FPGA, an ASIC, or a combination of some or all of these. "IC" stands for integrated circuit. "GA" stands for gate array. "FPGA" stands for field-programmable gate array. "ASIC" stands for application-specific integrated circuit.

[0097] The serial data analysis device 10 may include a plurality of electronic circuits that replace the electronic circuit 16. The plurality of electronic circuits collectively implement the functions of the calculation unit 21 and the decision unit 22. The electronic circuits are, for example, a single circuit, a compound circuit, a programmed processor, a parallel-programmed processor, a logic IC, a GA, an FPGA, an ASIC, or a combination of some or all thereof.

[0098] As another modification, the functions of the calculation unit 21 and the decision unit 22 can be implemented by a combination of software and hardware. That is, some of the functions of the calculation unit 21 and the decision unit 22 can be implemented by dedicated hardware, while the others can be implemented by software.

[0099] The processor 11 and the electronic circuit 16 are both processing circuits. That is, even if the configuration of the serial data analysis device 10 is the one described in one of the Fig. 1 and Fig. 16, the operation of the calculation unit 21 and the decision unit 22 is performed by a processing circuit. List of reference symbols

[0100] 10: Series data analysis device; 11: Processor; 12: Memory; 13: Reference series data storage means; 14: Input series data storage means; 15: Anomaly information storage means; 16: Electronic circuit; 21: Calculation unit; 22: Decision unit; 23: Curvature path calculation unit; 24: Relative progress speed calculation unit; 25: Data shortage detection unit; 26: Non-standard data insertion detection unit; 27: Data exchange inference unit; 28: Anomaly information recording unit; 31: Reference series data; 32: Input series data; 33: Curvature path information; 34: Relative progress speed information; 35: Series data shortage section information; 36: Non-standard row data insertion section information; 37: Series data exchange section information; 38: Anomaly information.

Claims

[1] A serial data analysis device (10) comprising: a calculation unit (21) for extracting combinations each comprising one piece of data from the input series data and one piece of data from the reference series data, the data items corresponding to each other by comparing input series data, which is time series data obtained by observing a given event, with reference series data, which is standard time series data for the event, and for calculating a relative progress rate, which is a relative progress rate of the event in the input series data compared to the progress of the event in the reference series data, for each of the extracted combinations; and a decision unit (22) for determining, based on the relative progress speed calculated by the calculation unit (21), whether there is an anomaly in the progress of the event in the input series data; wherein if the decision unit (22) determines that a first anomaly exists, i.e., that data for a section corresponding to a given first section in the reference series data is missing in the input series data, and a second anomaly exists, i.e., data for a given second section included in the input series data is non-standard data, the decision unit (22) corrects the input series data by deleting from the input series data the data for the second section and inserting into the input series data the deleted data as the data for the section corresponding to the first section, where the non-standard data are series data that do not appear in the reference series data, wherein the calculation unit (21) calculates a first similarity, which is a similarity between the input series data before correction by the decision unit (22) and the reference series data, and a second similarity, which is a similarity between the input series data after correction by the decision unit (22) and the reference series data, and wherein the decision unit (22) compares the first similarity with the second similarity and determines that a third anomaly exists, ie ordinal positions in the event in the input series data are exchanged, if the second similarity is higher than the first similarity. [2] The series data analysis device (10) according to claim 1, wherein the calculation unit (21) calculates a curvature path between the input series data and the reference series data, and extracts an element of the calculated curvature path as a combination of a data element of the input series data and a data element of the reference series data. [3] A series data analysis apparatus (10) according to claim 1 or 2, wherein the input series data is data of a video, and each of the frames of the video is treated as a data item of the input series data. [4] The series data analysis device (10) according to any one of claims 1 to 3, wherein the decision unit (22) determines whether an abnormality exists based on the relative progress speed and whether consecutive data items of the reference series data correspond to corresponding data items of the input series data in two or more of the combinations extracted by the calculation unit (21). [5] A series data analysis device (10) according to any one of claims 1 to 4, wherein the decision unit (22) determines whether an anomaly exists, ie whether data for a section corresponding to the section is missing in the input series data, based on a relative speed of progress of the event in data for a given section included in the reference series data. [6] The series data analysis device (10) according to any one of claims 1 to 5, wherein the decision unit (22) determines whether an abnormality exists based on the relative progress speed and whether consecutive data items of the input series data correspond to corresponding data items of the reference series data in two or more of the combinations extracted by the calculation unit (21). [7] A series data analysis device (10) according to any one of claims 1 to 6, wherein the decision unit (22) determines whether an anomaly exists, ie whether the data for the section is non-standard data, based on a relative speed of progress of the event in the data for a given section included in the input series data. [8] Serial data analysis methods, including: Extracting, by a calculation unit (21), combinations each comprising one piece of data of the input series data and one piece of data of the reference series data, the data items corresponding to each other by comparing input series data, which is time series data obtained by observing a given event, with reference series data, which is standard time series data for the event, and calculating a relative progress speed, which is a relative progress speed of the event in the input series data to the progress of the event in the reference series data, for each of the extracted combinations; and Determining, by a decision unit (22), based on the relative progress speed calculated by the calculation unit (21), whether there is an anomaly in the progress of the event in the input series data; wherein if the decision unit (22) determines that a first anomaly exists, i.e., that data for a section corresponding to a given first section in the reference series data is missing in the input series data, and a second anomaly exists, i.e., data for a given second section included in the input series data is non-standard data, the decision unit (22) corrects the input series data by deleting from the input series data the data for the second section and inserting into the input series data the deleted data as the data for the section corresponding to the first section, where the non-standard data are series data that do not appear in the reference series data, wherein the calculation unit (21) calculates a first similarity, which is a similarity between the input series data before correction by the decision unit (22) and the reference series data, and a second similarity, which is a similarity between the input series data after correction by the decision unit (22) and the reference series data, and wherein the decision unit (22) compares the first similarity with the second similarity and determines that a third anomaly exists, ie ordinal positions in the event in the input series data are exchanged, if the second similarity is higher than the first similarity. [9] Serial data analysis program that causes a computer to perform: a calculation process of extracting combinations each comprising one piece of data of the input series data and one piece of data of the reference series data, the data items corresponding to each other by comparing input series data, which is time series data obtained by observing a given event, with reference series data, which is standard time series data for the event, and calculating a relative progress speed, which is a relative speed of the progress of the event in the input series data to the progress of the event in the reference series data, for each of the extracted combinations; and a decision process of determining, based on the relative progress rate calculated by the calculation process, whether there is an anomaly in the progress of the event in the input series data; wherein, if it is determined in the decision process that a first anomaly exists, i.e., that data for a section corresponding to a given first section in the reference series data is missing in the input series data, and a second anomaly exists, i.e., data for a given second section contained in the input series data is non-standard data, in the decision process, the input series data is corrected by deleting from the input series data the data for the second section and inserting into the input series data the deleted data as the data for the section corresponding to the first section, where the non-standard data are series data that do not appear in the reference series data, wherein in the calculation process, a first similarity is calculated, which is a similarity between the input series data before the correction in the decision process and the reference series data, and a second similarity is calculated, which is a similarity between the input series data after the correction in the decision process and the reference series data, and wherein in the decision process the first similarity is compared with the second similarity and it is determined that a third anomaly exists, ie ordinal positions in the event in the input series data are exchanged if the second similarity is higher than the first similarity.

Images