Class boundary detection device, control method, and program

Abstract

A class boundary detection device (2000) acquires target time-series data (10). The class boundary detection device (2000) extracts, from the target time-series data (10), a plurality of extracted time-series data items, and calculates a similarity between reference time-series data (30) and each of the extracted time-series data items. The reference time-series data (30) is time-series data indicating a class boundary, and has the head portion of time-series data, which belongs to a posterior class and is indicated by the class boundary, after the tail portion of time-series data, which belongs to a prior class and is indicated by the class boundary. The class boundary detection device (2000) detects a class boundary indicated by the reference time-series data (30), from at least one of the extracted time-series data items in which the calculated similarity is not less than a threshold value.

Description

[Technical field]

[0001] The present disclosure relates to techniques for analyzing time series data. [Background technology]

[0002] One piece of time-series data may contain multiple pieces of time-series data that belong to one of multiple classes. For example, the time-series data may be video data obtained by shooting a sequence of multiple work processes. In this case, the video data contains multiple pieces of video data that represent each work process.

[0003] Patent Document 1 discloses a technique for assigning a label of a class to which each data (hereinafter, frame) constituting time series data belongs to. The data processing device of Patent Document 1 extracts frames from the time series data at regular intervals. A user manually assigns a class label to each extracted frame. Furthermore, the data processing device determines whether or not the difference between adjacent frames is equal to or less than a threshold for the multiple extracted frames. If the difference between adjacent frames is equal to or less than the threshold, the labels assigned to these frames are automatically assigned to each frame present between these frames. On the other hand, if the difference between adjacent frames is not equal to or less than the threshold, multiple frames are extracted at smaller intervals from between these frames, and similar processing is performed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-076073 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the invention of Patent Document 1, the user needs to manually assign class labels to the extracted frames. The present invention has been made in consideration of this problem, and one of its objectives is to provide a technique for detecting class boundaries from time-series data. [Means for solving the problem]

[0006] The class boundary detection device of the present disclosure includes an acquisition unit that acquires target time series data that is time series data to be analyzed, a calculation unit that extracts multiple pieces of extractable time series data from the target time series data and calculates reference time series data representing class boundaries between two different classes and similarities between each of the extractable time series data, and a detection unit that detects the class boundary represented by the reference time series data from the extractable time series data whose calculated similarities are equal to or greater than a threshold. The reference time series data includes a tail of time series data belonging to a previous class indicated by the class boundary, followed by a head of time series data belonging to a subsequent class indicated by the class boundary.

[0007] The control method of the present disclosure is executed by a computer. The control method includes an acquisition step of acquiring target time series data that is time series data to be analyzed, a calculation step of extracting multiple pieces of extracted time series data from the target time series data, and calculating a similarity between reference time series data representing class boundaries between two different classes and each of the extracted time series data, and a detection step of detecting a class boundary represented by the reference time series data from the extracted time series data whose calculated similarity is equal to or greater than a threshold. The reference time series data includes a tail of time series data belonging to a previous class indicated by the class boundary, followed by a head of time series data belonging to a subsequent class indicated by the class boundary.

[0008] The non-transitory computer-readable medium of the present disclosure stores a program that causes a computer to execute the control method of the present disclosure. Effect of the Invention

[0009] According to the present disclosure, a technique is provided for detecting class boundaries from time-series data. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of target time-series data handled by a class boundary detection device according to an embodiment; [Diagram 2] FIG. 2 is a diagram illustrating an overview of the operation of a class boundary detection device. [Diagram 3] FIG. 10 is a diagram illustrating reference time-series data. [Figure 4] FIG. 13 is a diagram illustrating detection of class boundaries using reference time series data. [Diagram 5] 1 is a block diagram illustrating a functional configuration of a class boundary detection device; [Figure 6] FIG. 2 is a block diagram illustrating a hardware configuration of a computer that realizes a class boundary detection device. [Figure 7] 1 is a flowchart illustrating a flow of a process executed by a class boundary detection device. [Figure 8] FIG. 13 is a diagram illustrating an example of reference data. [Figure 9] 11 is a diagram illustrating that a plurality of extracted time series data extracted from around a class boundary may be similar to reference time series data corresponding to the type of the class boundary. [Figure 10] 1 is a block diagram illustrating a functional configuration of a class boundary detection device having an output unit. [Figure 11] 13 is a diagram illustrating an example of output information indicating the boundary type and position of each class boundary. [Figure 12] 1 is a block diagram illustrating a functional configuration of a class boundary detection device having a verification unit. [Figure 13] FIG. 13 is a diagram illustrating verification of the consistency of class boundaries. [Figure 14] FIG. 13 is a diagram illustrating verification of the consistency of class boundaries. [Figure 15] 1 is a block diagram illustrating a functional configuration of a class boundary detection device having a correction unit; [Figure 16] FIG. 13 is a diagram illustrating a method for modifying a boundary type. [Figure 17] FIG. 13 is a diagram illustrating an example of a modification screen for modifying a boundary type. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are given the same reference numerals, and duplicated explanations are omitted as necessary for clarity of explanation. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or more arbitrary number of storage devices.

[0012] <Summary> Fig. 1 is a diagram illustrating target time series data 10 handled by a class boundary detection device of an embodiment. The target time series data 10 is composed of multiple time series data 20 each belonging to a different class. For example, the target time series data 10 in Fig. 1 has, in this order, time series data 20-1 belonging to class C1, time series data 20-2 belonging to class C2, and time series data 20-3 belonging to class C3. Hereinafter, each piece of data constituting the time series data will be referred to as a frame. The time series data can be expressed as a data string in which frames are arranged in chronological order.

[0013] The class boundary detection device 2000 can handle various types of time series data as the target time series data 10. For example, the target time series data 10 is video data. The video data is time series data in which a plurality of video frames generated by a video camera are arranged in the order in which they were generated (in ascending order of frame numbers). When the target time series data 10 is video data, for example, each of the time series data 20 included in the target time series data 10 is classified into a class according to the content of the video represented by the time series data 20.

[0014] For example, suppose that a worker performing work consisting of three processes P1, P2, and P3 is filmed by a video camera, and the video data obtained by the filming is treated as target time series data 10. In this case, each work process can be treated as a class. That is, the target time series data 10 can be divided into three time series data 20: time series data 20 including the work status of process P1, time series data 20 including the work status of process P2, and time series data 20 including the work status of process P3.

[0015] The target time series data 10 is not limited to video data. For example, the target time series data 10 may be audio data generated by recording audio with a microphone. Alternatively, the target time series data 10 may be sensing data representing the results of repeated sensing by an arbitrary sensor (e.g., a three-dimensional acceleration sensor).

[0016] FIG. 2 is a diagram illustrating an example of an outline of the operation of the class boundary detection device 2000 according to the embodiment. Here, FIG. 2 is a diagram for facilitating understanding of the outline of the class boundary detection device 2000, and the operation of the class boundary detection device 2000 is not limited to that shown in FIG. 2. For example, in FIG. 2, reference time series data 30 is input to the class boundary detection device 2000. However, as described below, the class boundary detection device 2000 only needs to calculate the similarity between the time series data extracted from the target time series data 10 and the reference time series data 30, and does not need to acquire the reference time series data 30 itself. For example, the class boundary detection device 2000 may acquire the feature amount of the reference time series data 30 instead of the reference time series data 30 itself.

[0017] The class boundary detection device 2000 detects class boundaries from target time series data 10 whose class boundaries are unknown. A class boundary in the target time series data 10 means a boundary between time series data 20 belonging to a certain class and other time series data 20 belonging to another class. For example, assume that the class boundary detection device 2000 handles the target time series data 10 illustrated in FIG. 1. In this case, the class boundary detection device 2000 detects the boundary between the time series data 20-1 and the time series data 20-2 (the boundary between class C1 and class C2) and the boundary between the time series data 20-2 and the time series data 20-3 (the boundary between class C2 and class C3) as class boundaries.

[0018] A class boundary can be represented by the data immediately before or after the class boundary. For example, in the example of Fig. 1, the boundary between the time series data 20-1 and the time series data 20-2 can be represented by the data at the end of the time series data 20-1 or the data at the beginning of the time series data 20-2.

[0019] The class boundary detection device 2000 uses reference time series data 30 to detect class boundaries. Fig. 3 is a diagram illustrating an example of the reference time series data 30. The reference time series data 30 is obtained from time series data 50. The time series data 50 includes time series data 40-1 belonging to a certain class followed by time series data 40-2 belonging to another class. The reference time series data 30 is time series data composed of frames located around the boundary between the time series data 40-1 and the time series data 40-2 (for example, a predetermined number of frames before and after the class boundary).

[0020] 3, time series data 40-1 belongs to class C1, while time series data 40-2 belongs to class C2. The reference time series data 30 is composed of T frames (the last T frames of the time series data 40-1 and the first T frames of the time series data 40-2) before and after the boundary between class C1 and class C2 (T is a natural number). Note that in the reference time series data 30, the number of frames before the class boundary and the number of frames after the class boundary may be different from each other.

[0021] Hereinafter, the type of class boundary represented by the reference time series data 30 will be referred to as the “boundary type corresponding to the reference time series data 30.” For example, in FIG 3, the boundary type corresponding to the reference time series data 30 is the “boundary between class C1 and class C2.”

[0022] The class boundary detection device 2000 extracts one or more pieces of time series data having the same length as the reference time series data 30 from the target time series data 10, and calculates the similarity between each of the extracted time series data and the reference time series data 30. Herein, the time series data extracted from the target time series data 10 is called extracted time series data. The class boundary detection device 2000 detects extracted time series data having a high similarity to the reference time series data 30 (for example, the similarity is equal to or greater than a threshold), and detects a class boundary of a boundary type corresponding to the reference time series data 30 from the detected extracted time series data. Hereinafter, extracted time series data having a high similarity to the reference time series data 30 is also called "extracted time series data matching the reference time series data 30".

[0023] Fig. 4 is a diagram illustrating an example of detection of a class boundary using reference time series data 30. In the example of Fig. 4, the type of class boundary to be detected is the boundary between class C1 and class C2. Therefore, reference time series data 30 corresponding to the boundary type, the boundary between class C1 and class C2, is used.

[0024] The class boundary detection device 2000 extracts a plurality of pieces of extracted time series data 60 from the target time series data 10, and judges whether each piece of extracted time series data 60 has a high similarity to the reference time series data 30. For example, in Fig. 4, extracted time series data 60-X is detected as extracted time series data 60 having a high similarity to the reference time series data 30. Therefore, the class boundary detection device 2000 detects the boundary between class C1 and class C2 from the extracted time series data 60-X.

[0025] 4, the number of frames belonging to class C1 is equal to the number of frames belonging to class C2. In this case, the number of frames belonging to class C1 is likely to be equal to the number of frames belonging to class C2 in the extracted time series data 60-X as well. Therefore, the class boundary detection device 2000 detects the part that divides the extracted time series data 60-X in half as the boundary between class C1 and class C2.

[0026] <Examples of effects> According to this embodiment, it is possible to automatically detect class boundaries from the target time series data 10 based on the similarity between each of the extracted time series data 60 extracted from the target time series data 10 and the reference time series data 30. Therefore, it is possible to more easily detect class boundaries from the target time series data 10 compared to a case in which class labels must be manually assigned to frames constituting the target time series data 10.

[0027] Furthermore, as described below, by detecting class boundaries of each type that may be included in the target time series data 10 from the target time series data 10, it is possible to identify the class to which each frame located between adjacent class boundaries belongs. For example, if a boundary between classes C2 and C3 exists after the boundary between classes C1 and C2, it is clear that each frame located between these boundaries belongs to class C2. This makes it possible to automatically assign class labels to each frame constituting the target time series data 10, making it easier to label the frames.

[0028] The time series data in which each frame is labeled in this manner can be used as training data when generating a machine learning model that automatically classifies the time series data. For example, in a product manufacturing site, in order to improve production quality, production efficiency, or worker safety, it is possible to automatically classify the work process by analyzing video data obtained by shooting a worker performing work with a machine learning model. In order to train such a machine learning model, it is necessary to prepare a large amount of labeled video data. However, it takes a lot of time and effort to manually prepare a large amount of labeled video data.

[0029] In this regard, as described above, if the class boundaries are detected from the target time series data 10 using the class boundary detection device 2000, it is possible to easily label each frame of the target time series data 10 in a short time. Therefore, it is possible to easily prepare the time series data required for training the machine learning model in a short time.

[0030] The class boundary detection device 2000 of this embodiment will be described in more detail below.

[0031] <Example of functional configuration> 5 is a block diagram illustrating a functional configuration of a class boundary detection device 2000 according to an embodiment. The class boundary detection device 2000 includes an acquisition unit 2020, a calculation unit 2040, and a detection unit 2060. The acquisition unit 2020 acquires the target time series data 10. The calculation unit 2040 extracts a plurality of pieces of extracted time series data 60 from the target time series data 10, and calculates the similarity between each piece of extracted time series data 60 and the reference time series data 30. The detection unit 2060 detects a class boundary from the target time series data 10 based on the similarity calculated for each piece of extracted time series data 60. For example, when the reference time series data 30 is the boundary between class C1 and class C2 and time series data around the boundary, the class boundary detection device 2000 detects the boundary between class C1 and class C2 from the target time series data 10.

[0032] <Example of hardware configuration> Each functional component of the class boundary detection device 2000 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a further explanation will be given of the case where each functional component of the class boundary detection device 2000 is realized by a combination of hardware and software.

[0033] 6 is a block diagram illustrating a hardware configuration of a computer 500 that realizes the class boundary detection device 2000. The computer 500 is any computer. For example, the computer 500 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 500 is a portable computer such as a smartphone or a tablet terminal. The computer 500 may be a dedicated computer designed to realize the class boundary detection device 2000, or may be a general-purpose computer.

[0034] For example, by installing a specific application on the computer 500, each function of the class boundary detection device 2000 is realized on the computer 500. The application is configured with a program for realizing each functional component of the class boundary detection device 2000. The method of acquiring the program is arbitrary. For example, the program can be acquired from a storage medium (such as a DVD disk or a USB memory) in which the program is stored. Alternatively, the program can be acquired by downloading the program from a server device that manages the storage device in which the program is stored.

[0035] The computer 500 has a bus 502, a processor 504, a memory 506, a storage device 508, an input / output interface 510, and a network interface 512. The bus 502 is a data transmission path for the processor 504, the memory 506, the storage device 508, the input / output interface 510, and the network interface 512 to transmit and receive data to and from each other. However, the method of connecting the processor 504 and other components to each other is not limited to a bus connection.

[0036] The processor 504 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 506 is a main storage device realized using a random access memory (RAM) or the like. The storage device 508 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read only memory (ROM) or the like.

[0037] The input / output interface 510 is an interface for connecting the computer 500 to an input / output device. For example, the input / output interface 510 is connected to an input device such as a keyboard and an output device such as a display device.

[0038] The network interface 512 is an interface for connecting the computer 500 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0039] The storage device 508 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the class boundary detection device 2000. The processor 504 realizes each functional component of the class boundary detection device 2000 by reading this program into the memory 506 and executing it.

[0040] The class boundary detection device 2000 may be realized by one computer 500, or may be realized by multiple computers 500. In the latter case, the configurations of the computers 500 do not need to be the same, and may be different from each other.

[0041] Some or all of the functions of the class boundary detection device 2000 may be realized by a device that generates the target time series data 10. When all of the functions of the class boundary detection device 2000 are realized by a device that generates the target time series data 10, for example, the device detects class boundaries from the target time series data 10 that it generated, and outputs information indicating the class boundaries together with the target time series data 10.

[0042] For example, assume that the target time-series data 10 is video data. In this case, some or all of the functions of the class boundary detection device 2000 may be realized by a video camera that generates the video data. As such a video camera, for example, a camera called a network camera, an IP (Internet Protocol) camera, or an intelligent camera can be used. When all of the functions of the class boundary detection device 2000 are realized by a video camera, for example, the video camera detects class boundaries from the video data that it generates, and outputs information indicating the class boundaries (output information to be described later) together with the video data.

[0043] <Processing flow> FIG. 7 is a flowchart illustrating a process flow executed by the class boundary detection device 2000 of the embodiment. The acquisition unit 2020 acquires the target time series data 10 (S102). S104 to S116 constitute a loop process L1 executed for each of the to-be-extracted time series data 60 extracted from the target time series data 10. In S104, the class boundary detection device 2000 determines whether the loop process L1 has been executed for all of the to-be-extracted time series data 60. If the loop process L1 has already been executed for all of the to-be-extracted time series data 60, the process of FIG. 7 ends. On the other hand, if there is to-be-extracted time series data 60 that has not yet been the target of the loop process L1, the loop process L1 is executed for the next to-be-extracted time series data 60. Here, the to-be-extracted time series data 60 that is the target of the loop process L1 is denoted as to-be-extracted time series data 60-i.

[0044] S106 to S114 constitute a loop process L2 that is executed for each of one or more reference time series data 30. Each reference time series data 30 corresponds to a different class boundary. In S106, the class boundary detection device 2000 determines whether or not the loop process L2 has been executed for all reference time series data 30. If the loop process L2 has already been executed for all reference time series data 30, the process in FIG. 7 proceeds to S116. Since S116 is the end of the loop process L1, the process in FIG. 7 proceeds to S104.

[0045] On the other hand, if there is still reference time series data 30 that has not yet been the target of the loop process L2, the loop process L2 is executed for the next reference time series data 30. Here, the reference time series data 30 that is the target of the loop process L2 is denoted as reference time series data 30-j.

[0046] The calculation unit 2040 calculates the similarity between the extracted time series data 60-i and the reference time series data 30-j (S108). The detection unit 2060 detects the similarity between the extracted time series data 60-i and the reference time series data 30-j based on the similarity between the extracted time series data 60-i and the reference time series data 30-j. jIt is determined whether or not they match (S110). For example, when the similarity between the extracted time series data 60-i and the reference time series data 30-j is equal to or greater than a predetermined threshold, the detection unit 2060 determines that they match. On the other hand, when the similarity between them is less than the predetermined threshold, the detection unit 2060 determines that they do not match.

[0047] Extracted time series data 60-i and reference time series data 30- j If they match (S110: YES), the detection unit 2060 detects a class boundary of the boundary type corresponding to the reference time series data 30-j from the extracted time series data 60-i (S112). After S112 is executed, or after it is determined that the extracted time series data 60-i and the reference time series data 30-j do not match (S110: NO), the process in Fig. 7 proceeds to S114. Since S114 is the end of the loop process L2, the process in Fig. 7 proceeds to S106.

[0048] It should be noted that the process flow shown in Fig. 7 is merely an example, and the process flow executed by the class boundary detection device 2000 is not limited to the process flow shown in Fig. 7. For example, instead of placing the loop process L2 inside the loop process L1, the loop process L1 may be placed inside the loop process L2.

[0049] <Acquisition of target time series data 10: S102> The acquisition unit 2020 acquires the target time series data 10. Here, various methods can be adopted for acquiring the time series data to be analyzed. For example, the target time series data 10 is assumed to be stored in advance in an arbitrary storage device in a form that can be acquired from the class boundary detection device 2000. In this case, the acquisition unit 2020 acquires the target time series data 10 by reading the target time series data 10 from the storage device. Note that the storage device in which the target time series data 10 is stored may be provided either inside or outside the class boundary detection device 2000. When the storage device in which the target time series data 10 is stored is provided outside the class boundary detection device 2000, for example, the storage device is provided inside the device that generated the target time series data 10.

[0050] Alternatively, for example, the acquiring unit 2020 acquires the target time series data 10 by receiving the target time series data 10 transmitted from another device. The device that transmits the target time series data 10 is, for example, the device that generated the target time series data 10. When the target time series data 10 is video data, for example, the acquiring unit 2020 acquires the target time series data 10 from a video camera that generated the target time series data 10.

[0051] <Extraction of extracted time series data 60> The calculation unit 2040 extracts a plurality of different pieces of extractable time series data 60 from the target time series data 10. Here, the length of each piece of extracted time series data is assumed to be the same as the length of the reference time series data 30.

[0052] Various existing techniques can be used to extract multiple pieces of time series data of a specific length shorter than one piece of time series data. For example, the calculation unit 2040 uses a sliding window to extract multiple pieces of extracted time series data 60 from the target time series data 10. The width of the sliding window (i.e., the length of the extracted time series data 60) is set to the length of the reference time series data 30. An arbitrary value can be set to the stride of the sliding window (i.e., the interval between two adjacent pieces of extracted time series data 60). For example, if the stride value is set to 1, all pieces of extracted time series data 60 that can be extracted from the target time series data 10 are extracted in order from the beginning of the target time series data 10.

[0053] There are various methods for the calculation unit 2040 to specify the length of the reference time series data 30. For example, information indicating the length of the reference time series data 30 is set in advance in the calculation unit 2040. As another example, information indicating the length of the reference time series data 30 is stored in advance in a storage device in a form accessible by the class boundary detection device 2000. In this case, the calculation unit 2040 specifies the length of the reference time series data 30 by acquiring the information indicating the length of the reference time series data 30 from the storage device. As another example, the calculation unit 2040 may specify the length of the reference time series data 30 by acquiring the reference time series data 30.

[0054] <About obtaining information about reference time series data 30> The acquisition unit 2020 acquires data necessary for calculating the similarity between the reference time series data 30 corresponding to the type of class boundary to be detected and the extracted time series data 60. This data is called reference data. For example, the reference data is the reference time series data 30 itself. In addition, for example, the reference data is a feature extracted from the reference time series data 30. The feature will be described later.

[0055] For example, the reference data is stored in advance in a storage device together with boundary type identification information indicating the type of the corresponding class boundary in a manner that can be acquired from the class boundary detection device 2000. Fig. 8 is a diagram illustrating an example of the reference data. A table 100 in Fig. 8 shows reference data 90 corresponding to the boundary type specified by the boundary type identification information 80, in association with the boundary type identification information. In Fig. 8, the reference data 90 is a feature value of the reference time-series data 30.

[0056] Here, the type of class boundary can be expressed by an ordered pair of classes. For example, the type of the boundary between class C1 and class C2 (the boundary representing the transition portion from class C1 to class C2) can be expressed by an ordered pair (C1, C2). Therefore, in FIG. 8, the ordered pair of classes is used for the boundary type identification information 80.

[0057] There are various methods for identifying the reference data 90 to be acquired by the acquiring unit 2020. For example, the acquiring unit 2020 acquires all the reference data 90 stored in the storage device. Alternatively, for example, the acquiring unit 2020 identifies boundary types of class boundaries that may be included in the target time series data 10, and acquires the reference data 90 corresponding to each identified boundary type.

[0058] For example, it is assumed that the classes included in the target time series data 10 are known. In this case, the types of class boundaries that may be included in the target time series data 10 can be expressed as a set of all ordered pairs of classes that may be included in the target time series data 10. For example, it is assumed that the classes that may be included in the target time series data 10 are three classes, C1, C2, and C3. These three classes have six ordered pairs, namely (C1, C2), (C1, C3), (C2, C1), (C2, C3), (C3, C1), and (C3, C2). Therefore, the types of class boundaries included in the target time series data 10 are these six.

[0059] Therefore, for example, the acquiring unit 2020 acquires corresponding reference data 90 for boundary types represented by each ordered pair of classes that may be included in the target time series data 10. For example, when the above three classes may be included in the target time series data 10 in any order, the class boundary detection device 2000 acquires reference data 90 for boundary types represented by each of the above six ordered pairs. As a more specific example, for the boundary types represented by each of the above six ordered pairs, the corresponding reference time series data 30 or the feature amount of the corresponding reference time series data 30 is acquired as the reference data 90.

[0060] <Generation of Reference Time Series Data 30> In order to prepare the reference data 90, it is necessary to generate the reference time series data 30 as a prerequisite. As described with reference to Fig. 3, the reference time series data 30 is a part of the time series data 50 including two time series data 40-1 and 40-2. Therefore, in order to generate the reference time series data 30, it is necessary to generate the time series data 50.

[0061] There are various methods for generating the time series data 50. For example, it is assumed that the time series data 50 including the boundary between classes C1 and C2 is generated. In this case, for example, the time series data 50 is generated by having a device for generating the time series data 50 (such as a camera or various sensors) actually observe a desired situation. For example, it is assumed that the time series data 50 is video data, and the classes C1 and C2 are types of work processes. In this case, it is possible to obtain video data including the boundary between classes C1 and C2 by capturing with a camera the state in which the work process C2 is performed after the work process C1 is performed.

[0062] Alternatively, for example, the time series data 50 may be generated by separately generating the time series data 40-1 before the class boundary and the time series data 40-2 after the class boundary, and then connecting them together in any manner. For example, as described above, it is assumed that the classes C1 and C2 are work processes. In this case, the time series data 40-1 is generated by capturing images of the work process C1 being performed by a camera. Similarly, the time series data 40-2 is generated by capturing images of the work process C2 being performed by a camera. Thereafter, the time series data 40-2 is connected to the end of the time series data 40-1 using video editing software or the like, thereby generating the time series data 50 including the boundary between the classes C1 and C2.

[0063] The reference time series data 30 can be generated by extracting the boundary portion and its surrounding frames from the time series data 50. Here, an existing method can be used to extract a portion of the time series data.

[0064] In addition, when the time series data 40-1 and the time series data 40-2 are generated separately, the reference time series data 30 may be generated without generating the time series data 50. For example, the last T frames of the time series data 40-1 and the first T frames of the time series data 40-2 are extracted and joined together to generate the reference time series data 30 consisting of T frames before and after the boundary between classes C1 and C2.

[0065] <Similarity calculation: S108> The calculation unit 2040 calculates the similarity between the to-be-extracted time series data 60 and the reference time series data 30 (S108). The similarity between the to-be-extracted time series data 60 and the reference time series data 30 can be expressed, for example, by the similarity of their feature amounts. In this case, the calculation unit 2040 calculates the feature amount from the to-be-extracted time series data 60. In addition, when the reference data 90 represents the reference time series data 30 itself, the calculation unit 2040 also calculates the feature amount of the reference time series data 30. When the reference data 90 represents the feature amount of the reference time series data 30, it is assumed that the feature amount of the reference time series data 30 represented by the reference data 90 has been calculated in advance from the reference time series data 30 by the same method as the method for calculating the feature amount from the to-be-extracted time series data 60.

[0066] Here, various methods can be used to calculate the feature of the time series data. For example, the calculation unit 2040 calculates the feature f for each frame constituting the time series data, and uses data in which the feature f calculated for each frame is concatenated as the feature F of the time series data. For example, the feature of the i-th frame of the to-be-extracted time series data 60 is represented as fi, and the to-be-extracted time series data 60 is composed of N frames. In this case, the feature F of the to-be-extracted time series data 60 can be represented as F=(f1, f2, ..., fN). The same applies to the feature of the reference time series data 30.

[0067] There are various methods for calculating the feature amount from each frame. For example, assume that the target time series data is video data. In this case, the frames are video frames that constitute the video data. For example, the calculation unit 2040 extracts image features from the frames as the feature amount of the frames. Any existing technology can be used to extract image features from image data. For example, by inputting a video frame to a Convolutional Neural Network (CNN) that handles image data such as ResNet, the feature amount of the video frame can be obtained from the intermediate layer.

[0068] Furthermore, the calculation unit 2040 may calculate the feature amount of the video frame by adding any data to the image feature extracted from the video frame. Hereinafter, the data added to the image feature is referred to as additional data. For example, it is assumed that the image feature of the video frame is represented by an M-dimensional vector, and the additional data is represented by an L-dimensional vector. In this case, the feature amount of the video frame can be represented by an (M+L)-dimensional vector obtained by concatenating these vectors.

[0069] There are various types of additional data. For example, the calculation unit 2040 detects a specific object from a video frame and calculates additional data representing the posture of the object. For example, the target time-series data 10 is video data generated by filming a person's work. In this case, data representing a person's posture (hand posture, moving body posture, face posture, etc.) can be used as additional data. Note that various existing expressions can be used to represent posture data.

[0070] The feature amount of the time series data is not limited to data in which the feature amount of each frame is concatenated. For example, the calculation unit 2040 calculates the feature amount of the time series data taking into account the time series of the frames. Here, various existing technologies can be used as a technology for extracting the feature amount taking into account the time series from the time series data. For example, by inputting the time series data to a 3D CNN capable of taking into account the time series such as I3D, the feature amount of the time series data can be obtained from the intermediate layer.

[0071] The calculation unit 2040 calculates the similarity between the feature of the to-be-extracted time series data 60 and the feature of the reference time series data 30 as the similarity between the to-be-extracted time series data 60 and the reference time series data 30. Here, various methods can be used to calculate the similarity between the two feature amounts. For example, it is assumed that the feature amount of the time series data is data in which the feature amounts of each frame are linked. In this case, the calculation unit 2040 calculates the similarity for each frame for the feature amount of the to-be-extracted time series data 60 and the feature amount of the reference time series data 30, and treats the calculated similarity statistics (such as an average value) as the similarity of these feature amounts. Here, various indices such as norm and cosine similarity can be used to calculate the similarity between the feature amounts of the two frames. Note that, even when the feature amount includes the above-mentioned additional data, the similarity between the feature amount of the to-be-extracted time series data 60 and the feature amount of the reference time series data 30 can be calculated in a similar manner.

[0072] The method of calculating the similarity between the to-be-extracted time series data 60 and the reference time series data 30 is not limited to the method of using these feature quantities. For example, a machine learning model (e.g., a neural network) that is pre-trained to output the similarity between two pieces of time series data in response to the input of the two pieces of time series data is used to calculate the similarity. In this case, the calculation unit 2040 can obtain the similarity between the to-be-extracted time series data 60 and the reference time series data 30 by inputting them into the trained model. Note that existing technology can be used to train the machine learning model to calculate the similarity between the two pieces of time series data.

[0073] <Class boundary detection: S112> When the to-be-extracted time series data 60 and the reference time series data 30 match (S110: YES), the detection unit 2060 detects a class boundary from this to-be-extracted time series data 60. More specifically, the detection unit 2060 identifies the type and position of the class boundary contained in the to-be-extracted time series data 60 that is determined to match the reference time series data 30.

[0074] The type of class boundary included in the to-be-extracted time series data 60 is the boundary type corresponding to the reference time series data 30 that matches the to-be-extracted time series data 60. For example, assume that the to-be-extracted time series data 60 that matches the reference time series data 30 whose corresponding boundary type is (C1, C2) is detected. In this case, the detection unit 2060 identifies the type of class boundary included in the to-be-extracted time series data 60 as (C1, C2). In other words, the class boundary is identified as the boundary between class C1 and class C2.

[0075] The position of the class boundary in the to-be-extracted time series data 60 can be specified based on the position of the class boundary in the reference time series data 30 that matches the to-be-extracted time series data. For example, it is assumed that the reference time series data 30 determined to match the to-be-extracted time series data 60 has time series data of length Y that belongs to class C2 after time series data of length X that belongs to class C1. In this case, the calculation unit 2040 specifies the Xth frame from the beginning of the to-be-extracted time series data 60 or the (X+1)th frame from the beginning of the to-be-extracted time series data 60 as a frame that represents the position of the class boundary. Here, the Xth frame from the beginning of the to-be-extracted time series data 60 represents the end of the sequence of frames that belong to class C1. Meanwhile, the (X+1)th frame from the beginning of the to-be-extracted time series data 60 represents the start of the sequence of frames that belong to class C2.

[0076] Here, there is a possibility that a plurality of pieces of extracted time series data 60 extracted from around a class boundary are similar (have a high degree of similarity) to the reference time series data 30 corresponding to the type of the class boundary. In this case, for example, the detection unit 2060 detects the class boundary from any one of the plurality of pieces of extracted time series data 60. This point will be described with reference to FIG.

[0077] Fig. 9 shows that a plurality of extracted time series data 60 extracted from around a class boundary may be similar to the reference time series data 30 corresponding to the type of the class boundary. In Fig. 9, the reference time series data 30 corresponds to the boundary type (C1, C2), and its center portion is the class boundary. In addition, the target time series data 10 has a class boundary (C1, C2).

[0078] 9, the subject time series data 10 has a class boundary (C1, C2) at its center in the subject time series data 60-1. However, not only the subject time series data 60-1 but also other subject time series data 60 adjacent thereto may be located near the class boundary and therefore may have high similarity with the reference time series data 30. Therefore, subject time series data 60 having high similarity with the reference time series data 30 may be detected other than the subject time series data 60-1.

[0079] In this way, when a plurality of adjacent extracted time series data 60 are detected as the extracted time series data 60 similar to the reference time series data 30, for example, the detection unit 2060 detects a class boundary from one of the plurality of extracted time series data 60. Here, there are various methods for determining the extracted time series data 60 to be used for detecting the class boundary. For example, the detection unit 2060 detects a class boundary from the extracted time series data 60 having the highest similarity to the reference time series data 30 among the plurality of extracted time series data 60. In another example, the detection unit 2060 detects a class boundary from the extracted time series data 60 located in the middle in the chronological order among the plurality of extracted time series data 60. For example, it is assumed that all of the seven adjacent extracted time series data 60 are similar to the same reference time series data 30. In this case, the extracted time series data 60 that is the fourth from the front in the chronological order is determined as the extracted time series data 60 to be used for detecting the class boundary. Alternatively, for example, the detection unit 2060 randomly selects one of the multiple pieces of time-series data 60 to be extracted, and detects a class boundary from the selected piece of time-series data 60 to be extracted.

[0080] <Output of processing results> The class boundary detection device 2000 may output the processing results in any manner. Hereinafter, information output by the class boundary detection device 2000 is referred to as output information. Also, a functional configuration unit that generates and outputs the output information is referred to as an output unit. Fig. 10 is a block diagram illustrating an example of the functional configuration of the class boundary detection device 2000 having an output unit 2080.

[0081] For example, the output unit 2080 generates, as output information, information indicating the boundary type and position of each detected class boundary. FIG. 11 is a diagram illustrating an example of output information indicating the boundary type and position of each class boundary. A table 110 in FIG. 11 indicates boundary type identification information 112 and boundary position 114 for each class boundary detected from the target time series data 10. The boundary type identification information 112 indicates the type of class boundary as an ordered pair of classes. The boundary position 114 indicates the position of the class boundary. Here, it is assumed that the position of the class boundary indicates the identification information (frame number, etc.) of the frame located immediately before the class boundary.

[0082] Alternatively, for example, the output unit 2080 generates the target time series data 10 in which a label indicating the class to which each frame belongs is added to each frame as output information. For example, when each class boundary detected from the target time series data 10 is as shown in the example of FIG. 11, the first frame to the n1th frame of the target time series data 10 all belong to class C1. Therefore, the output unit 2080 adds a label indicating class C1 to each of these frames. Also, in the example of FIG. 11, each of the (n1+1)th to n2th frames of the target time series data 10 belongs to class C2. Therefore, the output unit 2080 assigns a label indicating class C2 to each of these frames.

[0083] Here, if a class label is manually added to each frame constituting the time series data, a long time and a large amount of effort is required. In this regard, if the class boundary detection device 2000 is configured to output the target time series data 10 to which a class label has been added as described above as output information, the process of adding the label of the class to which each frame of the target time series data 10 belongs to is automatically performed based on the detection result of the class boundary by the class boundary detection device 2000. Therefore, the process of adding a label to each frame of the target time series data 10 can be easily realized in a short time.

[0084] The output information may be output in any manner. For example, the output unit 2080 may store the output information in any storage device. For another example, the output unit 2080 may transmit the output information to another device. For another example, the output unit 2080 may display the output information on a display device.

[0085] <Verifying class boundary consistency> When multiple types of class boundaries are detected from the target time series data 10, the consistency of adjacent class boundaries may be verified. Hereinafter, a functional configuration unit that performs this verification is referred to as a verification unit. FIG. 12 is a block diagram illustrating a functional configuration of a class boundary detection device 2000 having a verification unit 2100.

[0086] Specifically, the verification unit 2100 judges whether or not the subsequent class indicated by the front class boundary and the previous class indicated by the rear class boundary match for two adjacent class boundaries. If they match, the verification unit 2100 judges that the two class boundaries are consistent. On the other hand, if they do not match, the verification unit 2100 judges that the two class boundaries are not consistent. Note that the previous class indicated by the class boundary means the first class in the ordered pair of the boundary type of the class boundary. For example, if the boundary type is (C1, C2), the previous class indicated by the class boundary is C1. On the other hand, the subsequent class indicated by the class boundary means the second class in the ordered pair of the boundary type of the class boundary. For example, if the boundary type is (C1, C2), the subsequent class indicated by the class boundary is C2.

[0087] 13 and 14 are diagrams illustrating verification of the consistency of class boundaries. In FIG. 13, after a class boundary B1 of (C1, C2) is detected, a class boundary B2 of (C2, C3) is detected. Here, the subsequent class indicated by the front class boundary B1 is C2, and the previous class indicated by the rear class boundary B2 is also C2. From this, both of these two class boundaries indicate that the time series data 20 between these class boundaries belongs to class C2. Therefore, it can be said that the class boundaries are consistent.

[0088] 14, after a class boundary B3 of (C1, C2) is detected, a class boundary B4 of (C3, C4) is detected. Here, the subsequent class indicated by the front class boundary B3 is C2, while the previous class indicated by the rear class boundary B4 is C3. In this case, the class boundary B3 indicates that the time series data 20 between the class boundaries B3 and B4 belongs to class C2. On the other hand, the class boundary B4 indicates that this time series data 20 belongs to class C3. Therefore, these class boundaries are not consistent.

[0089] The class boundary detection device 2000 may execute various processes in response to detection of class boundary inconsistency. For example, when it is determined that two class boundaries are inconsistent, the class boundary detection device 2000 corrects the type of one of the class boundaries. A functional configuration unit that corrects the type of the class boundary is called a correction unit. FIG. 15 is a block diagram illustrating a functional configuration of the class boundary detection device 2000 having a correction unit 2120.

[0090] For example, for each of two class boundaries (e.g., class boundaries B3 and B4 in FIG. 14) that are determined to be inconsistent with adjacent class boundaries, the correction unit 2120 corrects the type of class boundary based on the similarity between the extracted time series data 60 used to detect the class boundary and the reference time series data 30. Here, it is assumed that, when detecting a class boundary, the similarity of features is calculated for each frame in order to calculate the similarity between the extracted time series data 60 and the reference time series data 30.

[0091] The method of correcting the boundary type will be specifically described with reference to Fig. 16. Fig. 16 is a diagram showing the method of correcting the boundary type. In Fig. 16, similar to Fig. 14, after a class boundary B3 whose boundary type is (C1, C2) is detected, a class boundary B4 whose boundary type is (C3, C4) is detected.

[0092] Graph 130 shows the similarity for each frame between the extracted time series data 60 in which the class boundary B3 was detected and the reference time series data 30 corresponding to the boundary type (C1, C2). Looking at graph 130, the similarity for each frame is high for the portion before the class boundary B3, but is low for the portion before the class boundary B3. From this, it is considered that there is high confidence that the portion before the class boundary B3 is class C1, but low confidence that the portion after the class boundary B3 is class C2.

[0093] Graph 140 shows the similarity for each frame between the extracted time series data 60 in which class boundary B4 was detected and the reference time series data 30 corresponding to the boundary type (C3, C4). Graph 140 shows that the similarity for each frame is high for both the portion before and the portion after class boundary B4. This suggests that there is a high degree of confidence in both the fact that the portion before class boundary B4 is class C3 and the fact that the portion after class boundary B4 is class C4.

[0094] By analyzing the graphs 130 and 140 in this way, it can be said that the time-series data 20 between the class boundary B3 and the class boundary B4 is more likely to belong to class C3 than to class C2. Therefore, the modification unit 2120 modifies the boundary type of the class boundary B3 from (C1, C2) to (C1, C3).

[0095] In order to more specifically realize the process described with reference to FIG. 16, for example, the correction unit 2120 performs the following process for two adjacent class boundaries that are inconsistent with each other. First, the correction unit 2120 calculates a statistical value of similarity (e.g., an average value) of each frame in a portion before the class boundary and a statistical value of similarity of each frame in a portion after the class boundary for the extracted time series data 60 in which a previous class boundary (B3 in FIG. 16) is detected and the reference time series data 30 corresponding to the boundary type of the class boundary, and compares them. For example, in the example of FIG. 16, for the graph 130, the statistical value of similarity calculated for the portion before the class boundary B3 is compared with the statistical value of similarity calculated for the portion after the class boundary B3.

[0096] If the difference between the two calculated statistical values ​​is small (e.g., the difference or ratio between the two statistical values ​​is within a specified numerical range), the current class boundary is likely to be correct. On the other hand, if the difference between the two statistical values ​​is large (e.g., the difference or ratio between the two statistical values ​​is outside a specified numerical range), the current class boundary is likely to be incorrect.

[0097] Similarly, the correction unit 2120 calculates the statistical value of similarity of each frame before the class boundary (B4 in FIG. 16) for the extracted time series data 60 in which the later class boundary is detected and the statistical value of similarity of each frame after the class boundary for the reference time series data 30 corresponding to the boundary type of the class boundary, and compares them. For example, in the example of FIG. 16, the statistical value of similarity calculated for the part before the class boundary B4 for the graph 140 is compared with the statistical value of similarity calculated for the part after the class boundary B4. In this case as well, when the difference between the two calculated statistical values ​​is small, the current class boundary is highly likely to be correct, whereas when the difference between the two statistical values ​​is large, the current class boundary is highly likely to be incorrect.

[0098] Using the above-mentioned method, the correction unit 2120 determines whether the previous class boundary or the subsequent class boundary is in error. If the difference in the similarity statistics described above is large at the previous class boundary, the correction unit 2120 determines that the class subsequent to the class boundary at the previous class boundary is in error. On the other hand, if the difference in the similarity statistics described above is large at the subsequent class boundary, the correction unit 2120 determines that the class prior to the class boundary at the subsequent class boundary is in error.

[0099] 16, there is a large difference between the similarity statistics calculated for the portion before class boundary B3 and the similarity statistics calculated for the portion after class boundary B3. On the other hand, there is a small difference between the similarity statistics calculated for the portion before class boundary B4 and the similarity statistics calculated for the portion after class boundary B4. Therefore, the correction unit 2120 determines that there is an error in class boundary B3.

[0100] The correcting unit 2120 corrects the class in which an error is identified. For example, the correcting unit 2120 corrects the class boundary identified as having an error so as to match the class boundary identified as having no error. When it is identified that the previous class boundary has an error and the subsequent class boundary has no error, the correcting unit 2120 corrects the previous class boundary so as to match the subsequent class boundary. More specifically, the correcting unit 2120 changes the subsequent class indicated by the previous class boundary (C2 at class boundary B3 in the example of FIG. 16) to the previous class indicated by the subsequent class boundary (C3 at class boundary B4 in the example of FIG. 16).

[0101] On the other hand, if it is determined that the previous class boundary is not erroneous and the subsequent class boundary is erroneous, the modification unit 2120 modifies the subsequent class boundary to be consistent with the previous class boundary. More specifically, the modification unit 2120 changes the previous class indicated by the subsequent class boundary to the subsequent class indicated by the previous class boundary.

[0102] The method of correcting the class boundary is not limited to the above-mentioned method. For example, the correction unit 2120 corrects the class boundary by the following method. First, a case where an error is identified in the previous class boundary of two adjacent class boundaries will be described. For this case, time series data of the beginning part (for example, time series data consisting of the beginning T frames) of time series data belonging to each class is prepared. For example, it is assumed that there are three types of classes, namely, classes C1, C2, and C3. In this case, three time series data are prepared, namely, time series data of the beginning part of the time series data belonging to class C1, time series data of the beginning part of the time series data belonging to class C2, and time series data of the beginning part of the time series data belonging to class C3. These time series data are called second reference time series data.

[0103] When it is determined that the previous class boundary is erroneous, the correction unit 2120 calculates the similarity between the time series data in the part after the class boundary of the extracted time series data 60 in which the class boundary is detected and each second reference time series data. Then, when there is second reference time series data having a high similarity (for example, the similarity is equal to or greater than a threshold) with the time series data after the class boundary in the extracted time series data 60, the correction unit 2120 changes the class after the class boundary to the class corresponding to the second reference time series data. Here, the correction unit 2120 may obtain the second reference time series data itself and calculate the feature amount thereof, or may obtain the feature amount of the second reference time series data. In addition, the second reference time series data is associated with the identification information of the corresponding class and is stored in advance in a storage device in a manner accessible from the class boundary detection device 2000.

[0104] For example, in the example of Fig. 16, it is determined that there is an error in the class boundary B3. Therefore, the correction unit 2120 calculates the similarity between the second reference time series data of each class and the time series data of the part after the class boundary B3 in the extracted time series data 60 in which the class boundary B3 is detected. As a result, it is determined that the similarity calculated for the second reference time series data corresponding to class C3 is high. In this case, the correction unit 2120 changes the class after the class boundary B3 from C2 to C3.

[0105] Next, a case where an error is identified in the latter class boundary among the adjacent class boundaries will be described. In this case, time series data of the end part (for example, time series data consisting of the last T frames) is prepared for the time series data belonging to each class. This time series data is called the third reference time series data.

[0106] When it is determined that the subsequent class boundary is erroneous, the correction unit 2120 calculates the similarity between the time series data in the portion before the class boundary of the extracted time series data 60 in which the class boundary is detected and each third reference time series data. Then, when there is third reference time series data having a high similarity (for example, the similarity is equal to or greater than a threshold) with the time series data before the class boundary in the extracted time series data 60, the correction unit 2120 changes the class before the class boundary to the class corresponding to the third reference time series data. Here, the correction unit 2120 may obtain the third reference time series data itself and calculate the feature amount thereof, or may obtain the feature amount of the third reference time series data. In addition, it is assumed that the third reference time series data is associated with the identification information of the corresponding class and is stored in advance in a storage device in a manner accessible from the class boundary detection device 2000.

[0107] The correction of the class boundaries may be performed manually by a user of the class boundary detection device 2000, instead of being performed automatically by the correction unit 2120 as described above. In this case, the correction unit 2120 provides the user with a screen for correcting the boundary type, and corrects the class boundaries according to the result of the user's input operation. This will be described in detail below.

[0108] Fig. 17 is a diagram illustrating an example of a correction screen for correcting a boundary type. The correction screen 200 has display areas 210, 220, and 230. The display area 210 includes a graph 240 showing the magnitude of similarity between the extracted time series data 60 and the reference time series data 30. In this graph, the reference time series data 30 used for comparison with the extracted time series data 60 can be selected in an input area 250. In Fig. 17, (C1, C2) is selected. The display area 210 further includes a boundary display 260 showing the detected class boundary, and a selection display 270 showing the focused time series data.

[0109] Display area 220 shows a predetermined number of frames (five in FIG. 17 ) including the center of the time-series data displayed in selection display 270. Mark 280 is a mark indicating a frame displayed in display area 230. Display area 230 is an area in which a frame to which mark 280 is attached is displayed. The user can perform an input operation to specify one of the frames displayed in display area 220. Then, mark 280 is attached to the frame specified by the input operation, and the frame is displayed in display area 230.

[0110] A user can modify a class boundary by using the modification screen 200. For example, when a class boundary is included in the time series data selected by the selection display 270, the user can modify the boundary type of the class boundary by using the input area 290 and the input area 300. The input area 290 is used to modify the previous class indicated by the class boundary. On the other hand, the input area 300 is used to modify the next class indicated by the class boundary. For example, in the example of FIG. 17, the class boundary currently selected by the selection display 270 is (C1, C2). Therefore, C1 is displayed in the input area 290, and C2 is displayed in the input area 300. Here, if the user selects C3 in the input area 290, the modification unit 2120 modifies the class boundary (C1, C2) to (C3, C2). Also, if the user selects C3 in the input area 300, the modification unit 2120 modifies the class boundary (C1, C2) to (C1, C3).

[0111] Here, the correction unit 2120 may specify an erroneous class boundary by the above-mentioned method, and present the correction screen 200 using the result of the specification. For example, the correction unit 2120 exemplifies the correction screen 200 in response to the determination that there is an inconsistency between adjacent class boundaries. At this time, the correction unit 2120 specifies which of the two adjacent class boundaries has an error by the above-mentioned method. Then, the correction unit 2120 generates the correction screen 200 focused on the class boundary specified to have an error (the selection display 270 is displayed). At this time, the graph 240 displays a graph showing the similarity between the reference time series data 30 corresponding to the boundary type of the class boundary specified to have an error and the extracted time series data 60. In this way, the correction screen 200 focusing on the class boundary specified to have an error is provided to the user. Therefore, the user can easily correct the class boundary specified to have an error.

[0112] An input operation for correcting the position of a class boundary may be possible on the correction screen 200. For example, in the graph 240, the user may move the boundary display 260 left or right to change the position of the class boundary.

[0113] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0114] In the above example, the program includes a set of instructions (or software code) for making the computer perform one or more functions described in the embodiment when the program is loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagating signals.

[0115] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) An acquisition unit that acquires target time series data that is time series data to be analyzed; a calculation unit that extracts a plurality of pieces of extracted time series data from the target time series data and calculates a similarity between each of the pieces of extracted time series data and reference time series data that represents a class boundary between two mutually different classes; a detection unit that detects a class boundary represented by the reference time series data from the extracted time series data in which the calculated similarity is equal to or greater than a threshold value; The reference time series data has a tail portion of time series data belonging to a previous class indicated by the class boundary, followed by a head portion of time series data belonging to a subsequent class indicated by the class boundary. (Appendix 2) The class boundary detection device according to claim 1, wherein the calculation unit calculates a similarity between a feature of the reference time series data and a feature of the extracted time series data as the similarity between the reference time series data and the extracted time series data. (Appendix 3) the target time series data is video data that is a sequence of video frames; the feature amount of the extracted time series data indicates a feature amount of each frame constituting the extracted time series data, The class boundary detection device according to claim 2, wherein the calculation unit calculates, for each video frame constituting the extracted time series data, a feature amount indicating an image feature of the video frame and data indicating a posture of a predetermined object contained in the video frame. (Appendix 4) The calculation unit calculates a similarity between each of the reference time series data corresponding to each of a plurality of types of class boundaries and each of the extracted time series data; The class boundary detection device according to any one of claims 1 to 3, wherein the detection unit detects a class boundary of a type corresponding to the reference time series data from the extracted time series data whose similarity to the reference time series data is equal to or greater than a threshold. (Appendix 5) a verification unit that determines whether or not two adjacent class boundaries detected from the target time series data are consistent; A correction unit that corrects the class boundary when it is determined that two adjacent class boundaries are not consistent. (Appendix 6) The class boundary detection device according to claim 5, wherein the correction unit compares, for each of two class boundaries determined to be inconsistent, the degree of similarity between each frame in the extracted time series data in which the class boundary was detected and the reference time series data corresponding to the type of the class boundary, between a portion before the class boundary and a portion after the class boundary, and determines that the class boundary is erroneous if the difference is equal to or greater than a threshold. (Appendix 7) 7. The class boundary detection device according to claim 6, wherein the correction unit corrects a class boundary determined to be erroneous so as to match a class boundary determined to be non-erroneous. (Appendix 8) The correction unit is Regarding the similarity between each frame of the to-be-extracted time-series data in which a class boundary determined to be erroneous has been detected and the reference time-series data corresponding to the type of the class boundary, if the similarity calculated for the portion before the class boundary is lower than the similarity calculated for the portion after the class boundary, calculate the similarity between the to-be-extracted time-series data and the tail portion of the time-series data belonging to each class, and correct the class before the class boundary determined to be erroneous for the class in which the similarity is equal to or greater than a threshold value; The class boundary detection device according to claim 6, wherein, regarding the inter-frame similarity between the to-be-extracted time series data in which a class boundary determined to be erroneous has been detected and the reference time series data corresponding to the type of the class boundary, when the similarity calculated for the portion after the class boundary is lower than the similarity calculated for the portion before the class boundary, the device calculates a similarity between the to-be-extracted time series data and a leading portion of time series data belonging to each class, and corrects the class after the class boundary determined to be erroneous among the classes in which the similarity is equal to or greater than a threshold. (Appendix 9) The class boundary detection device according to claim 5, wherein the correction unit outputs a correction screen that accepts an input operation to correct the previous class, the next class, or both indicated by the class boundary, and corrects the class boundary in accordance with the input operation performed on the correction screen. (Appendix 10) 1. A computer-implemented control method comprising: An acquisition step of acquiring target time series data which is time series data to be analyzed; A calculation step of extracting a plurality of extracted time series data from the target time series data, and calculating a similarity between each of the extracted time series data and reference time series data that represents a class boundary between two mutually different classes; a detection step of detecting a class boundary represented by the reference time series data from the extracted time series data in which the calculated similarity is equal to or greater than a threshold value; A control method, wherein the reference time series data has a leading portion of time series data belonging to a succeeding class indicated by the class boundary, followed by a trailing portion of time series data belonging to a preceding class indicated by the class boundary. (Appendix 11) The control method according to claim 10, wherein in the calculation step, a similarity between a feature of the reference time series data and a feature of the extracted time series data is calculated as the similarity between the reference time series data and the extracted time series data. (Appendix 12) the target time series data is video data that is a sequence of video frames; the feature amount of the extracted time series data indicates a feature amount of each frame constituting the extracted time series data, The control method according to claim 11, wherein in the calculation step, for each video frame constituting the extracted time series data, a feature quantity indicating an image feature of the video frame and data representing a posture of a predetermined object contained in the video frame is calculated. (Appendix 13) In the calculation step, a similarity between each of the reference time series data corresponding to each of a plurality of types of class boundaries and each of the extracted time series data is calculated; 13. The control method according to claim 10, wherein in the detection step, a class boundary of a type corresponding to the reference time series data is detected from the extracted time series data whose similarity to the reference time series data is equal to or greater than a threshold. (Appendix 14) a verification step of determining whether or not two adjacent class boundaries detected from the target time series data are consistent; A control method according to any one of appendixes 10 to 13, further comprising a correction step of correcting the class boundary when it is determined that two adjacent class boundaries are not consistent. (Appendix 15) The control method according to claim 14, wherein, for each of two class boundaries determined to be inconsistent in the correction step, the similarity between each frame of the extracted time series data in which the class boundary was detected and the reference time series data corresponding to the type of the class boundary is compared between a part before the class boundary and a part after the class boundary, and if the difference is equal to or greater than a threshold, it is determined that the class boundary is erroneous. (Appendix 16) A control method according to claim 15, wherein in the correcting step, a class boundary determined to be erroneous is corrected so as to be consistent with a class boundary determined to be non-erroneous. (Appendix 17) In the modification step, Regarding the similarity between each frame of the to-be-extracted time-series data in which a class boundary determined to be erroneous has been detected and the reference time-series data corresponding to the type of the class boundary, if the similarity calculated for the portion before the class boundary is lower than the similarity calculated for the portion after the class boundary, calculate the similarity between the to-be-extracted time-series data and the tail portion of the time-series data belonging to each class, and correct the class before the class boundary determined to be erroneous for the class in which the similarity is equal to or greater than a threshold value; The control method according to claim 15, wherein, regarding the inter-frame similarity between the to-be-extracted time series data in which a class boundary determined to be erroneous has been detected and the reference time series data corresponding to the type of the class boundary, when the similarity calculated for the portion after the class boundary is lower than the similarity calculated for the portion before the class boundary, the control method calculates a similarity between the to-be-extracted time series data and a leading portion of time series data belonging to each class, and corrects the class after the class boundary determined to be erroneous among the classes in which the similarity is equal to or greater than a threshold. (Appendix 18) The control method according to claim 14, wherein in the modification step, a modification screen is output that accepts an input operation to modify the previous class, the next class, or both indicated by the class boundary, and the class boundary is modified in accordance with the input operation performed on the modification screen. (Appendix 19) A non-transitory computer-readable medium having a program stored thereon, The program includes: An acquisition step of acquiring target time series data which is time series data to be analyzed; A calculation step of extracting a plurality of extracted time series data from the target time series data, and calculating a similarity between each of the extracted time series data and reference time series data that represents a class boundary between two mutually different classes; a detection step of detecting a class boundary represented by the reference time series data from the extracted time series data having the calculated similarity equal to or greater than a threshold value; A non-transitory computer-readable medium, wherein the reference time series data has a tail portion of time series data belonging to a previous class indicated by the class boundary followed by a head portion of time series data belonging to a subsequent class indicated by the class boundary. (Appendix 20) 20. The computer-readable medium of claim 19, wherein in the calculating step, a similarity between a feature of the reference time series data and a feature of the extracted time series data is calculated as the similarity between the reference time series data and the extracted time series data. (Appendix 21) the target time series data is video data that is a sequence of video frames; the feature amount of the extracted time series data indicates a feature amount of each frame constituting the extracted time series data, 21. The computer-readable medium of claim 20, wherein in the calculation step, for each video frame constituting the extracted time series data, a feature value indicating an image feature of the video frame and data representing a posture of a predetermined object contained in the video frame is calculated. (Appendix 22) In the calculation step, a similarity between each of the reference time series data corresponding to each of a plurality of types of class boundaries and each of the extracted time series data is calculated; 22. The computer-readable medium of claim 19, wherein in the detection step, a class boundary of a type corresponding to the reference time series data is detected from the extracted time series data whose similarity to the reference time series data is equal to or greater than a threshold. (Appendix 23) a verification step of determining whether or not two adjacent class boundaries detected from the target time series data are consistent; and modifying the class boundaries if two adjacent class boundaries are determined to be inconsistent. (Appendix 24) The computer-readable medium of claim 23, wherein in the correction step, for each of two class boundaries determined to be inconsistent, the similarity between each frame of the extracted time series data in which the class boundary was detected and the reference time series data corresponding to the type of the class boundary is compared between a part before the class boundary and a part after the class boundary, and if the difference is equal to or greater than a threshold, it is determined that the class boundary is erroneous. (Appendix 25) 25. The computer-readable medium of claim 24, wherein in the modifying step, class boundaries determined to be erroneous are modified to match class boundaries determined to be non-erroneous. (Appendix 26) In the modification step, Regarding the similarity between each frame of the to-be-extracted time-series data in which a class boundary determined to be erroneous has been detected and the reference time-series data corresponding to the type of the class boundary, if the similarity calculated for the portion before the class boundary is lower than the similarity calculated for the portion after the class boundary, calculate the similarity between the to-be-extracted time-series data and the tail portion of the time-series data belonging to each class, and correct the class before the class boundary determined to be erroneous for the class in which the similarity is equal to or greater than a threshold value; The computer-readable medium of claim 24, wherein, for a similarity between each frame of the to-be-extracted time series data in which a class boundary determined to be erroneous has been detected and the reference time series data corresponding to the type of the class boundary, if a similarity calculated for a portion after the class boundary is lower than a similarity calculated for a portion before the class boundary, a similarity between the to-be-extracted time series data and a leading portion of time series data belonging to each class is calculated, and for a class in which the similarity is equal to or greater than a threshold, the class after the class boundary determined to be erroneous is corrected. (Appendix 27) The computer-readable medium of claim 23, wherein in the modification step, a modification screen is output that accepts input operations to modify the previous class, the next class, or both indicated by the class boundary, and the class boundary is modified in accordance with the input operations performed on the modification screen. [Explanation of symbols]

[0116] 10 Target time series data 20 Time Series Data 30 Reference Time Series Data 40 Time Series Data 50 Time Series Data 60 Extracted time series data 80 Boundary Type Identification Information 90 Reference Data 100 Tables 110 Table 112 Boundary type identification information 114 Boundary position 130 Graphs 140 Graphs 200 Modification screen 210 Display Area 220 Display Area 230 Display Area 240 Graphs 250 Input Area 260 Boundary display 270 Selected Display 280 marks 290 Input Area 300 Input Area 500 Computers 502 Bus 504 Processor 506 Memory 508 Storage Devices 510 Input / Output Interface 512 Network Interface 2000 Class Boundary Detector 2020 Acquisition Department 2040 Calculation Department 2060 Detection unit 2080 Output section 2100 Verification Department 2120 Correction Department

Claims

1. An acquisition unit that acquires target time series data that is time series data to be analyzed; a calculation unit that extracts a plurality of pieces of extracted time series data from the target time series data and calculates a similarity between each of the pieces of extracted time series data and reference time series data that represents a class boundary between two mutually different classes; a detection unit that detects a class boundary represented by the reference time series data from the extracted time series data in which the calculated similarity is equal to or greater than a threshold value; A class boundary detection device, wherein the reference time series data has a tail portion of time series data belonging to a previous class indicated by the class boundary, followed by a head portion of time series data belonging to a subsequent class indicated by the class boundary.

2. The class boundary detection device according to claim 1 , wherein the calculation unit calculates a similarity between a feature amount of the reference time series data and a feature amount of the extracted time series data as the similarity between the reference time series data and the extracted time series data.

3. the target time series data is video data that is a sequence of video frames; the feature amount of the extracted time series data indicates a feature amount of each frame constituting the extracted time series data, 3. The class boundary detection device according to claim 2, wherein the calculation unit calculates, for each video frame constituting the extracted time series data, a feature amount indicating an image feature of the video frame and data indicating a posture of a predetermined object included in the video frame.

4. The calculation unit calculates a similarity between each of the reference time series data corresponding to each of a plurality of types of class boundaries and each of the extracted time series data; The class boundary detection device according to claim 1 , wherein the detection unit detects a class boundary of a type corresponding to the reference time series data from the extracted time series data whose similarity to the reference time series data is equal to or greater than a threshold.

5. a verification unit that determines whether or not two adjacent class boundaries detected from the target time series data are consistent; 5. The class boundary detection device according to claim 1, further comprising: a correction unit that corrects the class boundary when it is determined that two adjacent class boundaries are not consistent.

6. 6. The class boundary detection device according to claim 5, wherein the correction unit compares, for each of two class boundaries determined to be inconsistent, a level of similarity between each frame in the extracted time series data in which the class boundary was detected and the reference time series data corresponding to a type of the class boundary, between a portion before the class boundary and a portion after the class boundary, and determines that the class boundary is erroneous if a difference therebetween is equal to or greater than a threshold.

7. The class boundary detection device according to claim 6 , wherein the correction unit corrects a class boundary determined to be erroneous so as to match a class boundary determined to be non-erroneous.

8. The correction unit is Regarding the similarity between each frame of the to-be-extracted time-series data in which a class boundary determined to be erroneous has been detected and the reference time-series data corresponding to the type of the class boundary, if the similarity calculated for the portion before the class boundary is lower than the similarity calculated for the portion after the class boundary, calculate the similarity between the to-be-extracted time-series data and the tail portion of the time-series data belonging to each class, and correct the class before the class boundary determined to be erroneous for the class in which the similarity is equal to or greater than a threshold value; 7. The class boundary detection device according to claim 6, wherein, with regard to similarities between each frame of the to-be-extracted time series data in which a class boundary determined to be erroneous has been detected and the reference time series data corresponding to the type of the class boundary, when a similarity calculated for a portion after the class boundary is lower than a similarity calculated for a portion before the class boundary, a similarity between the to-be-extracted time series data and a leading portion of time series data belonging to each class is calculated, and a class after the class boundary determined to be erroneous is corrected for a class in which the similarity is equal to or greater than a threshold value.

9. 1. A computer-implemented control method comprising: An acquisition step of acquiring target time series data which is time series data to be analyzed; a calculation step of extracting a plurality of pieces of extracted time series data from the target time series data, and calculating a similarity between each of the pieces of extracted time series data and reference time series data that represents a class boundary between two mutually different classes; a detection step of detecting a class boundary represented by the reference time series data from the extracted time series data in which the calculated similarity is equal to or greater than a threshold value; A control method, wherein the reference time series data has a leading portion of time series data belonging to a succeeding class indicated by the class boundary, followed by a trailing portion of time series data belonging to a preceding class indicated by the class boundary.

10. An acquisition step of acquiring target time series data which is time series data to be analyzed; a calculation step of extracting a plurality of pieces of extracted time series data from the target time series data, and calculating a similarity between each of the pieces of extracted time series data and reference time series data that represents a class boundary between two mutually different classes; a detection step of detecting a class boundary represented by the reference time series data from the extracted time series data in which the calculated similarity is equal to or greater than a threshold value; The program, wherein the reference time series data has a tail portion of time series data belonging to a previous class indicated by the class boundary, followed by a head portion of time series data belonging to a subsequent class indicated by the class boundary.

Images