Data extension device, data extension method, and program

Abstract

A data augmentation device (2000) acquires source time-sequence data (10). The data augmentation device (2000) generates augmented time-sequence data (30) by performing processing on at least one set of target time-sequence data included in the source time-sequence data (10). The target time-sequence data comprises a plurality of frames that mutually belong to a single class. The processing includes: deletion processing to delete the target time-sequence data; length-changing processing to change the length of the target time-series data; or position-changing processing to change the position of the target time-sequence data on a time axis.

Description

[Technical Field]

[0001] The present disclosure relates to data augmentation of time series data. [Background technology]

[0002] Systems have been developed that generate new data by processing data, i.e., perform data augmentation. For example, Non-Patent Document 1 discloses a technology for increasing training data by performing data augmentation on video data prepared as training data in order to train a discrimination model that performs class discrimination on input video data. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Taeoh Kim, Hyeongmin Lee, MyeongAh Cho, Ho Seong Lee, Dong Heon Cho, and Sangyoun Lee, "Learning Temporarily Invariant and Localizable Features via Data Augmentation for Video Recognition," [online], August 13, 2020, arXiv.org, [Retrieved January 13, 2022], Internet,<URL: https: / / arxiv.org / pdf / 2008.05721.pdf> Summary of the Invention [Problem to be solved by the invention]

[0004] Non-Patent Document 1 is based on the premise that class identification is performed on all video data input to the model (in other words, one class is assigned to all video data input to the model). The present disclosure has been made in light of this problem, and one of its purposes is to provide a new technology for data augmentation of time-series data. [Means for solving the problem]

[0005] The data extension device of the present disclosure includes an acquisition unit for acquiring source time series data consisting of a plurality of frames in a time series, and an extension processing unit for executing a deletion process for deleting one or more target time series data consisting of a plurality of consecutive frames belonging to the same class in the source time series data, a length change process for changing the length of the target time series data, or a position change process for changing the position of the target time series data on a time axis, thereby generating extended time series data from the source time series data. The source time series data includes a plurality of frames belonging to different classes.

[0006] A data augmentation method according to the present disclosure is executed by a computer. The method includes an acquisition step of acquiring source time series data consisting of multiple frames in a time series, and an augmentation step of performing a deletion process of deleting one or more target time series data consisting of multiple consecutive frames belonging to the same class from the source time series data, a length modification process of changing the length of the target time series data, or a position modification process of changing the position of the target time series data on a time axis, thereby generating augmented time series data from the source time series data. The source time series data includes multiple frames belonging to different classes.

[0007] The non-transitory computer-readable medium of the present disclosure stores a program that causes a computer to execute the data expansion method of the present disclosure. [Effects of the Invention]

[0008] According to the present disclosure, a new technique for data augmentation of time-series data is provided. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating source time series data 10 handled by a data extension device according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating an example of an outline of the operation of a data expansion device according to an embodiment. [Figure 3] FIG. 2 is a block diagram illustrating a functional configuration of a data extension device. [Figure 4] FIG. 1 is a block diagram illustrating a hardware configuration of a computer that realizes a data extension device. [Figure 5] 10 is a flowchart illustrating a flow of a process executed by a data extension device. [Figure 6] FIG. 10 is a diagram illustrating class information in a table format. [Figure 7] FIG. 10 is a first diagram illustrating a deletion process. [Figure 8] FIG. 2 is a second diagram illustrating the deletion process. [Figure 9] FIG. 10 is a diagram illustrating a speed reduction process. [Figure 10] 10 is a process illustrating a repeat process. [Figure 11] FIG. 10 is a diagram illustrating an example of a high-speed process. [Figure 12] FIG. 10 is a diagram illustrating a movement process. [Figure 13] FIG. 10 is a diagram illustrating a switch process. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0011] <Summary> 1 is a diagram illustrating source time series data 10 handled by a data extension device according to an embodiment. The source time series data 10 is composed of a plurality of time-series frames 12. In other words, the source time series data 10 can be said to be a data string in which the frames 12 are arranged in time series.

[0012] For example, the source time series data 10 is video data. Video data is time series data in which a plurality of video frames are arranged in order (in ascending order of frame numbers). Therefore, if the source time series data 10 is video data, the frame 12 is a video frame that constitutes the video data.

[0013] Each frame 12 belongs to one of multiple classes. The source time series data 10 includes multiple partial time series data 20. The partial time series data 20 is time series data consisting of multiple consecutive frames 12 that belong to the same class. For example, the source time series data 10 in FIG. 1 includes, in this order, partial time series data 20-1 consisting of multiple frames 12 belonging to class C1, partial time series data 20-2 consisting of multiple frames 12 belonging to class C2, and partial time series data 20-3 consisting of multiple frames 12 belonging to class C3. Hereinafter, time series data consisting of multiple frames 12 belonging to class C will also be referred to as "time series data belonging to class C."

[0014] Here, the source time series data 10 includes at least two partial time series data 20 that belong to different classes. However, the source time series data 10 may also include two or more partial time series data 20 that belong to the same class. For example, in the example of Figure 1, the partial time series data 20-1 and the partial time series data 20-3 may belong to class C1, and the partial time series data 20-2 may belong to class C2.

[0015] A class represents, for example, the content of the partial time series data 20 (e.g., a scene or situation represented by the partial time series data 20). For example, suppose a worker performing work consisting of three processes P1, P2, and P3 is filmed with a video camera, and the video data obtained by the filming is treated as source time series data 10. In this case, each work process can be treated as a class. That is, the source time series data 10 can be divided into three partial time series data 20: partial time series data 20 including the work status of process P1, partial time series data 20 including the work status of process P2, and partial time series data 20 including the work status of process P3.

[0016] The type of source time series data 10 is not limited to video data. For example, the source time series data 10 may be audio data. Alternatively, the source time series data 10 may be sensing data representing the results of repeated sensing by an arbitrary sensor (e.g., a three-dimensional acceleration sensor).

[0017] 2 is a diagram illustrating an example of an outline of the operation of the data extension device 2000 according to the embodiment. Note that FIG. 2 is a diagram for facilitating understanding of the outline of the data extension device 2000, and the operation of the data extension device 2000 is not limited to that shown in FIG.

[0018] The data extension device 2000 processes at least one partial time series data 20 included in the source time series data 10 to generate extended time series data 30 different from the source time series data 10. This achieves data extension. Hereinafter, the partial time series data 20 included in the source time series data 10 that are the target of the processing process will be referred to as "target time series data."

[0019] The processing performed on the source time series data 10 includes, for example, 1) deletion processing, 2) length change processing, or 3) position change processing. Here, "the processing includes 1), 2), or 3") means that the processing may include a combination of two or more of 1) to 3).

[0020] The deletion process is a process of deleting target time series data. In other words, the target time series data that has been deleted will not be included in the extended time series data 30. The length change process is a process of changing the length of the target time series data on the time axis. Therefore, the number of frames 12 included in the processed target time series data will be different from the number of frames 12 included in the unprocessed target time series data. The position change process is a process of changing the position of the target time series data on the time axis. Therefore, the position of the target time series data on the time axis in the extended time series data 30 will be different from the position of the target time series data on the time axis in the source time series data 10. A possible combination of the length change process and the position change process is, for example, "performing a length change process to change the length of one or both of two target time series data, and then performing a position change process to swap their positions."

[0021] Here, when multiple types of processing are performed on the source time series data 10, these multiple types of processing may be performed on the same target time series data or on different target time series data. In the former case, for example, a length change process to change the length of the partial time series data 20-1 is performed, followed by a position change process to change the position of the partial time series data 20-1. In the latter case, for example, a deletion process to delete the partial time series data 20-1 is performed, followed by a position change process to change the position of the partial time series data 20-2.

[0022] <Examples of effects> According to the data extension device 2000 of this embodiment, one or more partial time series data 20 included in the source time series data 10 are processed to generate extended time series data 30. Here, the source time series data 10 includes a plurality of partial time series data 20 each belonging to a different class. Therefore, the data extension device 2000 can generate time series data including a plurality of partial time series data 20 each belonging to a different class through data extension.

[0023] Such data augmentation is useful, for example, for training a classifier that identifies the class of each frame constituting time-series data in response to input time-series data. For example, a classifier that identifies the class of each video frame in response to input video data is conceivable. The training data used to train such a classifier, for example, indicates the time-series data as input data and indicates the class of each frame included in the time-series data as ground-truth data.

[0024] In order to obtain a classifier with high classification accuracy, it is preferable to train the classifier using a large amount of training data. However, preparing a large amount of training data takes time and effort. In this regard, by using the data expansion device 2000 of this embodiment, the amount of training data can be increased by data expansion. Therefore, the time and effort required to prepare training data can be reduced, and a large amount of training data can be prepared more easily.

[0025] Furthermore, in order to obtain a highly accurate classifier, it is preferable to have a large variation of training data. However, when generating training data, there are cases where the variation is prone to bias. For example, such a case can be cited as an example where training data is prepared by observing real-world situations. A more specific example is when video data obtained by capturing daily factory work with a surveillance camera is used as training data.

[0026] When observing actual situations, abnormal situations are less likely to be observed than normal situations. For example, in the case of filming factory work as described above, it is considered that most work is performed according to normal procedures, and work performed according to incorrect procedures is rarely filmed. Therefore, the number of training data representing abnormal situations is smaller than the number of training data representing normal situations. However, in order to increase the variety of training data, it is preferable to have a large number of training data representing abnormal situations.

[0027] In this regard, by using the data extension device 2000, time series data representing a normal situation can be acquired as source time series data 10, and by processing the source time series data, extended time series data 30 representing an abnormal situation can be generated. For example, video data representing an abnormal work situation can be generated from video data recording a normal daily work situation. Therefore, the data extension device 2000 can easily increase the variety of training data.

[0028] The data expansion device 2000 of this embodiment will be described in more detail below.

[0029] <Example of functional configuration> 3 is a block diagram illustrating a functional configuration of a data extension device 2000 according to an embodiment. The data extension device 2000 includes an acquisition unit 2020 and an extension processing unit 2040. The acquisition unit 2020 acquires source time series data 10. The extension processing unit 2040 processes one or more target time series data included in the source time series data 10 to generate extended time series data 30 from the source time series data 10.

[0030] <Example of hardware configuration> Each functional component of the data expansion device 2000 may be realized by hardware (e.g., a hardwired electronic circuit, etc.) that realizes each functional component, or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a case where each functional component of the data expansion device 2000 is realized by a combination of hardware and software will be further described.

[0031] 4 is a block diagram illustrating an example of the hardware configuration of a computer 500 that realizes the data extension device 2000. The computer 500 is any computer. For example, the computer 500 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 500 may be a portable computer such as a smartphone or a tablet terminal. The computer 500 may be a dedicated computer designed to realize the data extension device 2000, or may be a general-purpose computer.

[0032] For example, by installing a predetermined application on the computer 500, each function of the data extension device 2000 is realized on the computer 500. The application is configured with a program for realizing each functional component of the data extension device 2000. The method for acquiring the program is arbitrary. For example, the program can be acquired from a storage medium (such as a DVD disc or USB memory) on which the program is stored. Alternatively, the program can be acquired by downloading the program from a server device that manages the storage device on which the program is stored.

[0033] The computer 500 includes a bus 502, a processor 504, a memory 506, a storage device 508, an input / output interface 510, and a network interface 512. The bus 502 is a data transmission path for the processor 504, the memory 506, the storage device 508, the input / output interface 510, and the network interface 512 to transmit and receive data to and from each other. However, the method for connecting the processor 504 and other components to each other is not limited to a bus connection.

[0034] The processor 504 is a variety of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 506 is a main storage device realized using a random access memory (RAM) or the like. The storage device 508 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read-only memory (ROM) or the like.

[0035] The input / output interface 510 is an interface for connecting the computer 500 with input / output devices. For example, the input / output interface 510 is connected to an input device such as a keyboard and an output device such as a display device.

[0036] The network interface 512 is an interface for connecting the computer 500 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0037] The storage device 508 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the data extension device 2000. The processor 504 reads this program into the memory 506 and executes it, thereby realizing each functional component of the data extension device 2000.

[0038] The data extension device 2000 may be realized by one computer 500 or by multiple computers 500. In the latter case, the configurations of the computers 500 do not need to be the same, and can be different from each other.

[0039] <Processing flow> 5 is a flowchart illustrating a process executed by the data extension device 2000 according to the embodiment. The acquisition unit 2020 acquires source time series data 10 (S102). The extension processing unit 2040 processes one or more target time series data to generate extended time series data 30 (S104).

[0040] <Acquisition of source time series data 10: S102> The acquiring unit 2020 acquires the source time series data 10. Here, various methods can be adopted to acquire the time series data to be processed. For example, the source time series data 10 is assumed to be stored in advance in an arbitrary storage device in a format that allows it to be acquired from the data extension device 2000. In this case, the acquiring unit 2020 acquires the source time series data 10 by reading the source time series data 10 from the storage device.

[0041] Alternatively, for example, the acquiring unit 2020 acquires the source time series data 10 by receiving the source time series data 10 transmitted from another device. The device transmitting the source time series data 10 is, for example, the device that generated the source time series data 10. If the source time series data 10 is video data, for example, the acquiring unit 2020 acquires the source time series data 10 from the video camera that generated the source time series data 10.

[0042] <<Information to identify the class of frame 12>> The data extension device 2000 needs to be able to identify the class to which each piece of partial time series data 20 belongs. Therefore, for example, the data extension device 2000 acquires information indicating the class to which each piece of partial time series data 20 belongs (hereinafter, class information).

[0043] For example, the class information indicates the correspondence between the identification information (e.g., frame number) of each frame 12 included in the source time series data 10 and the identification information of the class to which the frame 12 belongs. Alternatively, for each partial time series data 20 included in the source time series data 10, the class information may indicate the identification information of either or both of the first frame 12 and the last frame 12.

[0044] 6 is a diagram illustrating an example of class information in table format. Table 200 indicates, for each frame 12, the class to which that frame 12 belongs. More specifically, the table indicates, in association with the identification information of the frame 12 (frame identification information 202), the identification information of the class to which that frame 12 belongs (class identification information 204).

[0045] On the other hand, the table 300 indicates the class to which each piece of partial time series data 20 belongs, for each piece of partial time series data 20. More specifically, for each piece of partial time series data 20, the table 300 indicates the identification information (class identification information 306) of the class to which the piece of partial time series data 20 belongs, in association with a combination of the identification information (first frame identification information 302) of the first frame 12 and the identification information (last frame identification information 304) of the last frame 12.

[0046] The class information may be information integrated with the source time series data 10, or may be information separate from the source time series data 10. In the former case, for example, identification information of the class to which each frame 12 belongs is added as metadata to each frame 12 included in the source time series data 10. When the source time series data 10 and the class information are configured separately, for example, the acquiring unit 2020 acquires the class information for the source time series data 10 in addition to the source time series data 10. The method of acquiring the class information is the same as the method of acquiring the source time series data 10.

[0047] <Processing: S104> The extension processing unit 2040 performs processing on one or more pieces of target time series data (S104). The processing includes 1) deletion processing, 2) length change processing, or 3) position change processing. Each of 1) to 3) will be explained below. Note that a method for determining the processing processing to be performed and a method for determining the target time series data to be processed will be described later.

[0048] <<Deletion process>> The deletion process is a process for generating extended time series data 30 by removing target time series data from source time series data 10. FIG. 7 is a first diagram illustrating the deletion process. In FIG. 7, partial time series data 20-2 is treated as target time series data. Therefore, the extended time series data 30 is generated by removing the partial time series data 20-2 from the source time series data 10.

[0049] After removing the target time series data, the extension processing unit 2040 may perform processing to overlap portions of the partial time series data 20 before and after the target time series data. FIG. 8 is a second diagram illustrating the deletion processing. In FIG. 8, the partial time series data 20-2 has been deleted, as in FIG. 7. Furthermore, in FIG. 8, the last m frames of the partial time series data 20-1 that preceded the partial time series data 20-2 and the first m frames of the partial time series data 20-3 that followed the partial time series data 20-2 are overlapped with each other. In this way, the two partial time series data 20 that were originally separated in time are connected relatively naturally.

[0050] When the above-described overlapping is performed, the tail of the front partial time series data 20 may be faded out and the head of the rear partial time series data 20 may be faded in. This makes the connection between these two partial time series data 20 more natural.

[0051] The extension processing unit 2040 generates the class information of the extended time series data 30 based on the class information of the source time series data 10 and the processing operations performed on the source time series data 10. When deletion processing is performed, for example, the extension processing unit 2040 generates the class information of the extended time series data 30 by deleting the information related to the deleted partial time series data 20 from the class information of the source time series data 10. When the class information is added to the frame 12, the class information is also deleted simultaneously by deleting the partial time series data 20.

[0052] Here, when the processing of overlapping the partial time series data 20 before and after the deleted partial time series data 20 is performed, the extension processing unit 2040 determines the class for each newly generated frame 12 by overlapping. For example, the class of the new frame 12 is set to the class of the frame 12 with a higher opacity among the two overlapped frames 12. For example, after setting the frame 12 of class C1 to opacity D1 and the frame 12 of class C2 to opacity D2 and overlapping them. In this case, for example, if D1 >= D2, the class of the new frame 12 is set to C1, and if D1 < D2, the class of the new frame 12 is set to C2.

[0053] <<Length change processing>> The length change processing is a process of changing the length on the time axis of the target time series data. The process of changing the length of the target time series data is roughly classified into a process of lengthening the target time series data and a process of shortening the target time series data. The following will explain each of them.

[0054] <<<Process of lengthening the target time series data>>>> For example, the extension processing unit 2040 lengthens the length of the target time series data by performing a process of slowing down the flow of time (hereinafter referred to as the slow-down processing) on a part or all of the target time series data. For this process, for example, any method used when playing back video or audio at a low speed can be adopted.

[0055] FIG. 9 is a diagram illustrating the slowing process. In the example of FIG. 9, the length of the target time series data 40 is increased by copying each frame 12 included in a time range 50. When the source time series data 10 is video data, the time range 50 is slowed down by the process of FIG. 9. Here, instead of copying each frame 12, the extension processing unit 2040 may insert a frame 12 representing the middle between two adjacent frames 12 (for example, a frame 12 obtained by averaging two frames 12) between the two adjacent frames 12. This process can be realized by frame interpolation.

[0056] 9, one copy of each frame 12 is generated, but the number of copies is not limited to 1. The number of copies may be predetermined or may be determined dynamically (for example, randomly).

[0057] There are various methods for determining the time range 50 to be subjected to the speed reduction process. For example, the extension processing unit 2040 determines the time range 50 by randomly selecting a frame 12 to be the start point of the time range 50 from the target time series data 40. The length of the time range 50 may be predetermined or may be determined dynamically (for example, randomly). The start point of the time range 50 may be selected based on a predetermined rule. For example, when multiple pieces of extended time series data 30 are generated, each of which has been subjected to a speed reduction process with a different pattern, a conceivable method is to shift the start point of the time range 50 by a predetermined number of frames.

[0058] Alternatively, for example, the extension processing unit 2040 may identify a time range in which there is a large change between frames 12 from within the target time series data 40, and perform the slowing process on that time range. This is because there is a high probability that the content of time series data in a time range in which there is a large change between frames 12 will not become unnatural even if the magnitude of the time change is reduced to some extent by the slowing process.

[0059] Between frame 12 of change The magnitude can be calculated based on the difference between frames 12. When the source time series data 10 is video data, the magnitude of change between frames 12 can be expressed, for example, by a statistical value (average, maximum, or median) of the magnitude of optical flow. More specifically, for example, the extension processing unit 2040 divides the target time series data 40 into time ranges of a predetermined length, and calculates, for each time range, a statistical value of the magnitude of change for multiple frames 12 included therein. Then, the extension processing unit 2040 treats a time range in which the statistical value of the magnitude of change is equal to or greater than a threshold as a time range 50 to be subjected to the speed reduction process.

[0060] Instead of performing the slowing process on the time range 50, the slowing process may be performed by selecting one or more frames 12 that are located apart from each other and generating copies of each of the selected frames 12. In this case, for example, the extension processing unit 2040 randomly selects frames 12 from the target time series data 40 to be subjected to the slowing process. Alternatively, for example, the extension processing unit 2040 may individually select frames 12 from the target time series data 40 whose magnitude of change from the previous frame 12 is equal to or greater than a threshold, to be subjected to the slowing process.

[0061] The process of lengthening the target time series data 40 is not limited to the slowing process. For example, the extension processing unit 2040 may lengthen the target time series data 40 by performing a process (hereinafter referred to as a repeat process) on the target time series data 40 such that part or all of the data is repeated. Fig. 10 illustrates a process of repeat process. In the example of Fig. 10, the repeat process is performed so that a portion of a time range 60 is repeated twice.

[0062] There are various methods for selecting a target for repeat processing. For example, the extension processing unit 2040 randomly selects a time range from the target time series data 40 and performs repeat processing on that time range. The length of the selected time range may be predetermined or may be determined dynamically (for example, randomly). In addition, when multiple pieces of extended time series data 30 are generated by performing repeat processing with different patterns, a method may be considered in which the length of the selected time range is increased by a predetermined number.

[0063] Alternatively, for example, the extension processing unit 2040 may detect a time range having a specific meaning from the target time series data 40 and perform repeat processing for that time range. When the target time series data 40 is video data, for example, the extension processing unit 2040 may detect a time range representing a series of changes (e.g., an action) from the target time series data 40 and perform repeat processing for that time range. A series of changes may be periodic changes such as tightening a screw or turning a lever. Alternatively, for example, a series of changes may be non-periodic changes such as opening a door or lowering a lever. By repeating such a series of changes, it is possible to generate extended time series data 30 that represents a more natural situation than when the target of repeat processing is randomly determined.

[0064] There are various methods for detecting a series of changes in the target time series data 40. When the source time series data 10 is video data, for example, the extension processing unit 2040 performs a skeleton detection process on each frame 12 of the target time series data 40 to detect a time range in which the skeleton position periodically changes from the target time series data 40. The extension processing unit 2040 then detects the detected time range as a time range representing a series of changes.

[0065] In another example, when the source time series data 10 is signal data such as audio data, the extension processing unit 2040 detects a time range in which the signal strength, etc. changes periodically from the target time series data 40, and detects the detected time range as a time range representing a series of changes.

[0066] Alternatively, for example, the extension processing unit 2040 may divide the target time series data 40 into time ranges of a predetermined length and determine whether a specific event has occurred within each time range. This allows a time range in which a specific event has occurred to be detected from the target time series data 40. Existing technology can be used to determine whether a specific event is represented by the time series data. For example, if the source time series data 10 is video data, the event is a specific action performed by a person.

[0067] <<<Process to shorten the target time series data>>> For example, the extension processing unit 2040 performs a process of speeding up the flow of time (hereinafter referred to as a speed-up process) on a part or all of the target time series data 40, thereby shortening the length of the target time series data 40. For this process, any method used for high-speed playback of video or audio can be adopted.

[0068] FIG. 11 is a diagram illustrating the speed-up process. In the example of FIG. 11, one out of every two frames 12 included in a time range 70 of the target time series data 40 is deleted, thereby shortening the length of the target time series data. This can also be expressed as "frames 12 are thinned out for the time range 70." Note that the thinning rate is not limited to "one out of every two," but can be generalized as "B out of A" (A>B). Note that the values ​​of A and B may be predetermined or may be determined dynamically (for example, randomly).

[0069] The extension processing unit 2040 may shorten the length of the target time series data 40 by deleting the entire time range 70 instead of thinning out the time range 70. In this case, similar to the deletion process described above, the extension processing unit 2040 may overlap time ranges of a predetermined length before and after the time range 70.

[0070] The method for determining the time range 70 to be subjected to the speed-up processing can be the same as the method for determining the time range 50 to be subjected to the speed-up processing. However, when focusing on the magnitude of change between frames 12, it is preferable for the extension processing unit 2040 to subject the range in which the magnitude of change is small to the speed-up processing. This is because time series data in which the magnitude of change between frames 12 is small is likely to not appear unnatural even if the magnitude of change is increased to some extent by the speed-up processing. For example, the extension processing unit 2040 divides the target time series data 40 into time ranges of a predetermined length and calculates, for each time range, statistics of the magnitude of change for multiple frames 12 included within that time range. Then, the extension processing unit 2040 treats the range in which the statistical value of the magnitude of change is equal to or less than a threshold as the time range 70 to be subjected to the speed-up processing.

[0071] Instead of performing the speed-up process on the time range 70, the speed-up process may be performed by selecting one or more frames 12 that are located apart from each other and deleting the selected frames 12. In this case, for example, the extension processing unit 2040 randomly selects frames 12 from the target time series data 40 and sets them as targets for the speed-up process. Alternatively, for example, the extension processing unit 2040 may individually select frames 12 from the target time series data 40 whose change from the previous frame 12 is equal to or less than a threshold and set them as targets for the speed-up process.

[0072] The extension processing unit 2040 generates class information of the extended time series data 30 based on the class information of the source time series data 10 and the content of the length change processing performed on the source time series data 10. When the processing of lengthening the target time series data 40 is performed by copying a frame 12 included in the target time series data 40, the extension processing unit 2040 generates class information of the extended time series data 30 so that the class of the newly generated frame 12 is the same as the class of the frame 12 from which it was copied. On the other hand, when the processing of shortening the target time series data 40 is performed by deleting a frame 12 included in the target time series data 40, the extension processing unit 2040 generates class information of the extended time series data 30 by deleting the class information of the deleted frame 12 from the class information of the source time series data 10. Note that when a new frame 12 is generated by superimposition, the extension processing unit 2040 assigns a class to the frame 12. The method of assigning a class to the frame 12 generated by superimposition is as described in the deletion processing.

[0073] <<Position change processing>> The position change process is a process of changing the position of the partial time series data 20 in the target time series data 40. The position change process can be broadly divided into 1) a movement process of moving one partial time series data 20 to another position, and 2) a switch process of exchanging the positions of two partial time series data 20.

[0074] Fig. 12 is a diagram illustrating the movement process. In the example of Fig. 12, the partial time series data 20-2 is moved to after the partial time series data 20-4. In other words, the position of the partial time series data 20-2 is changed to a position after the partial time series data 20-4.

[0075] 13 is a diagram illustrating the switching process, in which the positions of the partial time series data 20-2 and the partial time series data 20-4 are swapped.

[0076] Here, the position change process may include a process of overlapping the beginning of the moved partial time series data 20 with the end of the preceding partial time series data 20, or a process of overlapping the end of the moved partial time series data 20 with the beginning of the succeeding partial time series data 20. For example, in the example of Fig. 12, a process of overlapping the beginning of partial time series data 20-2 with the end of partial time series data 20-4, or a process of overlapping the end of partial time series data 20-2 with the beginning of partial time series data 20-5 may be performed.

[0077] Furthermore, a process may be performed to overlap the tails and heads of partial time series data 20 that are adjacent to each other as a result of moving other partial time series data 20. For example, in the example of Fig. 12, the partial time series data 20-1 and the partial time series data 20-3 are adjacent to each other as a result of moving the partial time series data 20-2. Therefore, a process may be performed to overlap the tail of the partial time series data 20-1 with the head of the partial time series data 20-3.

[0078] Here, the method for overlapping the ends and heads of adjacent partial time series data 20 is as explained in the deletion process.

[0079] The extension processing unit 2040 generates class information of the extended time series data 30 based on the class information of the source time series data 10 and the content of the position change processing executed on the source time series data 10. Specifically, for the partial time series data 20 whose position has been changed between the source time series data 10 and the extended time series data 30, the position is also changed in the class information. Furthermore, a class is assigned to the frame 12 generated by superposition. The method of assigning a class to the frame 12 generated by superposition is as described in the deletion processing.

[0080] <<Selection of processing method>> The processing performed on the source time series data 10 may be predetermined or may be arbitrarily selected from a plurality of types. The number of processing performed on the source time series data 10 may be one or more. The type and number of processing may be selected randomly or according to some rule. When a plurality of extended time series data 30 are generated from the source time series data 10, the processing is selected in order, for example, according to a predetermined order.

[0081] Similarly, the target time series data to be processed may be predetermined or may be selected arbitrarily.

[0082] <<<Using Hint Information>>> Assume that time series data that reproduces a situation with a high occurrence probability is required as the extended time series data 30. In this case, the data extension device 2000 may obtain hint information that indicates the type of situation with a high occurrence probability and the target behavior of the situation, and may use the hint information to determine the type and target of processing.

[0083] For example, suppose that it is required to generate extended time series data 30, which is video data representing work performed according to an incorrect procedure, from source time series data 10, which is video data generated by filming work performed according to a correct procedure. Here, in an actual work site, some work errors occur frequently, while others rarely occur. Therefore, for example, the data extension device 2000 acquires, as hint information, information indicating frequently occurring work errors and the work classes in which the errors occur. By using this hint information, the data extension device 2000 can easily generate extended time series data 30 representing frequently occurring work errors from the source time series data 10 representing normal work.

[0084] The types of situations indicated by the hint information are various. For example, the situation indicated by the hint information is an absence of an action (forgetting to perform a task). In this case, the hint information indicates a class corresponding to an action that is likely to be absent (for example, a class of tasks that are likely to be forgotten). When hint information indicating such information is acquired, the extension processing unit 2040 detects, from the source time series data 10, partial time series data 20 that belong to the class indicated in the hint information, and performs a deletion process on the partial time series data 20.

[0085] Another example of a situation indicated by the hint information is a situation in which the actual duration of an action deviates from the ideal duration (e.g., a delay in work). In this case, the hint information indicates a class corresponding to an action whose actual duration is likely to deviate from the ideal duration (e.g., a class of work that is likely to be delayed). When hint information indicating such information is acquired, the extension processing unit 2040 detects, from the source time series data 10, partial time series data 20 belonging to the class indicated in the hint information, and performs length modification processing on the partial time series data 20. Preferably, the hint information further includes information indicating whether the duration should be increased or decreased.

[0086] Another example of a situation indicated by the hint information is a situation in which the actual time position of an action deviates from the ideal time position (e.g., switching two tasks). In this case, the hint information indicates a class representing an action whose actual time position is likely to deviate from the ideal time position (e.g., each class of two actions that are likely to be erroneously switched). When hint information indicating such information is acquired, the extension processing unit 2040 detects, from the source time series data 10, partial time series data 20 belonging to the class indicated in the hint information, and performs a position change process on the partial time series data 20. Note that, when the hint information indicates each class of two tasks that are likely to be switched, the extension processing unit 2040 detects, from the source time series data 10, partial time series data 20 corresponding to each of the two classes, and performs a switch process on the detected two partial time series data 20.

[0087] When hint information is used, the obtaining unit 2020 further obtains the hint information in the same manner as the method for obtaining the source time series data 10.

[0088] Here, the hint information may be general-purpose information that does not depend on the type of source time series data 10, or may be information that depends on the type of source time series data 10 (in other words, information specialized for a specific type of source time series data 10). An example of the latter is a case where hint information is prepared for each work line at a work site, since the series of work steps performed on each work line differs and the work errors that are likely to occur also differ.

[0089] When hint information is prepared for each type of source time series data 10, the acquiring unit 2020 further acquires type information indicating the type of source time series data 10. Then, the acquiring unit 2020 acquires hint information corresponding to the type of source time series data 10 indicated in the type information.

[0090] Here, when generating multiple extended time series data 30 from one source time series data 10, the extension processing unit 2040 may generate both extended time series data 30 using hint information and extended time series data 30 without using hint information. However, it is preferable to make the number of extended time series data 30 generated using hint information larger than the number of extended time series data 30 generated without using hint information.

[0091] For example, the extension processing unit 2040 is configured to select whether or not to use hint information when generating each extended time-series data 30. In this case, the probability that the option "use hint information" is selected is set to be higher than the probability that the option "do not use hint information" is selected.

[0092] The hint information may also indicate the occurrence frequency (occurrence probability) of each event. In this case, the extension processing unit 2040 may increase the generation probability of the extended time-series data 30 corresponding to an event with a higher occurrence frequency.

[0093] <Result output> The data extension device 2000 outputs the execution result. Hereinafter, information output from the data extension device 2000 is referred to as output information. The output information includes the extended time series data 30. When the extended time series data 30 and the corresponding class information are configured separately, the output information further includes the class information of the extended time series data 30. Here, when multiple pieces of extended time series data 30 are generated, the output information includes multiple combinations of the extended time series data 30 and the class information.

[0094] The output information may be output in any manner. For example, the data extension device 2000 stores the output information in any storage device. Alternatively, the data extension device 2000 may transmit the output information to any device. For example, the destination device may be a device that uses the extended time series data 30 to train a classifier that identifies the class of each frame included in the time series data.

[0095] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0096] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0097] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. (Appendix 1) acquiring means for acquiring source time series data consisting of a plurality of frames in time series; and an extension processing means for generating extended time series data from the source time series data by executing a deletion process for deleting the target time series data, a length change process for changing the length of the target time series data, or a position change process for changing the position of the target time series data on a time axis, for one or more target time series data that are in the source time series data and are composed of a plurality of consecutive frames that belong to the same class, The source time series data includes a plurality of frames that belong to different classes. (Appendix 2) The length change process includes: A process of lengthening the target time series data by copying one or more frames included in the target time series data, or 2. The data extension device according to claim 1, wherein the processing shortens the target time series data by deleting one or more frames included in the target time series data. (Appendix 3) The data extension device according to claim 2, wherein the length change processing is a processing for lengthening the target time series data by repeating a part or all of the target time series data in the extended time series data. (Appendix 4) The data extension device described in Appendix 3, wherein the length changing process is a process of detecting a time range whose content periodically changes from the target time series data, and lengthening the target time series data by repeating the sequence of frames included in the detected time range in the extended time series data. (Appendix 5) 2. The data extension device according to claim 1, wherein the position change processing is processing for changing the position of the target time series data by swapping the positions of two of the target time series data. (Appendix 6) the acquiring means acquires hint information representing situations with a high occurrence probability and classes of actions that are targets of the situations; The data extension device according to any one of appendices 1 to 5, wherein the extension processing means generates the extended time series data by performing processing on the target time series data composed of the frames belonging to the class indicated in the hint information so as to reproduce the situation indicated in the hint information. (Appendix 7) The hint information indicates a class of an action in a situation where an action is missing, and 7. The data extension device according to claim 6, wherein the extension processing means generates the extended time series data by performing the deletion processing on the target time series data composed of the frames belonging to a class indicated in the hint information. (Appendix 8) the hint information indicates a class of the action in a situation where the actual duration of the action deviates from the ideal duration; 7. The data extension device according to claim 6, wherein the extension processing means generates the extended time series data by performing the length change processing on the target time series data composed of the frames belonging to a class indicated in the hint information. (Appendix 9) the hint information indicates a class of the action in a situation where an actual time position of the action deviates from an ideal time position of the action; 7. The data extension device according to claim 6, wherein the extension processing means generates the extended time series data by performing the position change processing on the target time series data composed of the frames belonging to a class indicated in the hint information. (Appendix 10) the source time series data is video data; The data expansion device according to any one of Supplementary Notes 1 to 9, wherein the class to which the frame belongs represents the type of work captured in the frame. (Appendix 11) 1. A computer-implemented method for data augmentation, comprising: an acquisition step of acquiring source time series data consisting of a plurality of frames in time series; an extension processing step of generating extended time series data from the source time series data by performing a deletion process of deleting the target time series data, a length change process of changing the length of the target time series data, or a position change process of changing the position of the target time series data on a time axis, for one or more target time series data that are in the source time series data and are composed of a plurality of consecutive frames that belong to the same class, 10. A data augmentation method, wherein the source time series data includes a plurality of frames that belong to different classes. (Appendix 12) The length change process includes: A process of lengthening the target time series data by copying one or more frames included in the target time series data, or 12. The data extension method according to claim 11, wherein the data extension method is a process of shortening the target time series data by deleting one or more frames included in the target time series data. (Appendix 13) 13. The data extension method according to claim 12, wherein the length change processing is a processing for lengthening the target time series data by repeating part or all of the target time series data in the extended time series data. (Appendix 14) The data extension method according to claim 13, wherein the length change process is a process of detecting a time range in the target time series data whose content changes periodically, and lengthening the target time series data by repeating the sequence of frames included in the detected time range in the extended time series data. (Appendix 15) 12. The data augmentation method according to claim 11, wherein the position change processing is processing for changing the position of the target time series data by swapping the positions of two of the target time series data. (Appendix 16) In the obtaining step, hint information is obtained that indicates a situation with a high occurrence probability and a class of an action that is a target of the situation; 16. The data extension method according to any one of appendices 11 to 15, wherein in the extension processing step, the target time series data consisting of the frames belonging to a class indicated in the hint information is subjected to processing that can reproduce a situation indicated in the hint information, thereby generating the extended time series data. (Appendix 17) The hint information indicates a class of an action in a situation where an action is missing, and 17. The data extension method according to claim 16, wherein in the extension processing step, the deletion processing is performed on the target time series data constituted by the frames belonging to a class indicated in the hint information, thereby generating the extended time series data. (Appendix 18) the hint information indicates a class of the action in a situation where the actual duration of the action deviates from the ideal duration; 17. The data extension method according to claim 16, wherein in the extension processing step, the target time series data consisting of the frames belonging to a class indicated in the hint information is subjected to the length change processing to generate the extended time series data. (Appendix 19) the hint information indicates a class of the action in a situation where an actual time position of the action deviates from an ideal time position of the action; 17. The data extension method according to claim 16, wherein in the extension processing step, the position change processing is performed on the target time series data composed of the frames belonging to a class indicated in the hint information, thereby generating the extended time series data. (Appendix 20) the source time series data is video data; 20. The data augmentation method according to any one of appendices 11 to 19, wherein the class to which the frame belongs represents the type of work captured in the frame. (Appendix 21) an acquisition step of acquiring source time series data consisting of a plurality of frames in time series; and an extension processing step of generating extended time series data from the source time series data by performing a deletion process of deleting the target time series data, a length change process of changing the length of the target time series data, or a position change process of changing the position of the target time series data on a time axis, for one or more target time series data that are in the source time series data and are composed of a plurality of consecutive frames that belong to the same class, A non-transitory computer-readable medium, wherein the source time-series data includes a plurality of frames belonging to different classes. (Appendix 22) The length change process includes: A process of lengthening the target time series data by copying one or more frames included in the target time series data, or 22. The computer-readable medium of claim 21, wherein the target time series data is shortened by deleting one or more frames included in the target time series data. (Appendix 23) 23. The computer-readable medium of claim 22, wherein the length change process is a process of lengthening the target time series data by repeating part or all of the target time series data in the extended time series data. (Appendix 24) 24. The computer-readable medium of claim 23, wherein the length changing process is a process of detecting a time range in the target time series data whose content changes periodically, and lengthening the target time series data by repeating the sequence of frames included in the detected time range in the extended time series data. (Appendix 25) 22. The computer-readable medium of claim 21, wherein the position change processing is processing for changing the position of the target time series data by swapping the positions of two of the target time series data. (Appendix 26) In the obtaining step, hint information is obtained that indicates a situation with a high occurrence probability and a class of an action that is a target of the situation; 26. The computer-readable medium of claim 21, wherein in the extension processing step, the target time series data consisting of the frames belonging to the class indicated in the hint information is subjected to processing that can reproduce the situation indicated in the hint information, thereby generating the extended time series data. (Appendix 27) The hint information indicates a class of an action in a situation where an action is missing, and 27. The computer-readable medium of claim 26, wherein in the extension processing step, the target time series data consisting of the frames belonging to a class indicated in the hint information is subjected to the deletion processing to generate the extended time series data. (Appendix 28) the hint information indicates a class of the action in a situation where the actual duration of the action deviates from the ideal duration; 27. The computer-readable medium of claim 26, wherein in the extension processing step, the target time series data consisting of the frames belonging to a class indicated in the hint information is subjected to the length change processing to generate the extended time series data. (Appendix 29) the hint information indicates a class of the action in a situation where an actual time position of the action deviates from an ideal time position of the action; 27. The computer-readable medium of claim 26, wherein in the extension processing step, the position change processing is performed on the target time series data consisting of the frames belonging to a class indicated in the hint information, thereby generating the extended time series data. (Appendix 30) the source time series data is video data; 30. The computer-readable medium of any one of Clauses 21 to 29, wherein the class to which the frame belongs represents the type of activity being captured in that frame. [Explanation of symbols]

[0098] 10 Source Time Series Data 12 frames 20 Partial time series data 30 Extended Time Series Data 40 Target time series data 50 hour range 60-hour range 70 hour range 200 tables 202 Frame Identification Information 204 Class Identification Information 300 tables 302 First frame identification information 304 Last frame identifier 306 Class Identification Information 500 computers 502 Bus 504 processor 506 memory 508 Storage Devices 510 Input / Output Interface 512 network interface 2000 Data Expansion Device 2020 Acquisition Department 2040 Extended Processing Unit

Claims

1. acquiring means for acquiring source time series data consisting of a plurality of frames in time series; and an extension processing means for generating extended time series data from the source time series data by executing a length change process for changing the length of the target time series data or a position change process for changing the position on a time axis of the target time series data, for one or more target time series data that are in the source time series data and are composed of a plurality of consecutive frames that belong to the same class, The source time series data includes a plurality of frames that belong to different classes.

2. The length change process includes: A process of lengthening the target time series data by copying one or more frames included in the target time series data, or The data extension device according to claim 1 , wherein the processing shortens the target time series data by deleting one or more frames included in the target time series data.

3. The data extension device according to claim 2 , wherein the length change processing is processing for lengthening the target time series data by repeating a part or all of the target time series data in the extended time series data.

4. 4. The data extension device according to claim 3, wherein the length change process is a process of detecting a time range whose content periodically changes from the target time series data, and lengthening the target time series data by repeating the sequence of frames included in the detected time range in the extended time series data.

5. The data extension device according to claim 1 , wherein the position change processing is processing for changing the position of the target time series data by exchanging positions of two of the target time series data.

6. An acquisition means for acquiring source time series data consisting of a plurality of frames in a time series; and an extension processing means for generating extended time series data from the source time series data by executing a deletion process for deleting the target time series data, a length change process for changing the length of the target time series data, or a position change process for changing the position of the target time series data on a time axis, for one or more target time series data that are in the source time series data and are composed of a plurality of consecutive frames that belong to the same class, the source time-series data includes a plurality of frames belonging to different classes; the acquiring means acquires hint information representing situations with a high occurrence probability and classes of actions that are targets of the situations; The extension processing means generates the extended time series data by performing processing on the target time series data composed of the frames belonging to the class indicated in the hint information so as to reproduce the situation indicated in the hint information.

7. The hint information indicates a class of an action in a situation where an action is missing, and 7. The data extension device according to claim 6, wherein the extension processing means generates the extended time series data by performing the deletion process on the target time series data composed of the frames belonging to the class indicated in the hint information.

8. the hint information indicates a class of the action in a situation where the actual duration of the action deviates from the ideal duration; 7. The data extension device according to claim 6, wherein the extension processing means generates the extended time series data by performing the length change processing on the target time series data composed of the frames belonging to a class indicated in the hint information.

9. 1. A computer-implemented method for data augmentation, comprising: an acquisition step of acquiring source time series data consisting of a plurality of frames in time series; an extension processing step of generating extended time series data from the source time series data by performing a length change process for changing the length of the target time series data or a position change process for changing the position of the target time series data on a time axis on one or more target time series data that are included in the source time series data and are composed of a plurality of consecutive frames that belong to the same class, 10. A data augmentation method, wherein the source time series data includes a plurality of frames that belong to different classes.

10. A computer-implemented method for data augmentation, comprising: an acquisition step of acquiring source time series data consisting of a plurality of frames in time series; an extension processing step of generating extended time series data from the source time series data by performing a deletion process of deleting the target time series data, a length change process of changing the length of the target time series data, or a position change process of changing the position of the target time series data on a time axis, for one or more target time series data that are in the source time series data and are composed of a plurality of consecutive frames that belong to the same class; the source time-series data includes a plurality of frames belonging to different classes; In the obtaining step, hint information is obtained that indicates a situation with a high occurrence probability and a class of an action that is a target of the situation; In the extension processing step, the target time series data consisting of the frames belonging to the class indicated in the hint information is processed to reproduce the situation indicated in the hint information, thereby generating the extended time series data.

11. an acquisition step of acquiring source time series data consisting of a plurality of frames in time series; an extension processing step of generating extended time series data from the source time series data by performing a length change process or a position change process on a time axis for one or more target time series data that are in the source time series data and are composed of a plurality of consecutive frames that belong to the same class, and The source time series data includes a plurality of frames that belong to different classes.

12. An acquisition step of acquiring source time series data consisting of a plurality of frames in time series; an extension processing step of generating extended time series data from the source time series data by performing a deletion processing of deleting the target time series data, a length change processing of changing the length of the target time series data, or a position change processing of changing the position of the target time series data on a time axis, for one or more target time series data that are in the source time series data and are composed of a plurality of consecutive frames that belong to the same class; the source time-series data includes a plurality of frames belonging to different classes; In the obtaining step, hint information is obtained that indicates a situation with a high occurrence probability and a class of an action that is a target of the situation; In the extension processing step, the program generates the extended time series data by performing processing on the target time series data composed of the frames belonging to the class indicated in the hint information so as to reproduce the situation indicated in the hint information.

Images