Data augmentation device, data augmentation method, and program

Abstract

This data augmentation device acquires video data including a plurality of frame sequences, and executes, on the video data, either a deletion process to delete a frame sequence or a position modification process to modify a position of a frame sequence. The data augmentation device selects, from among a plurality of editing processes, an editing process to execute on an editing subject that is either: a linking portion between a first subject frame sequence and a second subject frame sequence that have become adjacent to one another due to the deletion process or the position modification process; or before / after the foregoing linking portion. The data augmentation device generates augmented video data resulting from the selected editing process being executed on the editing subject.

Description

Data extension device, data extension method, and program

[0001] The present disclosure relates to data expansion of a sequence of frames.

[0002] Systems have been developed that generate new data by processing data, i.e., perform data augmentation. For example, Non-Patent Document 1 discloses a technology for increasing training data by performing data augmentation on video data prepared as training data in order to train a discrimination model that performs class discrimination on input video data.

[0003] Taeoh Kim, Hyeongmin Lee, MyeongAh Cho, Ho Seong Lee, Dong Heon Cho, and Sangyoun Lee, "Learning Temporarily Invariant and Localizable Features via Data Augmentation for Video Recognition," [online], August 13, 2020, arXiv.org, [Retrieved January 13, 2022], Internet, <URL: https: / / arxiv.org / pdf / 2008.05721.pdf>

[0004] Non-Patent Document 1 assumes that class identification is performed on all video data input to the model (in other words, one class is assigned to all video data input to the model). The present disclosure has been made in light of this problem, and one of its objectives is to provide a new technology for performing data augmentation on a frame sequence.

[0005] The data extension device of the present disclosure comprises an acquisition means for acquiring video data including a plurality of frame sequences each consisting of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes; a processing means for performing a deletion process for deleting one or more of the frame sequences, a position change process for changing the position of one or more of the frame sequences, or both, on the video data; a selection means for selecting from a plurality of editing processes an editing process to be performed on an editing target that is the connecting portion between a first target frame sequence and a second target frame sequence that have become adjacent due to the deletion process or the movement process, or which is before or after said connecting portion; and a generation means for generating extended video data in which the selected editing process has been applied to the video data.

[0006] A data extension method according to the present disclosure, executed by a computer, includes an acquisition step of acquiring video data including a plurality of frame sequences each composed of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes, a processing step of performing a deletion process for deleting one or more of the frame sequences, a position change process for changing the positions of one or more of the frame sequences, or both, on the video data, a selection step of selecting from a plurality of editing processes an editing process to be performed on an editing target that is a connection portion between a first target frame sequence and a second target frame sequence that have become adjacent as a result of the deletion process or the position change process, or a portion before and after the connection portion, and a generation step of generating extended video data in which the selected editing process has been applied to the video data.

[0007] The program of the present disclosure causes a computer to execute the data expansion method of the present disclosure.

[0008] According to the present disclosure, a new technique for data extension of a frame sequence is provided.

[0009] 1 is a diagram illustrating video data handled by a data extension device; FIG. 2 is a diagram illustrating an overview of the operation of a data extension device; FIG. 3 is a block diagram illustrating a functional configuration of a data extension device; FIG. 4 is a block diagram illustrating a hardware configuration of a computer that realizes a data extension device; FIG. 5 is a flowchart illustrating a flow of processing executed by a data extension device; FIG. 6 is a diagram illustrating class information in a table format; FIG. 7 is a diagram illustrating a deletion process; FIG. 8 is a diagram illustrating a movement process; FIG. 9 is a diagram illustrating a switch process; FIG. 10 is a diagram illustrating a blending process in which a blending ratio changes with position; FIG. 11 is a first diagram illustrating a case in which an appropriateness score is calculated based on a movement feature amount; FIG. 12 is a second diagram illustrating a case in which an appropriateness score is calculated based on a movement feature amount; FIG. 13 is a diagram illustrating a method of calculating an appropriateness score for a case in which an editing process is not performed.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0011] [First Embodiment] <Overview> Fig. 1 is a diagram illustrating video data 10 handled by a data expansion device. The video data 10 is made up of a plurality of time-series video frames 12. In other words, the video data 10 is a frame sequence in which a plurality of video frames are arranged in time-series order (in ascending order of frame numbers).

[0012] Each video frame 12 belongs to one of a plurality of classes. The video data 10 includes a plurality of frame sequences 20. A frame sequence 20 is a frame sequence made up of a plurality of consecutive video frames 12 that belong to the same class. For example, the video data 10 in FIG. 1 includes, in this order, a frame sequence 20-1 made up of a plurality of video frames 12 that belong to class C1, a frame sequence 20-2 made up of a plurality of video frames 12 that belong to class C2, and a frame sequence 20-3 made up of a plurality of video frames 12 that belong to class C3. Hereinafter, a frame sequence made up of a plurality of video frames 12 that belong to class C will also be referred to as a "frame sequence belonging to class C."

[0013] Here, the video data 10 includes at least two frame sequences 20 that belong to different classes. Note that the video data 10 may also include two or more frame sequences 20 that belong to the same class. For example, as in the example of Figure 1, in a case where the video data 10 includes three frame sequences 20, frame sequence 20-1 to frame sequence 20-3, frame sequence 20-1 and frame sequence 20-3 may belong to class C1, and frame sequence 20-2 may belong to class C2.

[0014] A class represents, for example, the content of a frame sequence 20 (e.g., a scene or situation represented by the frame sequence 20). For example, suppose a worker performing work consisting of three processes P1, P2, and P3 is filmed with a video camera, and the video data obtained by the filming is treated as video data 10. In this case, each work process can be treated as a class. That is, the video data 10 can be divided into three frame sequences 20: a frame sequence 20 containing the work state of process P1, a frame sequence 20 containing the work state of process P2, and a frame sequence 20 containing the work state of process P3.

[0015] Fig. 2 is a diagram illustrating an outline of the operation of the data expansion device 2000. Here, Fig. 2 is a diagram for facilitating understanding of the outline of the data expansion device 2000, and the operation of the data expansion device 2000 is not limited to that shown in Fig. 2.

[0016] The data extension device 2000 processes at least a portion of the video data 10 to generate extended video data 30 that is different from the video data 10. This achieves data extension. The processing performed on the video data 10 includes 1) a deletion process that deletes at least one frame sequence 20, or 2) a position change process that changes the position of at least one frame sequence 20.

[0017] Here, as a result of performing a deletion process or a position change process (hereinafter referred to as a deletion process, etc.), two frame sequences 20 that were not adjacent to each other in the original video data 10 may become adjacent to each other. For example, assume that the video data 10 includes frame sequence 20-1, frame sequence 20-2, and frame sequence 20-3 in this order. In this case, when a deletion process is performed to delete frame sequence 20-2, frame sequence 20-1 and frame sequence 20-3 become adjacent to each other.

[0018] When frame sequences 20 that were not adjacent to each other in the original video data 10 become adjacent to each other as a result of deletion processing or the like, there is a high probability that the scene represented by the frame sequences will change significantly before and after the connecting portion. For example, if the scene represented by the video data 10 is a human task, the position and posture of body parts such as the human's hands, the position and posture of tools used in the task, or the position and posture of parts of the task being performed may change significantly before and after the connecting portion. Note that the connecting portion between two frame sequences 20 refers to the portion between these two frame sequences 20.

[0019] Therefore, the data expansion device 2000 may perform editing processing on the connecting portion of two frame sequences 20 that have become adjacent as a result of a deletion process or the like. Hereinafter, two frame sequences 20 that have become adjacent as a result of a deletion process or the like will also be referred to as the "editing target." Furthermore, of the two frame sequences 20 included in the editing target, the frame sequence 20 located earlier will also be referred to as the "first target frame sequence," and the frame sequence 20 located later will also be referred to as the "second target frame sequence."

[0020] There are several types of editing processes that the data expansion device 2000 can perform on the editing target. For example, one editing process is a process (hereinafter referred to as a blending process) in which the end of a first target frame sequence and the beginning of a second target frame sequence overlap with each other at a predetermined ratio. Note that the end of the first target frame sequence and the beginning of the second target frame sequence are frame sequences consisting of the same number of frames (one or more).

[0021] Another example of an editing process is an interpolation process in which one or more new frames are inserted between a first target frame sequence and a second target frame sequence, which may be implemented using a trained machine learning model (hereinafter, an interpolation model).

[0022] The data expansion device 2000 selects an editing process to be performed on the editing target from among a plurality of types of editing processes, and generates video data in which the selected editing process has been performed on the editing target included in the video data 10 as expanded video data 30.

[0023] Here, the data expansion device 2000 may select two or more editing processes. For example, suppose the data expansion device 2000 selects two editing processes, editing process E1 and editing process E2. In this case, the data expansion device 2000 can generate two pieces of extended video data 30: extended video data 30 in which the editing target has been subjected to editing process E1, and extended video data 30 in which the editing target has been subjected to editing process E2.

[0024] <Example of Action and Effect> According to the data extension device 2000, extended video data 30 is generated by processing one or more frame sequences 20 included in video data 10. Here, the video data 10 includes a plurality of frame sequences 20 each belonging to a different class. Therefore, according to the data extension device 2000, it is possible to generate a frame sequence including a plurality of frame sequences 20 each belonging to a different class through data extension.

[0025] Such data augmentation is useful, for example, for training a classifier that, in response to an input frame sequence, identifies the class of each video frame that constitutes the frame sequence. For example, a classifier that, in response to an input video data, identifies the class of each video frame. The training data used to train such a classifier may, for example, indicate the frame sequence as input data and indicate the class of each video frame included in the frame sequence as ground-truth data.

[0026] In order to obtain a classifier with high classification accuracy, it is preferable to train the classifier using a large amount of training data. However, preparing a large amount of training data takes time and effort. In this regard, by using the data expansion device 2000, the amount of training data can be increased by data expansion. This reduces the time and effort required to prepare training data, making it easier to prepare a large amount of training data.

[0027] Furthermore, in order to obtain a highly accurate classifier, it is preferable to have a large variation of training data. However, when generating training data, there are cases where the variation is prone to bias. For example, such a case can be cited as an example where training data is prepared by observing real-world situations. A more specific example is when video data obtained by capturing daily factory work with a surveillance camera is used as training data.

[0028] When observing actual situations, abnormal situations are less likely to be observed than normal situations. For example, in the case of filming factory work as described above, it is considered that most work is performed according to normal procedures, and work performed according to incorrect procedures is rarely filmed. Therefore, the number of training data representing abnormal situations is smaller than the number of training data representing normal situations. However, in order to increase the variety of training data, it is preferable to have a large number of training data representing abnormal situations.

[0029] In this regard, by using the data expansion device 2000, a frame sequence representing a normal situation can be acquired as video data 10, and by processing the video data, it is possible to generate expanded video data 30 representing an abnormal situation. For example, video data representing an abnormal work situation can be generated from video data recording normal everyday work situations. Therefore, the data expansion device 2000 can easily increase the variety of training data.

[0030] Furthermore, as described above, there is a high probability that the scene represented by the frame sequences will change significantly before and after the connection portion of two adjacent frame sequences 20 that have become adjacent as a result of deletion processing, etc. Therefore, it is preferable to apply some kind of editing processing to the connection portion or its surroundings to reduce the change in the scene before and after the connection portion, thereby making the scene represented by the extended video data 30 closer to the actual scene.

[0031] In this regard, there are multiple options for editing processes that can be performed on the connecting portion of two frame sequences 20, such as the blending process and interpolation process described above. Furthermore, as will be described later, there are also multiple options for the blending ratio in the blending process and the interpolation algorithm used in the interpolation process. Furthermore, which editing process is appropriate depends on the content of the frame sequences 20, and so it is difficult to determine the best editing process in advance.

[0032] Therefore, the data extension device 2000 dynamically selects an editing process to be applied to the editing target from among multiple editing processes. In this way, an appropriate editing process can be applied to the editing target. As a result, the scene represented by the extended video data 30 can be made to resemble a real scene more closely. In other words, unnaturalness can be reduced in the extended video data 30 generated by data extension. By using such extended video data 30 with less unnaturalness, for example, the classification accuracy of the classifier described above can be improved.

[0033] The data expansion device 2000 will be described in more detail below.

[0034] <Example of Functional Configuration> Fig. 3 is a block diagram illustrating an example of the functional configuration of the data expansion device 2000. The data expansion device 2000 has an acquisition unit 2020, a processing unit 2040, a selection unit 2060, and a generation unit 2080. The acquisition unit 2020 acquires video data 10. The processing unit 2040 performs processing on one or more frame sequences 20 included in the video data 10. The selection unit 2060 selects editing processing to be performed on the editing target. The generation unit 2080 generates extended video data 30 in which editing processing has been performed on the editing target.

[0035] <Example of Hardware Configuration> Each functional component of the data expansion device 2000 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a case where each functional component of the data expansion device 2000 is realized by a combination of hardware and software will be further described.

[0036] 4 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the data expansion device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 1000 may be a portable computer such as a smartphone or a tablet terminal. The computer 1000 may be a dedicated computer designed to realize the data expansion device 2000, or may be a general-purpose computer.

[0037] For example, by installing a predetermined application on the computer 1000, each function of the data expansion device 2000 is realized on the computer 1000. The application is configured with a program for realizing each functional component of the data expansion device 2000. The method for acquiring the program is arbitrary. For example, the program can be acquired from a storage medium (such as a DVD disc or USB memory) on which the program is stored. Alternatively, the program can be acquired by downloading the program from a server device that manages the storage device on which the program is stored.

[0038] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection.

[0039] The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, a read only memory (ROM), or the like.

[0040] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0041] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0042] The storage device 1080 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the data expansion device 2000. The processor 1040 reads this program into the memory 1060 and executes it to realize each functional component of the data expansion device 2000.

[0043] The data expansion device 2000 may be realized by one computer 1000 or by multiple computers 1000. In the latter case, the configurations of the computers 1000 do not need to be the same, and can be different from each other.

[0044] <Processing Flow> Figure 5 is a flowchart illustrating the processing flow executed by the data expansion device 2000. The acquisition unit 2020 acquires video data 10 (S102). The processing unit 2040 performs processing on the video data 10 (S104). The selection unit 2060 selects the editing process to be performed on the editing target (S106). The generation unit 2080 generates extended video data 30 based on the selection result (S108).

[0045] <Acquisition of Video Data 10: S102> The acquisition unit 2020 acquires the video data 10. Here, various methods can be used to acquire the frame sequence to be processed. For example, the video data 10 is assumed to be stored in advance in an arbitrary storage device in a format that allows it to be acquired from the data expansion device 2000. In this case, the acquisition unit 2020 acquires the video data 10 by reading the video data 10 from the storage device.

[0046] Alternatively, for example, the acquisition unit 2020 acquires the video data 10 by receiving the video data 10 transmitted from another device. The device that transmits the video data 10 is, for example, the device that generated the video data 10. If the video data 10 is video data, for example, the acquisition unit 2020 acquires the video data 10 from the video camera that generated the video data 10.

[0047] <<Information for Identifying the Class of Video Frames 12>> The data extension device 2000 needs to be able to identify the class to which each frame sequence 20 belongs. Therefore, for example, the data extension device 2000 acquires information indicating the class to which each frame sequence 20 belongs (hereinafter, class information).

[0048] For example, the class information indicates, for each video frame 12 included in the video data 10, a correspondence between its identification information (e.g., frame number) and the identification information of the class to which the video frame 12 belongs. Alternatively, for each frame sequence 20 included in the video data 10, the class information may indicate the identification information of either or both of the first and last video frames 12.

[0049] 6 is a diagram illustrating an example of class information in table format. Table 200 indicates, for each video frame 12, the class to which that video frame 12 belongs. More specifically, the table indicates, in association with the identification information of the video frame 12 (frame identification information 202), the identification information of the class to which that video frame 12 belongs (class identification information 204).

[0050] On the other hand, the table 300 indicates, for each frame sequence 20, the class to which that frame sequence 20 belongs. More specifically, for each frame sequence 20, the table 300 indicates the identification information of the class to which that frame sequence 20 belongs (class identification information 306) in association with a combination of the identification information of the first video frame 12 (first frame identification information 302) and the identification information of the last video frame 12 (last frame identification information 304).

[0051] The class information may be information that is integrated with the video data 10, or may be information that is separate from the video data 10. In the former case, for example, identification information of the class to which each video frame 12 included in the video data 10 belongs is added as metadata. When the video data 10 and the class information are configured separately, for example, the acquisition unit 2020 further acquires the class information about the video data 10 in addition to the video data 10. The method of acquiring the class information is the same as the method of acquiring the video data 10.

[0052] <Processing: S104> The processing unit 2040 performs one or more processing operations on the video data 10 (S104). As described above, the processing operations include a deletion operation and a position change operation. A deletion operation is a process of removing a target frame sequence 20 from the video data 10.

[0053] 7 is a diagram illustrating the deletion process. In FIG. 7, the processing unit 2040 deletes frame sequences 20-2 and 20-3 from the video data 10. As a result, frame sequences 20-1 and 20-4 become adjacent to each other. Therefore, in this example, the pair of frame sequences 20-1 and 20-4 becomes the editing target. Furthermore, frame sequences 20-1 and 20-4 become the first target frame sequence and the second target frame sequence, respectively.

[0054] The position change process is a process for changing the position of a target frame sequence 20. Here, the position change process includes a movement process for moving one frame sequence 20 to another position, a switch process for exchanging the positions of two frame sequences 20, and the like.

[0055] 8 is a diagram illustrating the movement process. In the example of FIG. 8, frame sequence 20-1 is moved to after frame sequence 20-3. As a result, frame sequence 20-3 and frame sequence 20-1 become adjacent to each other. Also, frame sequence 20-1 and frame sequence 20-4 become adjacent to each other.

[0056] Therefore, in the example of Figure 8, there are two editing targets. The first editing target is a pair of frame sequence 20-3 and frame sequence 20-1. The second editing target is a pair of frame sequence 20-1 and frame sequence 20-4. In the first editing target, the first target frame sequence and the second target frame sequence are frame sequence 20-3 and frame sequence 20-1, respectively. In the second editing target, the first target frame sequence and the second target frame sequence are frame sequence 20-1 and frame sequence 20-4, respectively.

[0057] 9 is a diagram illustrating the switching process. In the example of FIG. 9, the positions of frame sequence 20-1 and frame sequence 20-4 are swapped. As a result, frame sequence 20-4 and frame sequence 20-2 become adjacent to each other. Furthermore, frame sequence 20-3 and frame sequence 20-1 become adjacent to each other. Furthermore, frame sequence 20-1 and frame sequence 20-5 become adjacent to each other.

[0058] 9, there are three editing targets. The first editing target is the pair of frame sequence 20-4 and frame sequence 20-2. The second editing target is the pair of frame sequence 20-3 and frame sequence 20-1. The third editing target is the pair of frame sequence 20-1 and frame sequence 20-5.

[0059] In the first editing target, the first and second target frame sequences are frame sequences 20-4 and 20-2, respectively. In the second editing target, the first and second target frame sequences are frame sequences 20-3 and 20-1, respectively. In the third editing target, the first and second target frame sequences are frame sequences 20-1 and 20-5, respectively.

[0060] <<Selection of Processing>> The type of processing to be performed on the video data 10 may be specified in advance or may be selected arbitrarily. Furthermore, the number of processing operations to be performed on the video data 10 may be one or more. The type and number of processing operations may be selected randomly or according to some rule. When multiple pieces of extended video data 30 are generated from the video data 10, the processing operations are selected in order, for example, according to a predetermined order.

[0061] Similarly, the frame sequence 20 to be processed may be specified in advance or may be selected arbitrarily.

[0062] <Examples of Editing Process> As described above, the editing process is, for example, blending process or interpolation process. Blending process and interpolation process will be described in more detail below.

[0063] <<Blending Process>> Blending process is a process of superimposing the end portion of a first target frame sequence and the beginning portion of a second target frame sequence at a predetermined ratio (hereinafter referred to as the blending ratio). This type of process is also called alpha blending. Here, the process of superimposing two frame sequences at a predetermined blending ratio is achieved by superimposing each pair of video frames that are arranged at the same position on the time axis at the predetermined blending ratio. Note that existing processes can be used for the specific process of generating one video frame by superimposing two video frames at a predetermined ratio.

[0064] A plurality of types of blending processes may be prepared for execution by the selection unit 2060. For example, the plurality of blending processes are defined so that the lengths of the overlapping sections (hereinafter referred to as overlapping sections) are different from each other. For example, in a blending process in which the overlapping section is one second long, the last one second of a frame sequence of a first target frame sequence is overlapped with the first one second of a frame sequence of a second target frame sequence. As another example, the plurality of blending processes may be defined so that the blending ratios are different from each other. As another example, the plurality of blending processes may be defined so that the combinations of overlapping sections and blending ratios are different from each other.

[0065] Here, the blending ratio may be constant regardless of position, or may vary with position. Figure 10 is a diagram illustrating a blending process in which the blending ratio varies with position. In the example of Figure 10, the first target frame sequence 40 and the second target frame sequence 50 are overlaid at a ratio of a:(1-a).

[0066] Here, the value of the proportion a of the first target frame sequence 40 decreases the further back the frame is. Therefore, in the first half of the overlapping section, the proportion of the first target frame sequence 40 at the front increases, while in the second half of the overlapping section, the proportion of the second target frame sequence 50 at the rear increases. This corresponds to fading out the first target frame sequence 40 while fading in the second target frame sequence 50.

[0067] 10, the ratio a decreases in proportion to the position. However, the change in the ratio a does not need to be proportional to the position, and may be a curved change.

[0068] When the blending ratio is changed depending on the position in this way, the plurality of blending processes may be set so that the changes in the blending ratio differ from one another.

[0069] Interpolation is a process of inserting one or more new video frames between the first target frame sequence 40 and the second target frame sequence 50. For example, as described above, the interpolation is performed using an interpolation model.

[0070] The interpolation model is configured to receive two frame sequences as input and generate a frame sequence to be inserted between the two frame sequences. For example, the interpolation model is trained in advance using training data including a combination of an earlier frame sequence, a later frame sequence, and a frame sequence to be inserted between the two frame sequences. Here, various existing configurations can be used for the specific configuration of a machine learning model that performs interpolation by inserting a frame sequence between two frame sequences. Furthermore, various existing training methods can be used for the specific method of training a machine learning model that performs such interpolation.

[0071] A plurality of types of interpolation processes that can be executed by the selection unit 2060 may be prepared. For example, the plurality of interpolation processes may be defined so that different types of machine learning models are used as interpolation models (in other words, the interpolation process algorithms are different from one another). As another example, the plurality of interpolation processes may be defined so that interpolation models trained with different training data are used. As another example, the plurality of interpolation processes may be defined so that the lengths of the frame sequences generated by the interpolation models are different from one another.

[0072] Alternatively, for example, multiple interpolation processes may be defined so that the lengths of the frame sequences used for interpolation are different. For example, if the length of the frame sequence used for interpolation is 30, the interpolation model acquires a frame sequence consisting of 30 video frames from the end of the first target frame sequence 40 as the frame sequence at the end of the first target frame sequence 40. Similarly, the interpolation model acquires a frame sequence consisting of 30 video frames from the beginning of the second target frame sequence 50 as the frame sequence at the beginning of the second target frame sequence 50.

[0073] The length of the end portion of the first target frame sequence 40 and the length of the beginning portion of the second target frame sequence 50 may be defined to be different from each other. For example, suppose the length of the end portion of the first target frame sequence 40 and the length of the beginning portion of the second target frame sequence 50 are defined as 20 and 30, respectively. In this case, the interpolation model acquires 20 video frames from the end of the first target frame sequence 40 as the end portion of the first target frame sequence 40, and acquires 30 video frames of extended video data from the beginning of the second target frame sequence 50 as the beginning portion of the second target frame sequence 50.

[0074] <Selection of Editing Process: S106> The selection unit 2060 selects an editing process to be applied to the editing target from among a plurality of editing processes (S106). An example of a method for selecting an editing process will be described below.

[0075] For example, the selection unit 2060 applies each of a plurality of editing processes to the editing target and determines which editing process to select based on the application results. In this case, the selection unit 2060 calculates, for each editing process, an index value that indicates the degree of appropriateness of the editing target to which the editing process has been applied. Hereinafter, this index value is also referred to as an appropriacy score. The selection unit 2060 selects an editing process based on the appropriateness score calculated for each editing process.

[0076] For example, the selection unit 2060 selects the editing process with the highest appropriateness score. Alternatively, for example, the selection unit 2060 selects the editing process with an appropriateness score equal to or greater than a predetermined threshold.

[0077] Here, there may be multiple editing processes whose appropriateness scores are equal to or greater than the threshold. In this case, the selection unit 2060 may select all of these multiple editing processes, or may select some of the editing processes. In the latter case, for example, the number N (N is 1 or greater) of selectable editing processes is predetermined. The selection unit 2060 selects the top N editing processes in descending order of appropriateness score from among the editing processes whose appropriateness scores are equal to or greater than the threshold.

[0078] Furthermore, there may be a plurality of editing targets, as in the examples of Figures 8 and 9. In this case, the generating unit 2080 selects an editing process for each editing target.

[0079] <<Method of Calculating Appropriateness Score>> For example, the appropriateness score can be expressed as the degree of similarity between the characteristics of object movement in a portion of the editing target that has been edited (hereinafter referred to as the edited portion) and the characteristics of object movement in a portion of the editing target that has not been edited (hereinafter referred to as the non-edited portion). This is because the more similar the characteristics of object movement in the edited portion to the characteristics of object movement in the non-edited portion, the more naturally the scenes before and after the edited portion are connected by the scenes in the edited portion. Hereinafter, an index value that represents the characteristics of object movement represented by a frame sequence (in other words, object movement in the scenes recorded in the frame sequence) is referred to as a motion feature amount.

[0080] 11 is a first diagram illustrating a case where an appropriateness score is calculated based on a motion feature amount, in which a blending process is performed.

[0081] In this example, a frame sequence 60 is generated by applying blending to a first target frame sequence 40 and a second target frame sequence 50. In the first target frame sequence 40, the final portion that is the subject of blending is frame sequence 42-2. In the second target frame sequence 50, the subject of blending is frame sequence 52-1.

[0082] The frame sequence 60 is composed of 1) a frame sequence 60-1 corresponding to the frame sequence 42-1, which is the non-edited portion of the first target frame sequence 40, 2) a frame sequence 60-2 generated by superimposing the frame sequence 42-2 and the frame sequence 52-1, and 3) a frame sequence 60-3 corresponding to the frame sequence 52-2, which is the non-edited portion of the second target frame sequence 50.

[0083] When blending is performed, the edited portion is a frame sequence generated by superimposing. In the example of Figure 11, the edited portion is frame sequence 60-2. The non-edited portion is a portion other than the edited portion. In the example of Figure 11, the non-edited portions are frame sequence 60-1 and frame sequence 60-3.

[0084] The selection unit 2060 calculates a motion feature M1 for the frame sequence 60-2, which is the edited portion. The selection unit 2060 also calculates motion feature values ​​for each of the frame sequences 60-1 and 60-3, which are non-edited portions, and calculates a motion feature M2 from the two calculated motion feature values.

[0085] Here, as will be described in detail later, the motion feature is expressed as a scalar, a distribution, or the like. When the motion feature is expressed as a scalar, for example, the motion feature M2 is expressed as a statistical value (for example, an average value) of the motion feature calculated for the frame sequence 60-1 and the motion feature calculated for the frame sequence 60-3. When the motion feature is expressed as a distribution, for example, the motion feature M2 is expressed as a distribution that combines the distribution calculated for the frame sequence 60-1 and the distribution calculated for the frame sequence 60-3.

[0086] Here, the motion feature M2 calculated for the non-edited portion may be calculated using the entire non-edited portion, or may be calculated using only a portion of the non-edited portion. Figure 11 shows an example of the latter. Specifically, the selection unit 2060 calculates the motion feature M2 from a frame sequence of length L for each of the end of the non-edited portion in the first target frame sequence 40 and the beginning of the non-edited portion in the second target frame sequence 50.

[0087] The selection unit 2060 calculates an appropriateness score S based on the motion feature amounts M1 and M2. For example, when the motion feature amounts are expressed as scalars, the appropriateness score S is expressed as the ratio of the motion feature amount M1 to the motion feature amount M2 (i.e., M1 / M2). However, as will be described later, the way in which the motion feature amounts are expressed is not limited to scalars.

[0088] 12 is a second diagram illustrating a case where the appropriateness score is calculated based on the motion feature amount, in which an interpolation process is performed.

[0089] In this example, a frame sequence 70 is generated by applying interpolation processing to a first target frame sequence 40 and a second target frame sequence 50. The frame sequence 70 is made up of 1) a frame sequence 70-1 that corresponds to the entire first target frame sequence 40, 2) a frame sequence 60-2 that is a frame sequence generated by the interpolation processing, and 3) a frame sequence 70-3 that corresponds to the entire second target frame sequence 50.

[0090] When interpolation processing is performed, the edited portion is a frame sequence generated by the interpolation processing. In the example of Figure 12, the edited portion is frame sequence 70-2. The non-edited portion is a portion other than the edited portion. In the example of Figure 12, the non-edited portions are frame sequence 70-1 and frame sequence 70-3.

[0091] The selection unit 2060 calculates a motion feature M1 for the frame sequence 70-2, which is the edited portion. The selection unit 2060 also calculates a motion feature for each of the frame sequences 70-1 and 70-3, which are non-edited portions, and calculates a motion feature M2 from the two calculated motion feature values. The selection unit 2060 then calculates the appropriateness score S based on the motion feature values ​​M1 and M2. The method for calculating the motion feature M2 from the motion feature values ​​calculated for each of the frame sequences 70-1 and 70-3 is the same as the method for calculating the motion feature M2 from the motion feature values ​​calculated for each of the frame sequences 60-1 and 60-3. In the example of FIG. 12 , as in the example of FIG. 11 , the motion feature M2 is calculated using a portion of the non-edited portion.

[0092] <<Motion Feature Calculation Method>> There are various methods for calculating motion features. For example, the selection unit 2060 calculates a statistical value (e.g., average value) of the magnitude of motion of an object represented by a frame sequence as the motion feature for that frame sequence. Specifically, the selection unit 2060 calculates a value representing the magnitude of motion of an object for each pair of two adjacent video frames obtained from a certain frame sequence A. The selection unit 2060 then calculates the statistical value of all the calculated values ​​as the motion feature for frame sequence A. Here, the value representing the magnitude of object motion can be, for example, the magnitude of optical flow or a pixel difference value between video frames.

[0093] When the motion feature is expressed as a statistical value of the magnitude of the object's movement in this manner, for example, as described above, the selection unit 2060 calculates the appropriateness score S as the ratio of the motion feature M1 calculated for the edited portion to the motion feature M2 calculated for the non-edited portion.

[0094] Alternatively, the motion feature may be expressed as a distribution (e.g., a histogram) of values ​​representing the magnitude of object motion. In this case, the selection unit 2060 calculates the degree of similarity between the distribution calculated for the edited portion and the distribution calculated for the non-edited portion as the appropriateness score. Here, the value representing the degree of similarity between the two distributions may be, for example, KL divergence.

[0095] <<Regarding Cases in Which Editing Processing Is Not Performed on Editing Target>> The selection unit 2060 may generate extended video data 30 without performing editing processing on the editing target. For example, the selection unit 2060 calculates the appropriateness score even in cases in which editing processing is not performed on the editing target. Then, for example, if the appropriateness score for the case in which editing processing is not performed is greater than the appropriateness scores calculated for each editing processing, the selection unit 2060 generates extended video data 30 without performing editing processing on the editing target.

[0096] Alternatively, for example, the selection unit 2060 determines whether the appropriateness score for the case where no editing processing is performed is equal to or greater than a threshold, and if the appropriateness score is equal to or greater than the threshold, generates extended video data 30 for the case where no editing processing is performed. For example, assume that the appropriateness score calculated for the editing processing E1 and the appropriateness score calculated for the case where no editing processing is performed are equal to or greater than a threshold. In this case, the selection unit 2060 generates extended video data 30 in which the editing processing E1 has been performed on the editing target, and extended video data 30 in which the editing processing E1 has not been performed on the editing processing.

[0097] Here, a method for calculating the appropriateness score for a case where editing processing is not performed will be described. FIG. 13 is a diagram illustrating a method for calculating the appropriateness score for a case where editing processing is not performed. In this example, the selection unit 2060 calculates a motion feature M1 for a frame sequence consisting of a frame sequence 42-1 that is the end portion of the first target frame sequence 40 and a frame sequence 52-1 that is the beginning portion of the second target frame sequence 50. The lengths of both the frame sequence 42-1 and the frame sequence 52-1 are a predetermined length K (K is 1 or greater).

[0098] The selection unit 2060 also calculates motion features for a frame sequence 42-2, which is the end portion of the first target frame sequence 40 excluding frame sequence 42-1, and a frame sequence 52-2, which is the beginning portion of the second target frame sequence 50 excluding frame sequence 52-1. Both frame sequences 42-2 and 52-2 are frame sequences of length L (L is 1 or greater). The selection unit 2060 then calculates a motion feature M2 from the motion feature calculated for frame sequence 42-2 and the motion feature calculated for frame sequence 52-2. The selection unit 2060 then calculates an appropriateness score based on the motion feature M1 and the motion feature M2.

[0099] The method for calculating the motion feature M2 based on the motion feature calculated for frame sequence 42-2 and the motion feature calculated for frame sequence 52-2 is the same as the method for calculating the motion feature M2 based on the motion feature calculated for frame sequence 60-1 and the motion feature calculated for frame sequence 60-3.

[0100] <Generation of Extended Video Data 30: S108> The generation unit 2080 generates extended video data 30 that has been subjected to the selected editing process (S108). Here, in order to calculate the appropriateness score described above, the generation unit 2080 has already generated, for each of the multiple types of editing process, a frame sequence in which that editing process has been applied to the editing target. Therefore, the generation unit 2080 can generate the extended video data 30 by combining the portion of the video data 10 that is not the editing target with the frame sequence generated by applying the selected editing process to the editing target.

[0101] Here, multiple editing processes may be selected for one editing target. In this case, for example, the selection unit 2060 generates extended video data 30 in which each selected editing process has been applied to the editing target.

[0102] For example, suppose two editing processes, E1 and E2, are selected for an editing object T1. In this case, the selection unit 2060 generates extended video data 30 in which the editing process E1 has been applied to the editing object T1, and extended video data 30 in which the editing process E2 has been applied to the editing object T1.

[0103] There may also be multiple editing targets. In this case, for example, the selection unit 2060 generates extended video data 30 in which the selected editing process has been applied to each editing target. For example, assume that there are two editing targets, editing target T1 and editing target T2. Also assume that editing process E1 has been selected for editing target T1 and editing process E2 has been selected for editing target T2. In this case, the selection unit 2060 generates, as extended video data 30, video data in which editing process E1 has been applied to editing target T1 and editing process E2 has been applied to editing target T2.

[0104] In a case where there are multiple editing targets, multiple editing processes may be selected for each editing target. In this case, for example, the selection unit 2060 generates extended video data 30 for each combination of editing targets and editing processes.

[0105] For example, suppose editing processes E1 and E2 are selected for editing object T1, and editing processes E3 and E4 are selected for editing object T2. In this case, there are four possible combinations of editing objects and editing processes: {(T1, E1), (T2, E3)}, {(T1, E1), (T2, E4)}, {(T1, E2), (T2, E3)}, and {(T1, E2), (T2, E4)}. Here, {(T1, E1), (T2, E3)} indicates that editing process E1 is applied to editing object T1, and editing process E3 is applied to editing object T2. Therefore, in this example, the selection unit 2060 generates extended video data 30 for each of the four combinations.

[0106] As described above, the extended video data 30 may be generated without performing any editing processing on the editing target.

[0107] <Generation of Class Information> The generation unit 2080 generates class information of the extended video data 30 from the class information of the video data 10 based on the processing performed by the processing unit 2040 and the content of the editing processing performed on the editing target. For example, the generation unit 2080 also deletes from the class information a frame sequence 20 that has been deleted by the processing unit 2040. Furthermore, the generation unit 2080 also changes the position in the class information of a frame sequence 20 whose position has been changed by the processing unit 2040.

[0108] Furthermore, the generation unit 2080 assigns a class to each video frame included in the editing target based on the content of the editing process applied to the editing target. For example, the generation unit 2080 divides the editing target into two (e.g., equal parts), and assigns the class of the first target frame sequence 40 to each video frame included in the first half of the editing target, and assigns the class of the second target frame sequence 50 to each video frame included in the second half of the editing target.

[0109] Alternatively, for example, if the editing process is a blending process, the class to be assigned to each video frame may be determined based on the blending ratio of the video frame. Specifically, the generation unit 2080 assigns the class of the video frame 12 that is used at a higher rate to the video frame generated by the blending process.

[0110] For example, suppose the class of the first target frame sequence is C1 and the class of the second target frame sequence is C2. Then, suppose a video frame included in the editing target is generated by blending a video frame 12 of the first target frame sequence with a video frame 12 of the second target frame sequence at a blend ratio of a:b. In this case, the generation unit 2080 assigns class C1 to the video frame if a >= b, and assigns class C2 to the video frame if a >= b.

[0111] ​<Output of Results> The data expansion device 2000 outputs the execution results. Hereinafter, information output from the data expansion device 2000 will be referred to as output information. The output information includes extended video data 30. Furthermore, if the extended video data 30 and the corresponding class information are configured separately, the output information further includes the class information of the extended video data 30. Here, if multiple pieces of extended video data 30 are generated, the output information includes multiple combinations of the extended video data 30 and the class information.

[0112] The output information may also be associated with the extended video data 30 and indicate the details of the processing and editing that was performed when the extended video data 30 was generated.

[0113] The output information may be output in any manner. For example, the data extension device 2000 may store the output information in any storage device. Alternatively, the data extension device 2000 may transmit the output information to any device. For example, the destination device may be a device that uses the extended video data 30 to train a classifier that identifies the class of each video frame included in a frame sequence.

[0114] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0115] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0116] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A data extension device comprising: acquisition means for acquiring video data including a plurality of frame sequences each composed of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes; processing means for performing, on the video data, a deletion process for deleting one or more of the frame sequences, a position change process for changing the positions of one or more of the frame sequences, or both; selection means for selecting, from a plurality of edit processes, an edit process to be performed on an edit target that is a connection portion between a first target frame sequence and a second target frame sequence that have become adjacent as a result of the deletion process or the position change process, or before and after said connection portion; and generation means for generating extended video data in which the selected edit process has been performed on the video data. (Supplementary Note 2) The data extension device according to Supplementary Note 1, wherein the selection means calculates an appropriateness score representing the degree of appropriateness of each of the plurality of edit processes based on a result of performing the edit process on the edit target, and selects the edit process to be performed on the edit target based on the appropriateness score. (Supplementary Note 3) The data expansion device according to Supplementary Note 2, wherein the selection means: calculates a first motion feature representing a feature of a motion of an object represented by the first target frame sequence and the second target frame sequence; calculates a second motion feature representing a feature of a motion of an object represented by the editing target; and calculates a degree of similarity between the first motion feature and the second motion feature as the appropriateness score. (Supplementary Note 4) The data expansion device according to Supplementary Note 3, wherein the first motion feature represents a statistical value of a magnitude of object motion calculated for the first target frame sequence and the second target frame sequence; the second motion feature represents a statistical value of a magnitude of object motion calculated for the editing target; and the appropriateness score represents a ratio of the first motion feature to the second motion feature.(Supplementary Note 5) The data extension device according to Supplementary Note 3, wherein the first motion feature represents a distribution of magnitude of object motion calculated for the first target frame sequence and the second target frame sequence, the second motion feature represents a distribution of object size calculated for the editing target, and the appropriateness score represents a degree of similarity between the distribution represented by the first motion feature and the distribution represented by the second motion feature. (Supplementary Note 6) The data extension device according to any one of Supplementary Notes 1 to 5, wherein the plurality of editing processes include a blending process that overlaps the end of the first target frame sequence with the beginning of the second target frame sequence, an interpolation process that inserts one or more video frames between the first target frame sequence and the second target frame sequence, or both. and a generation step of generating extended video data in which the video data is subjected to the selected editing process. (Supplementary Note 7) The data extension device according to Supplementary Note 6, wherein the multiple editing processes include multiple blending processes that overlap the end of the first target frame sequence with the beginning of the second target frame sequence at different ratios, multiple blending processes in which the lengths of the sections in which the end of the first target frame sequence and the beginning of the second target frame sequence overlap are different, or multiple interpolation processes in which the generation algorithms for the video frames to be inserted are different. (Supplementary Note 8) A data extension method executed by a computer, comprising: an acquisition step of acquiring video data including multiple frame sequences made up of multiple consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes; a processing step of performing, on the video data, a deletion process of deleting one or more of the frame sequences, a position change process of changing the positions of one or more of the frame sequences, or both; a selection step of selecting from multiple editing processes an editing process to be performed on an editing target that is a connection portion between the first target frame sequence and the second target frame sequence that have become adjacent as a result of the deletion process or the position change process, or which is before or after the connection portion; and a generation step of generating extended video data in which the selected editing process has been performed on the video data.(Supplementary Note 9) The data extension method according to Supplementary Note 8, wherein in the selection step, for each of the plurality of editing processes, an appropriateness score representing a degree of appropriateness of the editing process is calculated based on a result of executing the editing process on the editing target, and the editing process to be executed on the editing target is selected based on the appropriateness score. (Supplementary Note 10) The data extension method according to Supplementary Note 9, wherein in the selection step, a first motion feature representing a feature of a motion of an object represented by the first target frame sequence and the second target frame sequence is calculated, a second motion feature representing a feature of a motion of the object represented by the editing target is calculated, and a degree of similarity between the first motion feature and the second motion feature is calculated as the appropriateness score. (Supplementary Note 11) The data extension method according to Supplementary Note 10, wherein the first motion feature represents a statistical value of a magnitude of object motion calculated for the first target frame sequence and the second target frame sequence, the second motion feature represents a statistical value of a magnitude of object motion calculated for the editing target, and the appropriateness score represents a ratio between the first motion feature and the second motion feature. (Supplementary Note 12) The data extension method according to Supplementary Note 10, wherein the first motion feature represents a distribution of a magnitude of object motion calculated for the first target frame sequence and the second target frame sequence, the second motion feature represents a distribution of object sizes calculated for the editing target, and the appropriateness score represents a degree of similarity between a distribution represented by the first motion feature and a distribution represented by the second motion feature. (Appendix 13) The data extension method described in any one of Appendices 8 to 12, wherein the plurality of editing processes include a blending process that overlaps the end of the first target frame sequence with the beginning of the second target frame sequence, an interpolation process that inserts one or more video frames between the first target frame sequence and the second target frame sequence, or both.(Supplementary Note 14) The data extension method according to Supplementary Note 13, wherein the plurality of editing processes include a plurality of blending processes that overlap the end of the first target frame sequence with the beginning of the second target frame sequence at different ratios, a plurality of blending processes that have different lengths of the section where the end of the first target frame sequence overlaps the beginning of the second target frame sequence, or a plurality of interpolation processes that have different generation algorithms for the video frames to be inserted. (Supplementary Note 15) A program that causes a computer to execute the following steps: an acquisition step of acquiring video data including a plurality of frame sequences made up of a plurality of consecutive video frames that belong to the same class, wherein adjacent frame sequences belong to different classes; a processing step of performing, on the video data, a deletion process of deleting one or more of the frame sequences, a position change process of changing the positions of one or more of the frame sequences, or both; a selection step of selecting from a plurality of editing processes an editing process to be performed on an editing target that is a connecting portion between the first target frame sequence and the second target frame sequence that have become adjacent as a result of the deletion process or the position change process, or before and after that. (Supplementary Note 16) The program according to Supplementary Note 15, wherein in the selecting step, for each of the plurality of editing processes, an appropriateness score representing a degree of appropriateness of the editing process is calculated based on a result of executing the editing process on the editing target, and the editing process to be executed on the editing target is selected based on the appropriateness score. (Supplementary Note 17) The program according to Supplementary Note 16, wherein in the selecting step, a first motion feature representing a feature of a motion of an object represented by the first target frame sequence and the second target frame sequence is calculated, a second motion feature representing a feature of a motion of the object represented by the editing target is calculated, and a degree of similarity between the first motion feature and the second motion feature is calculated as the appropriateness score.(Supplementary Note 18) The program according to Supplementary Note 17, wherein the first motion feature represents a statistical value of a magnitude of an object's motion calculated for the first target frame sequence and the second target frame sequence, the second motion feature represents a statistical value of a magnitude of an object's motion calculated for the editing target, and the appropriateness score represents a ratio between the first motion feature and the second motion feature. (Supplementary Note 19) The program according to Supplementary Note 17, wherein the first motion feature represents a distribution of a magnitude of an object's motion calculated for the first target frame sequence and the second target frame sequence, the second motion feature represents a distribution of an object's size calculated for the editing target, and the appropriateness score represents a degree of similarity between the distribution represented by the first motion feature and the distribution represented by the second motion feature. (Supplementary Note 20) The program according to any one of Supplementary Notes 15 to 19, wherein the plurality of editing processes include a blending process that overlaps the end of the first target frame sequence with the beginning of the second target frame sequence, an interpolation process that inserts one or more video frames between the first target frame sequence and the second target frame sequence, or both. (Supplementary Note 21) The program according to Supplementary Note 20, wherein the plurality of editing processes include a plurality of blending processes that overlap the end of the first target frame sequence with the beginning of the second target frame sequence at different ratios, a plurality of blending processes in which the lengths of the sections in which the end of the first target frame sequence and the beginning of the second target frame sequence overlap are different, or a plurality of interpolation processes in which the generation algorithms for the video frames to be inserted are different.

[0117] This application claims priority based on Japanese Patent Application No. 2023-024891, filed February 21, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0118] REFERENCE SIGNS LIST 10 Video data 12 Video frame 20 Frame sequence 30 Extended video data 40 First target frame sequence 42 Frame sequence 50 Second target frame sequence 52 Frame sequence 60 Frame sequence 200 Table 202 Frame identification information 204 Class identification information 300 Table 302 First frame identification information 304 Last frame identification information 306 Class identification information 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface 2000 Data extension device 2020 Acquisition unit 2040 Processing unit 2060 Selection unit 2080 Generation unit

Claims

1. an acquiring means for acquiring video data including a plurality of frame sequences each composed of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes; a processing unit for performing, on the video data, a deletion process for deleting one or more of the frame sequences, a position change process for changing the positions of one or more of the frame sequences, or both; a selection means for selecting, from a plurality of editing processes, an editing process to be performed on an editing target that is a connecting portion between a first target frame sequence and a second target frame sequence that have become adjacent due to the deletion process or the position change process, or a portion before or after said connecting portion; and generating means for generating extended video data obtained by performing the selected editing process on the video data.

2. The selection means calculating an appropriateness score representing the degree of appropriateness of each of the plurality of editing processes based on a result of executing the editing process on the editing target; The data extension device according to claim 1 , further comprising: selecting the editing process to be performed on the editing target based on the appropriateness score.

3. The selection means calculating a first motion feature amount representing a feature of a motion of an object represented by the first target frame sequence and the second target frame sequence; calculating a second movement feature amount representing a movement feature of the object represented by the editing target; The data expansion device according to claim 2 , wherein a degree of similarity between the first motion feature amount and the second motion feature amount is calculated as the appropriateness score.

4. the first motion feature represents a statistical value of the magnitude of motion of an object calculated for the first target frame sequence and the second target frame sequence; the second movement feature amount represents a statistical value of a magnitude of movement of an object calculated for the editing target; The data expansion device according to claim 3 , wherein the appropriateness score represents a ratio between the first motion feature amount and the second motion feature amount.

5. the first motion feature represents a distribution of magnitude of motion of an object calculated for the first target frame sequence and the second target frame sequence; the second movement feature amount represents a distribution of magnitude of movement of an object calculated for the editing target; The data expansion device according to claim 3 , wherein the appropriateness score represents a degree of similarity between a distribution represented by the first motion feature amount and a distribution represented by the second motion feature amount.

6. 6. A data extension device as described in any one of claims 1 to 5, wherein the multiple editing processes include a blending process that overlaps the end of the first target frame sequence with the beginning of the second target frame sequence, an interpolation process that inserts one or more video frames between the first target frame sequence and the second target frame sequence, or both.

7. 7. The data extension device of claim 6, wherein the multiple editing processes include multiple blending processes that overlap the end of the first target frame sequence with the beginning of the second target frame sequence at different ratios, multiple blending processes that have different lengths of the section in which the end of the first target frame sequence and the beginning of the second target frame sequence overlap, or multiple interpolation processes that have different generation algorithms for the video frames to be inserted.

8. an acquiring step of acquiring video data including a plurality of frame sequences each composed of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes; a processing step of performing a deletion process on the video data to delete one or more of the frame sequences, a position change process to change the position of one or more of the frame sequences, or both; a selection step of selecting, from a plurality of editing processes, an editing process to be performed on an editing target that is a connecting portion between a first target frame sequence and a second target frame sequence that have become adjacent due to the deletion process or the position change process, or before and after said connecting portion; generating extended video data in which the selected editing process has been applied to the video data;

9. In the selection step, calculating an appropriateness score representing the degree of appropriateness of each of the plurality of editing processes based on a result of executing the editing process on the editing target; The data augmentation method according to claim 8 , further comprising selecting the editing process to be performed on the editing target based on the appropriateness score.

10. an acquiring step of acquiring video data including a plurality of frame sequences each composed of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes; a processing step of performing a deletion process on the video data to delete one or more of the frame sequences, a position change process to change the position of one or more of the frame sequences, or both; a selection step of selecting, from a plurality of editing processes, an editing process to be performed on an editing target that is a connecting portion between a first target frame sequence and a second target frame sequence that have become adjacent due to the deletion process or the position change process, or before and after said connecting portion; a generating step of generating extended video data in which the selected editing process has been performed on the video data.