Training device, training method, and non-transitory computer-readable medium

Abstract

This training device: identifies a plurality of pairs of a first video frame and a second video frame in a training frame sequence which is composed of a plurality of video frames; generates a first partial frame sequence which is composed of the first video frames of the pairs and a second partial frame sequence which is composed of the second video frames of the pairs; uses a feature extraction model which is configured so as to extract a feature quantity related to the video frames that are included in the frame sequences to calculate a feature quantity related to each first video frame and a feature quantity related to each second video frame; and updates the feature extraction model so as to further reduce a difference between the feature quantity related to the first video frame and the feature quantity related to the second video frame for each pair.

Description

Training device, training method, and non-transitory computer-readable medium

[0001] The present disclosure relates to a training device, a training method, and a non-transitory computer-readable medium.

[0002] Techniques for classifying video frames constituting video data have been developed. For example, Non-Patent Document 1 discloses a technique for classifying each video frame constituting input video data while taking into account the context of the entire video data. In the system of Non-Patent Document 1, class classification is performed using a model with a multi-stage configuration. The class classification results for all video frames obtained in a given stage are input to the next stage.

[0003] Yazan Abu Farha and Juergen Gall, "MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation," [online], April 2, 2019, arXiv, [Retrieved February 26, 2024], <URL: https: / / arxiv.org / pdf / 1903.01945.pdf>

[0004] In the system of Non-Patent Document 1, it is necessary to perform class classification multiple times for all video frames included in video data. The present disclosure has been made in consideration of this problem, and one of its objectives is to provide a new technique for classifying video frames.

[0005] A training device according to the present disclosure includes: a generating unit that identifies a plurality of pairs of a first video frame and a second video frame for a training frame sequence consisting of a plurality of video frames, and generates a first partial frame sequence consisting of the first video frame of each pair and a second partial frame sequence consisting of the second video frame of each pair; a feature extraction unit that calculates a feature for each of the first video frame and a feature for each of the second video frame using a feature extraction model configured to extract a feature for each video frame included in the frame sequence; and an updating unit that updates the feature extraction model so as to reduce a difference between the feature for the first video frame and the feature for the second video frame for each pair. In the training frame sequence, neither the first video frame nor the second video frame included in another pair is present between the first video frame and the second video frame included in the same pair.

[0006] A training method according to the present disclosure is executed by a computer. The training method includes a generating step of identifying a plurality of pairs of a first video frame and a second video frame for a training frame sequence consisting of a plurality of video frames, and generating a first partial frame sequence consisting of the first video frame of each pair and a second partial frame sequence consisting of the second video frame of each pair, a feature extraction step of calculating a feature for each of the first video frame and a feature for each of the second video frame using a feature extraction model configured to extract a feature for each video frame included in the frame sequence, and an updating step of updating the feature extraction model so as to reduce a difference between the feature for the first video frame and the feature for the second video frame for each pair. In the training frame sequence, neither the first video frame nor the second video frame included in another pair is present between the first video frame and the second video frame included in the same pair.

[0007] A non-transitory computer-readable medium according to the present disclosure stores a program that causes a computer to execute the following steps: generating a plurality of pairs of a first video frame and a second video frame from a training frame sequence consisting of a plurality of video frames, and generating a first partial frame sequence consisting of the first video frame of each pair and a second partial frame sequence consisting of the second video frame of each pair; calculating a feature amount for each of the first video frame and each of the second video frame using a feature extraction model configured to extract a feature amount for each video frame included in the frame sequence; and updating the feature extraction model so as to reduce a difference between the feature amount for the first video frame and the feature amount for the second video frame for each pair. In the training frame sequence, neither the first video frame nor the second video frame included in another pair is present between the first video frame and the second video frame included in the same pair.

[0008] According to the present disclosure, a new technique for classifying video frames is provided.

[0009] FIG. 1 is a diagram illustrating an overview of a training device. FIG. 2 is a diagram illustrating class classification performed using a feature extraction model. FIG. 3 is a block diagram illustrating a functional configuration of a training device. FIG. 4 is a block diagram illustrating a hardware configuration of a computer that realizes the training device. FIG. 5 is a flowchart illustrating a flow of processing performed by the training device. FIG. 6 is a diagram illustrating a sampling interval and an intra-pair interval. FIG. 7 is a diagram illustrating an interpolation process performed by an update unit. FIG. 8 is a second diagram illustrating an overview of a training device. FIG. 9 is a diagram illustrating a case in which a feature extraction model is trained using a pair of a first extended sequence and a second partial frame sequence. FIG. 10 is a diagram illustrating a case in which a feature extraction model is trained using a pair of a first partial frame sequence and a second extended sequence. FIG. 11 is a second diagram illustrating the functional configuration of a training device. FIG. 12 is a second flowchart illustrating a flow of processing performed by the training device. FIG. 13 is a third diagram illustrating an overview of a training device. FIG. 14 is a third flowchart illustrating a flow of processing performed by the training device. FIG. 15 is a fourth diagram illustrating an overview of a training device.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0011] <Overview> Fig. 1 is a diagram illustrating an example of the overview of training device 2000. The operations of training device 2000 shown in Fig. 1 are examples intended to facilitate understanding of training device 2000. Operations that can be performed by training device 2000 are not limited to the operations shown in Fig. 1.

[0012] The training device 2000 is used to train the feature extraction model 100. The feature extraction model 100 calculates features for each video frame constituting a frame sequence. A frame sequence is a sequence of multiple video frames arranged in chronological order (in ascending order of frame numbers). For example, in response to an input frame sequence, the feature extraction model 100 outputs features extracted from each video frame constituting the input frame sequence.

[0013] The feature extraction model 100 trained by the training device 2000 of this embodiment receives a frame sequence as input. However, as will be described later, the data input to the feature extraction model 100 is not limited to a frame sequence.

[0014] The feature extraction model 100 is configured as any machine learning-based model such as a neural network, etc. For example, the feature extraction model 100 is realized by a Transformer encoder.

[0015] The following provides an overview of training of the feature extraction model 100 performed by the training device 2000. The training device 2000 acquires a training frame sequence 10. The training frame sequence 10 is a frame sequence used for training the feature extraction model 100.

[0016] The training device 2000 identifies a plurality of pairs 40 for the training frame sequence 10. Each pair 40 is a combination of a first video frame 42 and a second video frame 44. In each pair 40, there is no other first video frame 42 or second video frame 44 between the first video frame 42 and the second video frame 44. In other words, in the training frame sequence 10, the pairs 40 do not overlap with each other on the time axis.

[0017] The training device 2000 generates a first partial frame sequence 20 consisting of a plurality of first video frames 42 and a second partial frame sequence 30 consisting of a plurality of second video frames 44. The order of the first video frames 42 in the first partial frame sequence 20 is the same as the order of those video frames in the training frame sequence 10. Similarly, the order of the second video frames 44 in the second partial frame sequence 30 is the same as the order of those video frames in the training frame sequence 10.

[0018] The training device 2000 inputs the first partial frame sequence 20 into the feature extraction model 100, thereby calculating the feature quantities 50 for each of the first video frames 42 that make up the first partial frame sequence 20. The training device 2000 also inputs the second partial frame sequence 30 into the feature extraction model 100, thereby calculating the feature quantities 50 for each of the second video frames 44 that make up the second partial frame sequence 30.

[0019] The training device 2000 uses the features 50 to update the feature extraction model 100. Specifically, the training device 2000 updates the parameters of the feature extraction model 100 (e.g., weights and biases of the neural network) so that, for each pair 40, the difference between the features 50 for the first video frame 42 and the features for the second video frame 44 becomes smaller.

[0020] The training device 2000 repeatedly updates the feature extraction model 100 by performing the above-described series of processes using each of the multiple training frame sequences 10. In this way, the training device 2000 trains the feature extraction model 100.

[0021] <Example of Action and Effect> The following illustrates an effect obtained by training the feature extraction model 100 using the training device 2000. As a premise, the feature quantity 50 for each video frame extracted by the feature extraction model 100 is used to identify the class to which the video frame belongs (i.e., class classification). FIG. 2 is a diagram illustrating class classification performed using the feature extraction model 100. Hereinafter, the device that performs class classification is referred to as the classification device 3000. Hereinafter, the class to which a video frame belongs is also referred to as the "class of the video frame."

[0022] For example, the classification device 3000 acquires video data 60 and identifies the class of each video frame that constitutes the video data 60. To do this, the classification device 3000 divides the video data 60 into a sequence of multiple frames 70. The classification device 3000 inputs the sequence of frames 70 into a feature extraction model 100 to obtain feature quantities 50 for each frame that constitutes the sequence of frames 70. The classification device 3000 performs class classification for each video frame using the feature quantities 50.

[0023] Specifically, the classification device 3000 inputs features 50 to the classification model 200. The classification model 200 is configured to output a value representing the class of the video frame (hereinafter, referred to as a class label) in response to the input of features related to the video frame. The class label is, for example, a one-hot vector whose number of elements is equal to the total number of classes, and whose only element corresponding to the class of the video frame is 1.

[0024] The classification device 3000 inputs the feature 50 obtained from each video frame constituting the frame sequence 70 into the classification model 200. This identifies the class to which each video frame constituting the frame sequence 70 belongs.

[0025] The classification device 3000 performs the above process for each of the plurality of frame sequences 70. As a result, the class of each video frame constituting the video data 60 is identified.

[0026] The above-described process is performed for each of the multiple frame sequences 70. As a result, a class is identified for each of all video frames constituting the video data 60. The classification model 200 may be configured to receive a sequence of features 50 as input. For example, the classification model 200 may be configured to identify the class of each video frame constituting the frame sequence 70 in response to input of a sequence of features 50 obtained from one frame sequence 70.

[0027] For example, a class represents the type of behavior or situation captured in a video frame. For example, assume that the video data 60 records the state of multiple types of work performed by a worker. In this case, for example, a class represents the type of work. By using the classification device 3000, it is possible to understand which type of work is being performed in which part of the video data 60.

[0028] Now, suppose that each video frame constituting the video data is classified individually without considering other video frames (i.e., without considering the context). This method is prone to errors in the classification results, because it is difficult to accurately infer the class of a video frame (e.g., the type of activity captured in that video frame) from a single video frame.

[0029] In this regard, the feature extraction model 100 outputs the feature quantities 50 of each video frame constituting the frame sequence in response to the input of the frame sequence. Therefore, the feature extraction model 100 can calculate the feature quantities 50 of each video frame while taking into account the contents of other video frames. Therefore, by using the feature extraction model 100 for classification, it is possible to classify each video frame while taking into account the other video frames.

[0030] It can be said that a particular action or situation is likely to continue for a certain period of time, and therefore, in the video data 60, video frames that are close to each other on the time axis are likely to belong to the same class.

[0031] Therefore, the training device 2000 treats two video frames that are close to each other on the time axis of the training frame sequence 10 as a pair 40, and trains the feature extraction model 100 so that the difference between the feature quantities 50 of the two video frames (i.e., the first video frame 42 and the second video frame 44) included in the same pair 40 is small. This makes the feature quantities 50 of video frames that are close to each other on the time axis more likely to be similar to each other. Therefore, in classification using the feature extraction model 100, video frames that are close to each other on the time axis are more likely to be classified into the same class. Therefore, by using the feature extraction model 100, it is possible to classify video frames taking into account the fact that "specific actions or situations are highly likely to continue for a certain period of time."

[0032] The system of Non-Patent Document 1 can identify the class of each video frame by taking into account the context of the entire video data. However, in order to take into account the context of the entire video data, the system of Non-Patent Document 1 needs to repeatedly perform classification on the entire video data. Therefore, it is considered that responsiveness is relatively low. Furthermore, if classification is performed on a relatively short sequence of frames extracted from video data, the accuracy of the classification is likely to be low.

[0033] In this regard, the feature extraction model 100 is trained to extract features of each video frame, taking into consideration the fact that "features of video frames that are close to each other on the time axis are likely to be similar." Therefore, it can be said that the accuracy of classification using the feature extraction model 100 is high even when a relatively short frame sequence is used as the target. Therefore, by using the feature extraction model 100, it is possible to achieve high-accuracy classification of video frames for a relatively short frame sequence. Furthermore, since a relatively short frame sequence can be used as the target for classification, responsiveness is improved.

[0034] The training device 2000 of this embodiment will be described in more detail below.

[0035] <Example of Functional Configuration> FIG. 3 is a block diagram illustrating an example of the functional configuration of the training device 2000. The training device 2000 includes a generation unit 2020, a feature extraction unit 2040, and an update unit 2060. The generation unit 2020 generates a first partial frame sequence 20 by extracting a first video frame 42 from each of a plurality of pairs 40 included in the training frame sequence 10. The generation unit 2020 generates a second partial frame sequence 30 by extracting a second video frame 44 from each of the plurality of pairs 40. The feature extraction unit 2040 uses the feature extraction model 100 to obtain a feature 50 for each first video frame 42 and a feature 50 for each second video frame 44. The update unit 2060 updates the feature extraction model 100 so as to reduce the difference between the feature 50 for the first video frame 42 and the feature 50 for the second video frame 44 for each pair 40.

[0036] <Example of Hardware Configuration> Each functional component of the training device 2000 may be realized by hardware that realizes the functional component (e.g., a hardwired electronic circuit, etc.), or by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a case where each functional component of the training device 2000 is realized by a combination of hardware and software will be further described.

[0037] 4 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that implements the training device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 1000 may be a portable computer such as a smartphone or a tablet terminal. The computer 1000 may be a dedicated computer designed to implement the training device 2000, or may be a general-purpose computer.

[0038] For example, by installing a predetermined application on the computer 1000, the computer 1000 realizes each function of the training device 2000. The application is configured as a program for realizing each functional component of the training device 2000. The method for acquiring the program is arbitrary. For example, the program can be acquired from a storage medium on which the program is stored. The storage medium on which the program is stored may be any storage medium such as a DVD (Digital Versatile Disk) or a USB (Universal Serial Bus) memory. Alternatively, the program can be acquired by downloading the program from a server device that manages the storage device on which the program is stored.

[0039] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection.

[0040] The processor 1040 is one of various processors such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, a read only memory (ROM), or the like.

[0041] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0042] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0043] The storage device 1080 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the training device 2000. The processor 1040 reads this program into the memory 1060 and executes it to realize each functional component of the training device 2000.

[0044] The training device 2000 may be realized by one computer 1000 or by multiple computers 1000. In the latter case, the configurations of the computers 1000 do not need to be the same, and can be different from each other.

[0045] The hardware configuration of the classification device 3000 described above is also represented, for example, in FIG. 4 , similar to the hardware configuration of the training device 2000. However, the hardware configuration of the classification device 3000 may be different from the hardware configuration of the training device 2000.

[0046] <Processing Flow> Figure 5 is a flowchart illustrating the processing flow executed by the training device 2000. The training device 2000 acquires a training frame sequence 10 (S102). The generation unit 2020 generates a first partial frame sequence 20 and a second partial frame sequence 30 (S104). The feature extraction unit 2040 uses the feature extraction model 100 to calculate feature quantities 50 for each first video frame 42 constituting the first partial frame sequence 20 and for each second video frame 44 constituting the second partial frame sequence 30 (S106). The update unit 2060 updates the feature extraction model 100 using the feature quantities 50 for each first video frame 42 and each second video frame 44 (S108).

[0047] <Acquiring the training frame sequence 10: S102> The training device 2000 acquires the training frame sequence 10. Here, various methods can be used to acquire the frame sequence to be processed. For example, the training frame sequence 10 is stored in advance in an arbitrary storage device in a format that can be acquired by the training device 2000. In this case, the training device 2000 acquires the training frame sequence 10 by reading the training frame sequence 10 from the storage device.

[0048] Alternatively, for example, the training device 2000 may acquire the training frame sequence 10 by receiving the training frame sequence 10 transmitted from another device. The device that transmits the training frame sequence 10 is, for example, the device that generated the training frame sequence 10.

[0049] The training device 2000 may acquire video data and extract a plurality of training frame sequences 10 from the acquired video data. For example, the training device 2000 extracts a plurality of training frame sequences 10 from the video data in a time window having a predetermined window width and step width. The training device 2000 trains the feature extraction model 100 using each of the extracted training frame sequences 10.

[0050] <Generation of First Partial Frame Sequence 20 and Second Partial Frame Sequence 30: S104> The generation unit 2020 generates the first partial frame sequence 20 and the second partial frame sequence 30 (S104). To do so, the generation unit 2020 identifies a plurality of pairs 40 for the training frame sequence 10. For example, to identify the pairs 40, a sampling interval and an intra-pair interval are determined in advance. The sampling interval represents the interval between the first video frames 42 in two adjacent pairs 40. The intra-pair interval represents the interval between the first video frame 42 and the second video frame 44 included in the same pair 40.

[0051] 6 is a diagram illustrating an example of a sampling interval and an intra-pair interval. In the example in the upper part of FIG. 6, the sampling interval and the intra-pair interval are 3 and 0, respectively. Since the sampling interval is 3, there are three video frames between the first video frames 42 in two adjacent pairs 40. For example, there are three video frames between the first video frame 42-1 and the first video frame 42-2.

[0052] Furthermore, since the intra-pair interval is 0, there is no video frame between the first video frame 42 and the second video frame 44 that are included in the same pair 40. In other words, in each pair 40, the first video frame 42 and the second video frame 44 are adjacent video frames. For example, in pair 40-1, the first video frame 42-1 and the second video frame 44-1 are adjacent to each other.

[0053] 6, the sampling interval and intra-pair interval are 5 and 0, respectively. Since the sampling interval is 5, there are five video frames between the first video frames 42 in two adjacent pairs 40. For example, there are five video frames between the first video frames 42-4 and 42-5.

[0054] Furthermore, since the intra-pair interval is 1, there is one video frame between the first video frame 42 and the second video frame 44 which are included in the same pair 40. For example, in pair 40-4, there is one video frame between the first video frame 42-4 and the second video frame 44-4.

[0055] For example, the generation unit 2020 acquires a sampling interval and an intra-pair interval, and extracts a plurality of pairs 40 from the training frame sequence 10 based on the acquired sampling interval and intra-pair interval. The generation unit 2020 also extracts a first video frame 42 from each pair 40 and arranges the extracted first video frames 42 in the same order as in the training frame sequence 10, thereby generating a first partial frame sequence 20. The generation unit 2020 also extracts a second video frame 44 from each pair 40 and arranges the extracted second video frames 44 in the same order as in the training frame sequence 10, thereby generating a second partial frame sequence 30.

[0056] In each of the examples described above, the second video frame 44 is located after the first video frame 42. However, the second video frame 44 may be located before the first video frame 42. Whether the second video frame 44 is located before or after the first video frame 42 may be determined in advance, or may be dynamically determined by the generation unit 2020. In the latter case, for example, the generation unit 2020 randomly determines whether the second video frame 44 is to be obtained before or after the first video frame 42.

[0057] Here, the generating unit 2020 may change the sampling interval, the intra-pair interval, or both to generate a plurality of pairs of the first partial frame sequence 20 and the second partial frame sequence 30 from one training frame sequence 10. In this case, the training device 2000 updates the feature extraction model 100 using each of the generated pairs.

[0058] For example, the generation unit 2020 generates a first pair of a first partial frame sequence 20 and a second partial frame sequence 30 under the settings of "sampling interval: 3, intra-pair interval: 1" illustrated in the upper part of Fig. 6. Furthermore, the generation unit 2020 generates a second pair of a first partial frame sequence 20 and a second partial frame sequence 30 under the settings of "sampling interval: 5, intra-pair interval: 1" illustrated in the lower part of Fig. 6. The training device 2000 updates the feature extraction model 100 using both the first pair and the second pair.

[0059] As described above, the training device 2000 updates the feature extraction model 100 using each of the multiple training frame sequences 10. In this case, the sampling interval and the intra-pair interval may be common to all of the training frame sequences 10, or may be different for each training frame sequence 10.

[0060] In the latter case, for example, the generation unit 2020 may set the sampling interval relatively large in the initial stage of training of the feature extraction model 100, and decrease the sampling interval as the training of the feature extraction model 100 progresses. Similarly, the generation unit 2020 may set the intra-pair spacing relatively large in the initial stage of training of the feature extraction model 100, and decrease the intra-pair spacing as the training of the feature extraction model 100 progresses. In this way, by gradually decreasing the sampling interval and intra-pair spacing, the time required to train the feature extraction model 100 can be shortened.

[0061] <Feature Extraction: S106> The feature extraction unit 2040 uses the feature extraction model 100 to calculate feature quantities 50 for each first video frame 42 constituting the first partial frame sequence 20 and feature quantities 50 for each second video frame 44 constituting the second partial frame sequence 30 (S106). As described above, in the training device 2000 of embodiment 1, the feature extraction model 100 extracts feature quantities from each frame constituting the frame sequence in response to the input of the frame sequence. Thus, the feature extraction unit 2040 inputs the first partial frame sequence 20 to the feature extraction model 100 to obtain feature quantities 50 for each first video frame 42 included in the first partial frame sequence 20. Furthermore, the feature extraction unit 2040 inputs the second partial frame sequence 30 to the feature extraction model 100 to obtain feature quantities 50 for each second video frame 44 included in the second partial frame sequence 30.

[0062] Here, it is preferable that the feature extraction model 100 is configured to extract features from each video frame taking into account the context of the entire frame sequence, rather than being configured to extract features independently from each video frame constituting the frame sequence. For example, the Transformer encoder described above can be used as such a model.

[0063] <Updating the Feature Extraction Model 100: S108> The updating unit 2060 updates the feature extraction model 100 using the feature quantities 50 for each first video frame 42 and the feature quantities 50 for each second video frame 44 (S108). Specifically, the updating unit 2060 calculates, for each pair 40, an index value (hereinafter referred to as a difference index value) that represents the difference between the feature quantities 50 of the first video frame 42 and the feature quantities 50 of the second video frame 44. Hereinafter, the difference index value calculated for the i-th pair 40 will be represented by the symbol D[i]. The i-th pair 40 will also be represented as pair i.

[0064] The feature extraction unit 2040 updates the parameters of the feature extraction model 100 so as to reduce the difference between the feature amount 50 of the first video frame 42 and the feature amount 50 of the second video frame 44 for each pair 40. Specifically, the feature extraction unit 2040 calculates a loss (hereinafter referred to as smoothing loss) based on the difference index value, and updates the parameters of the feature extraction model 100 using the calculated smoothing loss.

[0065] The dissimilarity index D[i] may be any of various values ​​that can represent the difference between the features. For example, the dissimilarity index D[i] may be expressed as a norm (e.g., L2 norm) between the feature 50 for the first video frame 42 of the pair i and the feature 50 for the first video frame 42 of the pair i.

[0066] Alternatively, for example, the dissimilarity index value can be expressed using the result of class classification obtained from the classification model 200 by inputting the feature 50. The dissimilarity index value D[i] calculated using the classification model 200 can be determined, for example, as follows: f[i] represents the identifier of the first video frame 42 in pair i. s[i] represents the identifier of the second video frame 44 in pair i. c represents the class label. y[x,c] represents the probability that video frame x belongs to class c, calculated by inputting the features 50 related to video frame x into the classification model 200. C represents the set of classes. n(C) represents the total number of classes.

[0067] Here, an upper limit value τ may be set for the value of d[i][c] as follows.

[0068] The feature extraction unit 2040 calculates the smoothing loss using the difference index value calculated for each pair 40. For example, the smoothing loss Ls is defined as follows:

[0069] The feature extraction unit 2040 updates the feature extraction model 100 so as to reduce the smoothing loss. Note that the method for calculating the smoothing loss is not limited to the method using equation (3).

[0070] Here, the feature extraction model 100 may be trained together with the classification model 200 described above. In this case, the training device 2000 calculates, in addition to the smoothing loss, a loss (hereinafter referred to as classification loss) representing the low accuracy of class classification by the classification model 200. Specifically, the training device 2000 inputs the feature quantities 50 of each video frame included in the training frame sequence 10 into the classification model 200, thereby obtaining a value representing the classification result for each video frame. The value representing the classification result is, for example, a vector (hereinafter referred to as classification vector) representing the probability that, for each of a plurality of classes, the video frame belongs to that class.

[0071] The training device 2000 calculates the classification loss using the classification vector obtained for each video frame. The training device 2000 then updates the feature extraction model 100 so as to reduce the overall loss, which is determined based on the smoothing loss and the classification loss. The overall loss is determined, for example, as a weighted sum of the smoothing loss and the classification loss.

[0072] To calculate the classification loss, the training device 2000 needs to know the class (ground truth class) of each video frame. For example, the training device 2000 acquires the training frame sequence 10 along with information indicating the class of each video frame included in the training frame sequence 10.

[0073] <Interpolation of Feature Amounts 50> The update unit 2060 may increase the number of feature amounts 50 by performing interpolation processing on each of the plurality of feature amounts 50 calculated from the first partial frame sequence 20 and the plurality of feature amounts 50 calculated from the second partial frame sequence 30. This increases the number of pairs of feature amounts 50 available for training the feature extraction model 100.

[0074] 7 is a diagram illustrating an example of the interpolation process performed by the update unit 2060. In the example of FIG. 7, the length of each of the first partial frame sequence 20 and the second partial frame sequence 30 is 3. Therefore, three feature amounts 50 are calculated for each of the first partial frame sequence 20 and the second partial frame sequence 30.

[0075] The updating unit 2060 generates three interpolated features 55 between two adjacent feature amounts 50 for the plurality of feature amounts 50 calculated from the first partial frame sequence 20. Similarly, the updating unit 2060 generates three interpolated features 55 between two adjacent feature amounts 50 for the plurality of feature amounts 50 calculated from the second partial frame sequence 30.

[0076] For example, the features calculated from the first partial frame sequence 20 are feature 50-1, feature 50-2, and feature 50-3. Therefore, three interpolated features 55 (interpolated feature 55-1, interpolated feature 55-2, and interpolated feature 55-3) are generated between feature 50-1 and feature 50-2 by the interpolation process. Similarly, three interpolated features 55 (interpolated feature 55-4, interpolated feature 55-5, and interpolated feature 55-6) are generated between feature 50-2 and feature 50-3.

[0077] The updating unit 2060 calculates a difference index value by pairing the interpolated features 55 calculated from the first partial frame sequence 20 with the interpolated features 55 calculated from the second partial frame sequence 30. The updating unit 2060 then calculates a smoothing loss using the difference index value calculated for each of the multiple pairs of features 50 and the difference index value calculated for each of the multiple pairs of interpolated features 55.

[0078] Here, the specific process used for the interpolation is arbitrary. For example, the update unit 2060 performs linear interpolation as the interpolation process.

[0079] The number of interpolated features 55 to be generated is arbitrary. For example, the update unit 2060 generates the same number of interpolated features 55 as the sampling interval between features 50. In this way, features are virtually obtained from the feature extraction model 100 for all video frames included in the training frame sequence 10.

[0080] Increasing the number of features through interpolation processing can reduce the computer resources and time used to train the feature extraction model 100. In addition, for example, by creating more computer resources and calculation time, it is possible to add additional processing to the training of the feature extraction model 100, thereby further increasing the accuracy of the feature extraction model 100.

[0081] <Output of Results> The training device 2000 outputs the execution results. Hereinafter, information output from the training device 2000 will be referred to as output information. For example, the output information includes parameters of the trained feature extraction model 100. Here, the output information may or may not include parameters (such as hyperparameters) that are not updated by the training device 2000. Alternatively, for example, the output information may further include a program that realizes the feature extraction model 100 in addition to the parameters of the feature extraction model 100.

[0082] The output information may be output in any manner. For example, the training device 2000 may store the output information in any storage device. Alternatively, the training device 2000 may transmit the output information to any device. Alternatively, the training device 2000 may display the output information on any display device.

[0083] [Embodiment 2] Fig. 8 is a second diagram illustrating an overview of a training device 2000. The operation of the training device 2000 shown in Fig. 8 is an example intended to facilitate understanding of the training device 2000. The operations that can be performed by the training device 2000 are not limited to those shown in Fig. 8.

[0084] The training device 2000 of embodiment 2 performs data extension on the first partial frame sequence 20, the second partial frame sequence 30, or both. Hereinafter, a frame sequence obtained by performing data extension on a frame sequence will be generally referred to as an extended sequence. Furthermore, a frame sequence obtained by performing data extension on the first partial frame sequence 20 will be referred to as a first extended sequence 80. Furthermore, a frame sequence obtained by performing data extension on the second partial frame sequence 30 will be referred to as a second extended sequence 90. Note that in the example of FIG. 8 , data extension is performed on both the first partial frame sequence 20 and the second partial frame sequence 30.

[0085] The training device 2000 trains the feature extraction model 100 using at least one of the following: 1) a pair of the first extended sequence 80 and the second extended sequence 90; 2) a pair of the first extended sequence 80 and the second partial frame sequence 30; and 3) a pair of the first partial frame sequence 20 and the second extended sequence 90. The method for training the feature extraction model 100 using two pairs of frame sequences is the same as the method described in the first embodiment.

[0086] 8, assume that the feature extraction model 100 is trained using a pair of a first augmented sequence 80 and a second augmented sequence 90. In this case, the feature extractor 2040 inputs the first augmented sequence 80 to the feature extraction model 100 to obtain features 50 for each video frame of the first augmented sequence 80. The feature extractor 2040 also inputs the second augmented sequence 90 to the feature extraction model 100 to obtain features 50 for each video frame of the second augmented sequence 90.

[0087] The update unit 2060 identifies a plurality of pairs of video frames from the first extension sequence 80 and the second extension sequence 90. Here, video frames positioned at the same order in the first extension sequence 80 and the second extension sequence 90 are treated as a pair. Specifically, the i-th video frame in the first extension sequence 80 and the i-th video frame in the second extension sequence 90 are treated as the i-th pair.

[0088] The updating unit 2060 updates the feature extraction model 100 so as to reduce the difference between the feature quantities 50 in each identified pair. Specifically, the updating unit 2060 calculates the above-described difference index value for each identified pair. The updating unit 2060 then calculates the smoothing loss using the calculated difference index value for each pair, and updates the parameters of the feature extraction model 100 so as to reduce the smoothing loss.

[0089] Assume that the feature extraction model 100 is trained using a pair of a first augmented sequence 80 and a second partial frame sequence 30. Figure 9 illustrates a case in which the feature extraction model 100 is trained using a pair of a first augmented sequence 80 and a second partial frame sequence 30. In this case, the training device 2000 inputs the first augmented sequence 80 to the feature extraction model 100 to obtain feature quantities 50 for each video frame of the first augmented sequence 80. The training device 2000 also inputs the second partial frame sequence 30 to the feature extraction model 100 to obtain feature quantities 50 for each video frame of the second partial frame sequence 30.

[0090] The training device 2000 identifies multiple pairs of video frames from the first extended sequence 80 and the second partial frame sequence 30. Here, video frames located at the same position in the first extended sequence 80 and the second partial frame sequence 30 are treated as a pair. Specifically, the i-th video frame of the first extended sequence 80 and the i-th video frame of the second partial frame sequence 30 are treated as the i-th pair. The training device 2000 updates the feature extraction model 100 so that the difference between the feature quantities 50 of each identified pair is reduced.

[0091] Assume that the feature extraction model 100 is trained using a pair of a first partial frame sequence 20 and a second extended sequence 90. FIG. 10 illustrates a case in which the feature extraction model 100 is trained using a pair of the first partial frame sequence 20 and the second extended sequence 90. In this case, the feature extraction unit 2040 inputs the first partial frame sequence 20 to the feature extraction model 100 to obtain feature quantities 50 for each video frame of the first partial frame sequence 20. The feature extraction unit 2040 also inputs the second extended sequence 90 to the feature extraction model 100 to obtain feature quantities 50 for each video frame of the second extended sequence 90.

[0092] The update unit 2060 identifies a plurality of pairs of video frames from the first partial frame sequence 20 and the second extended sequence 90. Here, video frames positioned at the same order in the first partial frame sequence 20 and the second extended sequence 90 are treated as a pair. Specifically, the i-th video frame in the first partial frame sequence 20 and the i-th video frame in the second extended sequence 90 are treated as the i-th pair. The update unit 2060 updates the feature extraction model 100 so that the difference between the feature amounts 50 in each identified pair is reduced.

[0093] <Example of Action and Effect> In order to improve the accuracy of the feature extraction model 100, it is preferable to train the feature extraction model 100 using a variety of training data. In this regard, the training device 2000 of this embodiment performs data extension on the first partial frame sequence 20 or the second partial frame sequence 30. This allows the training device 2000 to easily increase the variety of training data. Therefore, it is possible to easily train the feature extraction model 100 using a variety of training data.

[0094] <Example of Functional Configuration> Fig. 11 is a second diagram illustrating the functional configuration of the training device 2000. In Fig. 11, the training device 2000 further includes a data extension unit 2080. The data extension unit 2080 performs data extension on the first partial frame sequence 20 to generate a first extended sequence 80, performs data extension on the second partial frame sequence 30 to generate a second extended sequence 90, or performs both. The update unit 2060 updates the feature extraction model 100 using at least one of the pair of the first extended sequence 80 and the second extended sequence 90, the pair of the first partial frame sequence 20 and the second extended sequence 90, and the pair of the first extended sequence 80 and the second partial frame sequence 30.

[0095] <Example of Hardware Configuration> The hardware configuration of the training device 2000 of embodiment 2 is shown in Fig. 4, for example, similar to the hardware configuration of the training device 2000 of embodiment 1. However, the storage device 1080 of embodiment 2 stores a program that realizes each functional component of the training device 2000 of embodiment 2.

[0096] <Processing Flow> FIG. 12 is a second flowchart illustrating the processing flow executed by the training device 2000. The flowchart in FIG. 12 illustrates the processing flow in a case where both a first extended sequence 80 and a second extended sequence 90 are generated. The training device 2000 acquires a training frame sequence 10 (S202). The generator 2020 generates a first partial frame sequence 20 and a second partial frame sequence 30 (S204). The data extension unit 2080 performs data extension on the first partial frame sequence 20 and the second partial frame sequence 30 to generate the first extended sequence 80 and the second extended sequence 90 (S206). The feature extractor 2040 uses the feature extraction model 100 to calculate feature quantities 50 for each first video frame 42 constituting the first extended sequence 80 and each second video frame 44 constituting the second extended sequence 90 (S208). The update unit 2060 updates the feature extraction model 100 using the feature quantities 50 for each video frame included in the first extended sequence 80 and the feature quantities 50 for each video frame included in the second extended sequence 90 (S210).

[0097] <Regarding Data Extension> The data extension unit 2080 performs data extension on the first partial frame sequence 20, the second partial frame sequence 30, or both. Data extension on a frame sequence is performed by performing data extension on one or more video frames that make up the frame sequence. Data extension does not need to be performed on all video frames included in the frame sequence, but may be performed on any one or more video frames.

[0098] Various data augmentations can be performed on video frames, such as rotating the entire video frame, adding noise to the video frame, or applying color transformation to the video frame. The type of data augmentation performed can be predetermined or randomly selected.

[0099] Data augmentation does not have to be performed on the entire video frame, but may be performed on one or more subregions included in the video frame, such as by rotating each of the one or more subregions included in the video frame, adding noise to each of the one or more subregions included in the video frame, or applying color conversion to each of the one or more subregions included in the video frame.

[0100] Here, the data extension unit 2080 may generate multiple extended sequences from one partial frame sequence. For example, the data extension unit 2080 generates a first extended sequence 80 by rotating each video frame of the first partial frame sequence 20. The data extension unit 2080 also generates a second extended sequence 80 by adding noise to each video frame of the first partial frame sequence 20. The same applies to the second partial frame sequence 30.

[0101] For example, the training device 2000 selects and pairs one frame sequence from each of a first group consisting of the first partial frame sequence 20 and a plurality of first extended sequences 80 and a second group consisting of the second partial frame sequence 30 and a plurality of second extended sequences 90. The training device 2000 then uses the paired two frame sequences to train the feature extraction model 100.

[0102] [Embodiment 3] Fig. 13 is a third diagram illustrating an overview of a training device 2000. The operation of the training device 2000 shown in Fig. 13 is an example intended to facilitate understanding of the training device 2000. The operations that can be performed by the training device 2000 are not limited to the operations shown in Fig. 13.

[0103] Here, it is assumed that the feature extraction model 100 is configured to handle, as input, not a frame sequence but a sequence of features corresponding to the frame sequence (hereinafter, a feature sequence). Therefore, the training device 2000 generates a feature sequence corresponding to the first partial frame sequence 20 (hereinafter, a first feature sequence 120) and inputs the first feature sequence 120 to the feature extraction model 100. This results in obtaining a feature 50 for each first video frame 42. The feature 50 for the first video frame 42 is obtained by inputting the feature corresponding to the first video frame 42 to the feature extraction model 100.

[0104] Similarly, the training device 2000 generates a feature sequence corresponding to the second partial frame sequence 30 (hereinafter, referred to as the second feature sequence 130) and inputs the second feature sequence 130 to the feature extraction model 100. This results in obtaining a feature 50 for each second video frame 44. The feature 50 for the second video frame 44 is obtained by inputting the feature corresponding to the second video frame 44 into the feature extraction model 100.

[0105] For example, the training device 2000 performs two or more types of feature extraction processing on each first video frame 42, thereby calculating two or more types of feature amounts for each first video frame 42. In the example of Fig. 13, two types of feature amounts are calculated for each first video frame 42. Similarly, the training device 2000 performs two or more types of feature extraction processing on each second video frame 44, thereby calculating two or more types of feature amounts for each second video frame 44.

[0106] The training device 2000 generates a combined feature 122 for each first video frame 42 by combining multiple features obtained for that first video frame 42. The combined feature 122 is, for example, a concatenation of multiple features. The training device 2000 generates a first feature sequence 120 by chronologically arranging the combined features 122 generated for each first video frame 42.

[0107] Similarly, the training device 2000 generates a combined feature 132 for each second video frame 44 by combining multiple features obtained for that second video frame 44. For example, the combined feature 132 is a concatenation of multiple features, similar to the combined feature 122. The training device 2000 generates a second feature sequence 130 by chronologically arranging the combined features 132 generated for each second video frame 44. Note that the feature extraction process performed for the second video frame 44 is the same as the feature extraction process performed for the first video frame 42.

[0108] The training device 2000 updates the feature extraction model 100 using the feature quantities 50 for each of the first video frames 42 and the feature quantities 50 for each of the second video frames 44 obtained in this manner. Here, the method for updating the feature extraction model 100 using the feature quantities 50 for each of the first video frames 42 and the feature quantities 50 for each of the second video frames 44 is as described in the first embodiment.

[0109] There are various types of feature extraction processes that can be performed to obtain the first feature sequence 120 and the second feature sequence 130. For example, the feature extraction process is a process that extracts features from video frames by analyzing the video frames. The features extracted from the video frames include, for example, image features of the video frames and skeletal features extracted from the video frames. A trained machine learning model can be used to calculate the image features and skeletal features, for example.

[0110] The feature extraction process may be a process of extracting features from data other than video frames. For example, the training device 2000 acquires sensing data (data representing the observation results of a sensor) corresponding to each first video frame 42 and performs feature extraction on the sensing data. Similarly, the training device 2000 acquires sensing data corresponding to each second video frame 44 and performs feature extraction on the sensing data.

[0111] The sensing data corresponding to a video frame may be, for example, audio data obtained from an audio sensor at the time the video frame is generated. Alternatively, the sensing data corresponding to a video frame may be acceleration data obtained from an acceleration sensor at the time the video frame is generated. Here, a trained machine learning model may be used to calculate the feature values ​​of the sensing data.

[0112] When sensing data is used, the training device 2000 acquires the sensing data in addition to the training frame sequence 10. For example, the sensing data is stored in advance in an arbitrary storage unit in association with the training frame sequence 10.

[0113] Alternatively, for example, the training device 2000 may acquire text data (hereinafter, caption data) representing the content of each first video frame 42 and perform feature extraction processing on the caption data. Similarly, the training device 2000 may acquire caption data for each second video frame 44 and perform feature extraction processing on the caption data. Here, for example, a trained machine learning model can be used to calculate the feature amounts of the text data.

[0114] The caption data may be prepared in advance or may be generated by the training device 2000. In the latter case, for example, the training device 2000 generates caption data from video frames using a trained machine learning model (hereinafter referred to as a caption generation model). The caption generation model is trained in advance to output text data representing the content of image data in response to input of the image data. The training device 2000 inputs video frames to the caption generation model, thereby obtaining caption data for the video frames from the caption generation model.

[0115] <Example of Action and Effect> The feature extraction model 100 handled by the training device 2000 of embodiment 3 extracts a feature 50 for each video frame using multiple types of feature extracted from a frame sequence. According to this method, the feature 50 used for class classification is calculated using various information related to the video frames. Therefore, according to the training device 2000 of embodiment 3, it is possible to train the feature extraction model 100 so as to calculate a feature 50 more suitable for class classification. Therefore, by using the feature extraction model 100 trained by the training device 2000 of embodiment 3, it is possible to perform class classification of video frames with higher accuracy.

[0116] <Example of Functional Configuration> Fig. 14 is a third diagram illustrating an example of the functional configuration of the training device 2000. In Fig. 14, the training device 2000 further includes a feature sequence generator 2100. The feature sequence generator 2100 generates a first feature sequence 120 corresponding to the first partial frame sequence 20 and a second feature sequence 130 corresponding to the second partial frame sequence 30.

[0117] <Example of Hardware Configuration> The hardware configuration of the training device 2000 of embodiment 3 is shown in Fig. 4, for example, similar to the hardware configuration of the training device 2000 of embodiment 1. However, the storage device 1080 of embodiment 3 stores a program that realizes each functional component of the training device 2000 of embodiment 3.

[0118] <Processing Flow> FIG. 15 is a third flowchart illustrating the processing flow executed by the training device 2000. The training device 2000 acquires a training frame sequence 10 (S302). The generation unit 2020 generates a first partial frame sequence 20 and a second partial frame sequence 30 (S304). The feature sequence generation unit 2100 generates a first feature sequence 120 corresponding to the first partial frame sequence 20 and a second feature sequence 130 corresponding to the second partial frame sequence 30 (S306). The feature extraction unit 2040 inputs the first feature sequence 120 to the feature extraction model 100 to calculate the feature 50 for each first video frame 42 (S308). The feature extraction unit 2040 inputs the second feature sequence 130 to the feature extraction model 100 to calculate the feature 50 for each second video frame 44 (S310). The update unit 2060 updates the feature extraction model 100 using the feature quantities 50 for each video frame included in the first extended sequence 80 and the feature quantities 50 for each video frame included in the second extended sequence 90 (S312).

[0119] Combination of Embodiments 2 and 3 The training device 2000 may be configured to generate an extended sequence and generate a feature sequence corresponding to the extended sequence. In this case, the training device 2000 includes both the data extension unit 2080 and the feature sequence generation unit 2100.

[0120] FIG. 16 is a fourth diagram illustrating an overview of the training device 2000. In the example of FIG. 16, both a first extended sequence 80 and a second extended sequence 90 are generated. A first feature sequence 120 corresponding to the first extended sequence 80 is obtained by performing a feature extraction process on the first extended sequence 80. A second feature sequence 130 corresponding to the second extended sequence 90 is obtained by performing a feature extraction process on the second extended sequence 90. The operations performed after the first feature sequence 120 and the second feature sequence 130 are obtained are as described above.

[0121] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0122] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0123] In the present disclosure, a program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0124] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes: (Supplementary Note 1) A training device comprising: a generation means for identifying a plurality of pairs of a first video frame and a second video frame for a training frame sequence consisting of a plurality of video frames, and generating a first partial frame sequence consisting of the first video frame of each of the pairs and a second partial frame sequence consisting of the second video frame of each of the pairs; a feature extraction means for calculating a feature amount for each of the first video frame and a feature amount for each of the second video frame using a feature extraction model configured to extract a feature amount for each video frame included in the frame sequence; and an update means for updating the feature extraction model so as to reduce a difference between the feature amount for the first video frame and the feature amount for the second video frame for each of the pairs, wherein in the training frame sequence, neither the first video frame nor the second video frame included in another pair is present between the first video frame and the second video frame included in the same pair. (Supplementary Note 2) The training device according to Supplementary Note 1, wherein the generating means acquires a sampling interval and an intra-pair interval, identifies each video frame in the training frame sequence that is separated by the sampling interval as the first video frame, and for each of the identified first video frames, identifies a video frame that is located before or after the intra-pair interval from the first video frame as the second video frame to be paired with the first video frame. (Supplementary Note 3) The training device according to Supplementary Note 1, further comprising: data extension means that performs a process of generating a first augmented sequence by performing data extension on the first partial frame sequence, a process of generating a second augmented sequence by performing data extension on the second partial frame sequence, or both of these processes, and wherein the feature extraction means performs a process of inputting the first augmented sequence to the feature extraction model to calculate features for each of the first video frames, a process of inputting the second augmented sequence to the feature extraction model to calculate features for each of the second video frames, or both of these processes.(Supplementary Note 4) The training device of Supplementary Note 3, wherein the feature extraction means: calculates features for each of the first video frames by inputting the first extended sequence into the feature extraction model, and calculates features for each of the second video frames by inputting the second extended sequence into the feature extraction model; calculates features for each of the first video frames by inputting the first extended sequence into the feature extraction model, and calculates features for each of the second video frames by inputting the second partial frame sequence into the feature extraction model; or calculates features for each of the first video frames by inputting the first partial frame sequence into the feature extraction model, and calculates features for each of the second video frames by inputting the second extended sequence into the feature extraction model. (Supplementary Note 5) The training device according to Supplementary Note 1, further comprising: feature sequence generation means for generating a first feature sequence by calculating a plurality of types of feature sequences for each of the first video frames, and generating a second feature sequence by calculating a plurality of types of feature sequences for each of the second video frames, wherein the feature extraction means calculates the feature sequences for each of the first video frames by inputting the first feature sequence to the feature extraction model, and calculates the feature sequences for each of the second video frames by inputting the second feature sequence to the feature extraction model. (Supplementary Note 6) The training device according to Supplementary Note 5, wherein the feature sequence generation means calculates a concatenation of the plurality of feature sequences calculated for the first video frames, and generates time series data of the concatenation generated for each of the first video frames as the first feature sequence, and calculates a concatenation of the plurality of feature sequences calculated for the second video frames, and generates time series data of the concatenation generated for each of the second video frames as the second feature sequence.(Supplementary Note 7) The training device described in Supplementary Note 5 or 6, wherein the multiple types of features calculated for the first video frame are any two or more of image features extracted from the first video frame, skeletal features extracted from the first video frame, features extracted from audio data or acceleration data corresponding to the first video frame, and features extracted from text data representing the content of the first video frame; and the multiple types of features calculated for the second video frame are any two or more of image features extracted from the second video frame, skeletal features extracted from the second video frame, features extracted from audio data or acceleration data corresponding to the second video frame, and features extracted from text data representing the content of the second video frame. (Supplementary Note 8) The training device described in Supplementary Note 1, wherein the updating means calculates one or more interpolated features by performing an interpolation process on a plurality of features for the first video frame calculated using the feature extraction model, calculates one or more interpolated features by performing an interpolation process on a plurality of features for the second video frame calculated using the feature extraction model, and updates the feature extraction model so that a difference between the interpolated features calculated from the features for the first video frame and the interpolated features calculated from the features for the second video frame becomes small.(Supplementary Note 9) A computer-executed training method, comprising: a generating step of identifying a plurality of pairs of a first video frame and a second video frame for a training frame sequence consisting of a plurality of video frames, and generating a first partial frame sequence consisting of the first video frame of each of the pairs and a second partial frame sequence consisting of the second video frame of each of the pairs; a feature extraction step of calculating features for each of the first video frames and features for each of the second video frames using a feature extraction model configured to extract features for each video frame included in the frame sequence; and an updating step of updating the feature extraction model so as to reduce a difference between the feature for the first video frame and the feature for the second video frame for each of the pairs, wherein in the training frame sequence, neither the first video frame nor the second video frame included in another pair is present between the first video frame and the second video frame included in the same pair. (Supplementary Note 10) A program that causes a computer to execute the following steps: a generation step of identifying a plurality of pairs of a first video frame and a second video frame for a training frame sequence consisting of a plurality of video frames, and generating a first partial frame sequence consisting of the first video frame of each of the pairs and a second partial frame sequence consisting of the second video frame of each of the pairs; a feature extraction step of calculating features for each of the first video frames and features for each of the second video frames using a feature extraction model configured to extract features for each video frame included in the frame sequence; and an update step of updating the feature extraction model so as to reduce a difference between the features for the first video frame and the features for the second video frame for each of the pairs; wherein, in the training frame sequence, neither the first video frame nor the second video frame included in another pair is present between the first video frame and the second video frame included in the same pair.

[0125] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 8 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Notes 9 and 10 in the same dependency relationship as Supplementary Notes 2 to 8. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods.

[0126] This application claims priority based on Japanese Patent Application No. 2024-034469, filed March 7, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0127] 10 training frame sequence 20 first partial frame sequence 30 second partial frame sequence 40 pair 42 first video frame 44 second video frame 50 feature 55 interpolated feature 60 video data 70 frame sequence 80 first extended sequence 90 second extended sequence 100 feature extraction model 120 first feature sequence 122 combined feature 130 second feature sequence 132 combined feature 200 classification model 1000 computer 1020 bus 1040 processor 1060 memory 1080 storage device 1100 input / output interface 1120 network interface 2000 training device 2020 generation unit 2040 feature extraction unit 2060 update unit 2080 data extension unit 2100 feature sequence generation unit 3000 classification device

Claims

1. A training device comprising: a generation means for identifying a plurality of pairs of a first video frame and a second video frame for a training frame sequence consisting of a plurality of video frames, and generating a first partial frame sequence consisting of the first video frame of each of the pairs and a second partial frame sequence consisting of the second video frame of each of the pairs; a feature extraction means for calculating features for each of the first video frames and features for each of the second video frames using a feature extraction model configured to extract features for each video frame included in the frame sequence; and an update means for updating the feature extraction model so as to reduce the difference between the features for the first video frame and the features for the second video frame for each of the pairs, wherein in the training frame sequence, neither the first video frame nor the second video frame included in another pair exists between the first video frame and the second video frame included in the same pair.

2. The training device of claim 1, wherein the generation means acquires a sampling interval and an intra-pair interval, identifies each video frame in the training frame sequence that is separated from the other video frames by the sampling interval as the first video frame, and for each identified first video frame, identifies a video frame that is located before or after the intra-pair interval from the first video frame as the second video frame that is to be paired with the first video frame.

3. The training device of claim 1, further comprising a data extension means for performing a process of generating a first extended sequence by performing data extension on the first partial frame sequence, a process of generating a second extended sequence by performing data extension on the second partial frame sequence, or both of these processes, and wherein the feature extraction means performs a process of calculating features for each of the first video frames by inputting the first extended sequence into the feature extraction model, a process of calculating features for each of the second video frames by inputting the second extended sequence into the feature extraction model, or both of these processes.

4. The training device of claim 3, wherein the feature extraction means: calculates features for each of the first video frames by inputting the first extended sequence into the feature extraction model, and calculates features for each of the second video frames by inputting the second extended sequence into the feature extraction model; calculates features for each of the first video frames by inputting the first extended sequence into the feature extraction model, and calculates features for each of the second video frames by inputting the second partial frame sequence into the feature extraction model; or calculates features for each of the first video frames by inputting the first partial frame sequence into the feature extraction model, and calculates features for each of the second video frames by inputting the second extended sequence into the feature extraction model.

5. A training device as described in claim 1, further comprising a feature sequence generation means for generating a first feature sequence by calculating multiple types of features for each of the first video frames, and for generating a second feature sequence by calculating multiple types of features for each of the second video frames, wherein the feature extraction means calculates the features for each of the first video frames by inputting the first feature sequence into the feature extraction model, and calculates the features for each of the second video frames by inputting the second feature sequence into the feature extraction model.

6. The training device described in claim 5, wherein the feature sequence generation means calculates a concatenation of multiple features calculated for the first video frame and generates time series data of the concatenation generated for each of the first video frames as the first feature sequence, and calculates a concatenation of multiple features calculated for the second video frame and generates time series data of the concatenation generated for each of the second video frames as the second feature sequence.

7. The training device described in claim 5 or 6, wherein the multiple types of features calculated for the first video frame are any two or more of image features extracted from the first video frame, skeletal features extracted from the first video frame, features extracted from audio data or acceleration data corresponding to the first video frame, and features extracted from text data representing the content of the first video frame; and the multiple types of features calculated for the second video frame are any two or more of image features extracted from the second video frame, skeletal features extracted from the second video frame, features extracted from audio data or acceleration data corresponding to the second video frame, and features extracted from text data representing the content of the second video frame.

8. The training device of claim 1, wherein the updating means calculates one or more interpolated features by performing an interpolation process on a plurality of features for the first video frame calculated using the feature extraction model, calculates one or more interpolated features by performing an interpolation process on a plurality of features for the second video frame calculated using the feature extraction model, and updates the feature extraction model so that a difference between the interpolated features calculated from the features for the first video frame and the interpolated features calculated from the features for the second video frame becomes small.

9. A computer-implemented training method comprising: a generation step of identifying a plurality of pairs of a first video frame and a second video frame for a training frame sequence consisting of a plurality of video frames, and generating a first partial frame sequence consisting of the first video frame of each of the pairs and a second partial frame sequence consisting of the second video frame of each of the pairs; a feature extraction step of calculating features for each of the first video frames and features for each of the second video frames using a feature extraction model configured to extract features for each video frame included in the frame sequence; and an update step of updating the feature extraction model so as to reduce differences between the features for the first video frame and the features for the second video frame for each of the pairs, wherein in the training frame sequence, neither the first video frame nor the second video frame included in another pair is present between the first video frame and the second video frame included in the same pair.

10. A non-transitory computer-readable medium storing a program, the program causing a computer to execute the following steps: a generation step of identifying a plurality of pairs of a first video frame and a second video frame for a training frame sequence consisting of a plurality of video frames, and generating a first partial frame sequence consisting of the first video frame of each of the pairs and a second partial frame sequence consisting of the second video frame of each of the pairs; a feature extraction step of calculating feature amounts for each of the first video frames and feature amounts for each of the second video frames using a feature extraction model configured to extract feature amounts for each video frame included in the frame sequence; and an update step of updating the feature extraction model so as to reduce, for each of the pairs, a difference between the feature amount for the first video frame and the feature amount for the second video frame; wherein, in the training frame sequence, neither the first video frame nor the second video frame included in another pair is present between the first video frame and the second video frame included in the same pair.