Video synchronization device and method

Abstract

A video synchronization device according to the present invention comprises: a training unit that, with respect to original training data including, as a pair, a feature amount of a first original video and a feature amount of a second original video, both the feature amounts including one-cycle operation, performs repetition on the feature amount of the second original video to generate synthetic training data in which the pair is synthesized; and an estimation unit that estimates the number of times of repetition performed on the feature amount of the second original video in the synthetic training data, and generates the same number of synchronization results as the estimated number of times by synchronizing the feature amount of the second original video and the feature amount of the first original video the estimated number of times.

Description

Video synchronization device and method

[0001] The present invention relates to a video synchronization device and method for synchronizing two videos.

[0002] Conventionally, there are known techniques for synchronizing two videos, such as Temporal Alignment (hereinafter abbreviated as "TA").

[0003] Specifically, TA includes techniques for achieving synchronization based on hand-crafted features and techniques for achieving synchronization through deep learning.

[0004] Techniques for achieving synchronization based on hand-crafted features are not robust to changes in the background or viewpoint.

[0005] A technique for achieving synchronization through deep learning is disclosed, for example, in Non-Patent Document 1. In Non-Patent Document 1, the same timing between two videos is learned in a self-supervised manner by cycle-consistency loss.

[0006] An outline of a conventional technique for achieving synchronization through deep learning will be described with reference to FIGS. 4 and 5. FIG.

[0007] FIG. 4 is a block diagram showing an example of the configuration of a conventional TA device to which deep learning is applied.

[0008] FIG. 5 is a diagram illustrating the flow of processing by a conventional TA device.

[0009] As shown in FIG. 4, a conventional TA device 100 includes a learning device 120 and an estimating device 130 .

[0010] The learning device 120 includes a learning unit 122 to which learning data is input, and a learning model database 124. The learning model database 124 is used in common by the learning device 120 and the estimation device 130.

[0011] The learning data is input data to the learning unit 122 and consists of a pair of two synchronized original videos. For example, if original video S has N frames and original video T has M frames among two synchronized original videos S and T, the learning data is represented as original video pair a in Fig. 5. Note that each box in Fig. 5 represents a frame.

[0012] The learning unit 122 includes a feature extractor F(·) 123 to which a model for extracting a feature pair b from an original video pair a is applied. The model may be, for example, a model (e.g., ResNET) configured with a 2D CNN or the like. The feature extractor F(·) 123 uses such a model to extract features for each frame from the original video pair a and obtain a feature pair b.

[0013] The learning unit 122 further uses the feature pair b to

[0014]

[0015] Calculate the cycle-consistency loss, i.e.,

[0016]

[0017] The model parameters are updated by backpropagation using the cross-entropy loss between y and y. The updated model parameters are stored in the training model database 124.

[0018] The estimation side device 130 includes an estimation unit 132. The estimation unit 132 also uses the feature extractor F(·) as described above to output feature quantities U and V using the test data and the model parameters stored in the learning model database 124. Then, using the feature quantities U and V, u ｉ v that shows the largest α for ｊ are regarded as having the same timing, thereby generating synchronized video and the like (see FIG. 5).

[0019] Dwibedi et al, “Temporal Cycle-Consistency Learning”. In CVPR, 2019.

[0020] However, in the above-mentioned conventional technology, it is assumed that the video contains one cycle of movement, and therefore if video showing multiple cycles of movement is input during estimation, it is not possible to achieve correct synchronization.

[0021] An example of an event in which multiple cycles of motion are included in a video is walking.

[0022] FIG. 6 is a diagram showing walking, which is an example of an event that includes multiple cycles of motion.

[0023] In walking, a cycle of right foot contact and left foot contact is continuously repeated multiple times (N cycles).

[0024] That is, in the case of walking, the two synchronized original images, original image S and original image T, may not be synchronized over one cycle each as shown in FIG. 6(a), but may be synchronized over one cycle of original image S and two cycles of original image T as shown in FIG. 6(b).

[0025] FIG. 7 is a diagram illustrating feature pairs in the case of walking.

[0026] In TA, when synchronizing a pair of two original videos S and T, the synchronization is achieved by determining that frames with similar features correspond to each other.

[0027] Therefore, when the original video S of one cycle and the original video T of two cycles are synchronized by TA, as shown in FIG. 7, in the feature pair, the left foot u ｉ and the left foot v of the feature V Ｍ If it is considered that the timing is the same, the left foot v ｊ etc. will be skipped.

[0028] As described above, conventional techniques are based on the assumption that the video contains only one cycle of motion, and therefore, if the video contains multiple cycles of motion during estimation, proper synchronization cannot be achieved.

[0029] The present invention has been made in consideration of the above circumstances, and aims to provide a video synchronization device and method that can synchronize two videos even when the video includes multiple cycles of motion during estimation.

[0030] In order to achieve the above object, a first aspect of the present invention is a video synchronization device for synchronizing between a first original video and a second original video, comprising: a learning unit that generates synthetic training data in which pairs of features of the first original video and features of the second original video, both of which include one-cycle operations, are combined by applying repetitions to the features of the second original video; and an estimation unit that estimates the number of repetitions applied to the features of the second original video in the synthetic training data, and generates the same number of synchronization results as the estimated number of repetitions by synchronizing the features of the second original video and the features of the first original video with the estimated number of repetitions.

[0031] According to the present invention, even when an image including a plurality of cycles of motion is input during estimation, it is possible to synchronize two images.

[0032] FIG. 1 is a block diagram showing an example of the configuration of a video synchronization device to which a video synchronization method according to an embodiment of the present invention is applied. FIG. 2 is a diagram for explaining an example of the flow of learning processing in a learning device. FIG. 3 is a diagram for explaining an example of the flow of estimation processing in an estimation device. FIG. 4 is a block diagram showing an example of the configuration of a conventional TA device to which deep learning is applied. FIG. 5 is a diagram for explaining the flow of processing by a conventional TA device. FIG. 6 is a diagram showing walking, which is an example of an event that includes multiple cycles of motion. FIG. 7 is a diagram illustrating feature pairs for walking.

[0033] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc., are not necessarily the same as those in reality. Furthermore, even when the same part is shown, the dimensions and ratios may be different depending on the drawing. In this specification and each drawing, elements similar to those described above with reference to the previous drawings are designated by the same reference numerals, and detailed and redundant explanations will be omitted as appropriate.

[0034] FIG. 1 is a block diagram showing an example of the configuration of a video synchronization device to which a video synchronization method according to an embodiment of the present invention is applied.

[0035] The video synchronization device 10 is a device for synchronizing a first original video and a second original video, and includes a learning device 20 and an estimation device 30.

[0036] The learning device 20 includes an input unit 22, a learning unit 24, and a learning model database 26. The learning model database 26 is shared by the learning device 20 and the estimation device 30.

[0037] The estimating device 30 includes an input unit 32 and an estimating unit 34 .

[0038] The learning unit 24 generates synthetic learning data in which the pair is synthesized by repeating the feature of the second original video from original learning data that includes a pair of feature of a first original video and feature of a second original video, both of which include one cycle of operation.

[0039] The estimation unit 34 estimates the number of repetitions applied to the features of the second original video in the synthetic training data, and generates the same number of synchronization results as the estimated number by synchronizing the features of the second original video with the features of the first original video for the estimated number of times. Specifically, when estimating the number of repetitions and synchronizing the features of the second original video with the features of the first original video for the estimated number of times, the estimation unit 34 extracts features for each cycle from all the features of the second original video, including the features of the original second original video that have been repeated and the features of the second original video generated by the repetition, up to the estimated number of cycles, and generates the same number of synchronization results as the estimated number by synchronizing the features of the cycle with the features of the first original video for each extracted cycle.

[0040] The video synchronization device 10 having such a configuration can be realized by a computer such as a PC. A computer generally comprises components such as a processor, memory, and storage device, but since computers having these components are well known, detailed explanations and illustrations of the general configuration of a computer will be omitted in this specification.

[0041] FIG. 2 is a diagram illustrating an example of the flow of the learning process in the learning device.

[0042] As illustrated in Fig. 2, during learning in the learning device 20, an original feature pair c is used, which is a pair of feature U and feature V that express two synchronized original videos, original video S and original video T, as one cycle of motion. In the original feature pair c, the feature U of the original video S is expressed as {u １ , u ２ ,...u Ｎ}, the feature value V of the original image T is {v １ , v ２ ,...v Ｍ}.

[0043] During learning, the learning unit 24 repeats the feature V of such an original feature pair c L times. L may be an integer or a decimal. The feature V after the repeat process is then combined with the feature U. In this way, the learning unit 24 obtains combined training data, such as the combined feature pair d illustrated in FIG. 2. Note that the combined feature pair d illustrated in FIG. 2 is, as an example, for the case where L=2.

[0044] The type of extension for repeating is not limited, and for example, rotation, scaling, cropping, etc. can be used. The number of repeats is also not limited. The learning unit 24 can perform L repeats, for example, while changing the extension method.

[0045] The learning unit 24 stores the obtained composite feature pair d in the learning model database 26 .

[0046] The combined feature pair d stored in the learning model database 26 is used by the estimation side device 30 during estimation.

[0047] FIG. 3 is a diagram illustrating an example of the flow of the estimation process in the estimating device.

[0048] The number of repetitions (L) performed by the learning unit 24 is unknown to the estimation device 30. Therefore, as illustrated on the left side of Fig. 3 , it is unknown to the estimation device 30 how many cycles of feature V for the original video T are included in the composite feature pair d used as input.

[0049] Therefore, the estimation unit 34 first predicts the number of repetitions performed on the original feature pair c until the synthesized feature pair d is obtained. The prediction method for this prediction is not limited, and the estimation unit 34 can use, for example, a time-series DNN model such as LSTM.

[0050] After predicting the number of repetitions, the estimation unit 34 divides the feature value V repeated in the learning unit 24 by the predicted number of repetitions. Then, each divided feature value V is synchronized with the feature value U.

[0051] For example, if the predicted number of repetitions is two (N=2), the estimation unit 34 extracts the feature amounts of the first cycle and the feature amounts of the second cycle from the feature amounts V repeated by the learning unit 24. Then, as shown on the right side of FIG. 3 , the feature amounts V (={v1, v2, ..., vj, ..., vM / 2}) of the first cycle are synchronized with the feature amounts U (={u1, u2, ..., ui, ..., uN}), and the feature amounts V (={vM / 2+1, vM / 2+2, ..., vk, ..., vM}) of the second cycle are synchronized with the feature amounts U (={u1, u2, ..., ui, ..., uN}).

[0052] In this way, the video synchronizer 10 generates a plurality of types of synchronization results between the original video S and the original video T.

[0053] In this way, according to a video synchronization device to which the video synchronization method of this embodiment is applied, when synchronizing two videos, if one of the videos contains a repeated portion, it is possible to estimate the number of times it is repeated and generate multiple types of synchronization results between the two videos based on the estimated number of times.

[0054] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0055] REFERENCE SIGNS LIST 10 Video synchronization device 20 Learning device 22 Input unit 24 Learning unit 26 Learning model database 30 Estimation device 32 Input unit 34 Estimation unit 100 TA device 120 Learning device 122 Learning unit 124 Learning model database 130 Estimation device 132 Estimation unit a Original video pair b Feature pair c Original feature pair d Synthesized feature pair F Feature extractor L Number of repetitions S Original video T Original video U Feature V Feature

Claims

1. A video synchronization device for synchronizing between a first original video and a second original video, comprising: a learning unit that generates synthetic training data in which the pair of features of the first original video and the second original video, both of which include one cycle of operation, is synthesized by repeating the features of the second original video; and an estimation unit that estimates the number of repetitions applied to the features of the second original video in the synthetic training data, and generates the same number of synchronization results as the estimated number of repetitions by synchronizing the features of the second original video and the features of the first original video with the estimated number of repetitions.

2. The video synchronization device described in claim 1, wherein the estimation unit estimates the number of repetitions, and when synchronizing the features of the second original video and the features of the first original video for the estimated number of times, extracts features from all features of the second original video, including the features of the original second original video to which the repeat has been applied and the features of the second original video generated by the repeat, for each cycle up to the same number of cycles as the estimated number, and generates a number of synchronization results equal to the estimated number of times by synchronizing the features of that cycle with the features of the first original video for each extracted cycle.

3. A video synchronization device as described in claim 1 or 2, further comprising a learning model database that stores the synthetic learning data generated by the learning unit, and the estimation unit acquires the synthetic learning data from the learning model database.

4. A video synchronization method for synchronizing between a first original video and a second original video, wherein a processor of a video synchronization device executes the steps of: generating synthesized training data in which original training data containing a pair of features of the first original video and a pair of features of the second original video, both of which include one-cycle operations, is repeated on the features of the second original video; and estimating the number of repetitions and synchronizing the features of the second original video and the features of the first original video the estimated number of times, thereby generating the same number of synchronization results as the estimated number.

Images