Summary video generation device and program

Abstract

To provide a summary video generating device capable of generating a summary video of higher quality than conventional ones, wherein a video section to be extracted as a summary video is learned in relation to adjacent sections and the entire video.SOLUTION: A summary video generating device 1 is provided with a feature vector calculating unit 10 that calculates, for each video section, a plurality of modal feature vectors for a plurality of time scales including the video section, a video section importance calculating unit 20 that calculates the importance of a relevant video section by using a neural network learned in advance from the plurality of feature vectors for each video section, and a video summarizing unit 40 that cuts out video sections from the video to be summarized up to a predetermined total length in order of importance and concatenates the video sections in a time series to generate a summary video.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a video summary generating device and a program for generating a video summary that summarizes a video. [Background technology]

[0002] In recent years, with the development of social media services, there has been an increase in the distribution of video summaries over the network, primarily for the purpose of promoting broadcast programs and independently produced videos. However, editing video summaries requires a great deal of effort, so there is a demand for technology that can automatically generate video summaries.

[0003] Conventionally, methods for automatically generating a video summary have been proposed in, for example, Patent Documents 1 to 4. The technique described in Patent Document 1 generates a summary video by extracting important video sections from the video based on the image features of key frames of the divided videos obtained by dividing the video. The technique described in Patent Document 2 analyzes a graph in which video segments are nodes and the similarity of video features between nodes is an edge, extracts video segments with high importance from the video, and generates a summary video.

[0004] The technique described in Patent Document 3 divides a video into multiple cuts and calculates scores for multiple elements. This technique then calculates an overall score for the cuts based on the weighting of each element set by the user, extracts cuts with high overall scores, and generates a summary video. The technique described in Patent Document 4 assigns a high score to the feature vectors of video segments used in a video summary and a low score to the feature vectors of video segments not used, and calculates the scores of video segments using a neural network trained to optimize the weight distribution of multiple elements. This technique then extracts video segments with high scores to generate a video summary. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 5537285 [Patent Document 2] Patent No. 5898117 [Patent Document 3] Patent No. 6917788 [Patent Document 4] Patent Publication No. 2021-189967 Summary of the Invention [Problem to be solved by the invention]

[0006] When manually generating a video summary from a video, when determining the importance of a certain video section within the video, the worker generally takes into account not only that section but also neighboring sections and the content of the entire video. However, conventional methods calculate importance based only on the characteristics of individual video segments. As a result, they do not take into account the relationship between the video segment extracted as a summary and the content of neighboring segments or the entire video, and may not be able to extract videos that are highly important in relation to neighboring segments or the entire video. Therefore, when generating a video summary, a more natural video summary generation method similar to that generated by a human has been desired.

[0007] The present invention has been made in response to such demands, and its objective is to provide a video summary generation device and a program therefor that can generate video summary of higher quality than conventional methods by learning the video sections to be extracted as summary videos in relation to neighboring sections and the entire video. [Means for solving the problem]

[0008] In order to solve the above problem, the summary video generation device of the present invention is a summary video generation device that generates a summary video from a video to be summarized that is composed of multiple video segments, and is configured to include a feature vector calculation unit, a video segment importance calculation unit, and a video summarization unit.

[0009] In this configuration, the summary video generation device calculates, for each video section, a plurality of modal (type) feature vectors for a plurality of time scales including that video section, using the feature vector calculation unit. Then, the video summary generation device uses a pre-trained neural network to calculate the importance of each video section from the multiple feature vectors for each video section calculated by the feature vector calculation unit using the video section importance calculation unit. The video summary generating device then uses a video summarizing unit to extract video sections up to a predetermined total length from the video to be summarized, in descending order of importance, and connect these in chronological order to generate a summarized video. This allows the video summary generation device to calculate the importance of a video section in combination with the features of the videos before and after it, and generate a video summary. The summary video generating device can be operated by a summary video generating program that causes a computer to function as each of the above-mentioned units. [Effects of the Invention]

[0010] According to the present invention, the importance of a video section can be calculated not only based on the characteristics of the video section itself but also based on the characteristics of the video sections that follow it. As a result, the present invention can generate a high-quality video summary that takes into account the content of the video. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing the overall configuration of a summary video generation device according to an embodiment of the present invention; [Figure 2] FIG. 2 is a block diagram showing the internal configuration of a feature vector calculation unit in FIG. 1. [Figure 3] 3 is an explanatory diagram for explaining an example of a process for setting multiple time scales in the multiple time scale setting unit of FIG. 2; FIG. [Figure 4] 2 is a block diagram showing the internal configuration of a video section importance calculation unit in FIG. 1. FIG. [Figure 5]5 is a block diagram showing the internal configuration of a multiple-time scale feature integration unit in FIG. 4. FIG. [Figure 6] FIG. 5 is a block diagram showing the internal configuration of a multi-modal feature integrator in FIG. 4. [Figure 7] 2 is a block diagram showing the internal configuration of a video summarizing unit in FIG. 1. FIG. [Figure 8] 3 is a flowchart illustrating an operation of the summary video generation device according to the embodiment of the present invention. [Figure 9] 1 is a block diagram showing the configuration of a video section importance calculation model learning device that learns a video section importance calculation model.

[0023] FIG. [Figure 10] 10 is a flowchart showing the operation of the video section importance calculation model learning device of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] <Configuration of summary video generation device> First, with reference to FIG. 1, the configuration of a video summary generation device according to an embodiment of the present invention will be described.

[0013] The video summary generating device 1 generates a video summary from a video to be summarized that is made up of a plurality of video segments. The video to be summarized may also include audio. Here, the summary video generation device 1 receives as input a summary target video V for which a summary video is to be generated, and the IN points (start points) and OUT points (end points) of a plurality of video sections k (k=1, 2, ..., K; K is the number of video sections set in the summary target video V) into which the summary target video V has been pre-divided.The summary video generation device 1 then extracts video sections of high importance from the summary target video V and outputs a summary video SV.

[0014] A video section can be a section obtained by dividing a video at each cut point, which is an editing point or a camera change that is an image feature of the video to be summarized V. Furthermore, the IN point / OUT point of a video section can be, for example, a start time and duration, as long as the section can be identified. As shown in FIG. 1, the video summary generation device 1 includes a feature vector calculation unit 10, a video section importance calculation unit 20, a parameter storage unit 30, and a video summarization unit 40.

[0015] The feature vector calculation unit 10 calculates, for each video section of the video V to be summarized, feature vectors of multiple modalities for multiple time scales that include that video section, and calculates multiple feature vectors by time scale and modality. Here, modal refers to the type of feature, such as features related to the type of subject, features related to the face area, features related to camera movement information, features based on speech content, features related to the pitch and volume of the voice, etc.

[0016] Here, the internal configuration of the feature vector calculation unit 10 will be described with reference to FIG. As shown in FIG. 2, the feature vector calculation unit 10 includes a multiple time scale setting unit 11 and multiple modal feature vector calculation units 12 (121, 122, ..., 12 NM [NM〔=N M ] is a modal number ]) and

[0017] The multiple time scale setting unit 11 sets, for each video section k, multiple time scales that include the video section k. The multiple time scale setting unit 11 sets the video section k specified by the IN point / OUT point of the input video section k as the minimum time scale T1, and the entire video V to be summarized as the maximum time scale T NT [NT(=N T ) is the number of time scales to be set, and multiple time scales T1,...,T NT Set.

[0018] An example of setting time scales in the multiple-time scale setting section 11 will now be described with reference to FIG. Here, the video section for which multiple time scales are set is designated as video section k, and the time scale is designated as T K , the total time scale of the video V to be summarized is T W Let's say. The multiple time scale setting unit 11 sets the time scale T1 (first time scale) to the time scale T K Set as. The multiple time scale setting unit 11 sets the time scale T NT (N T th time scale) is the total time scale T of the video V to be summarized. W Set as.

[0019] The multiple time scale setting unit 11 sets the time scale T1<time scale T2<time scale T3<...<time scale T NT Time scales T2, T3, ... are set so that: Here, T2=T K +(T W -T K ) / (N T -1), T3=T K +2(T W -T K ) / (N T -1).

[0020] Time scale T2,T3,…,T NT is basically a section extending equally in the forward and backward directions in the time direction with time scale T1 (video section k) at the center. However, if the start point of the time scale is before the start point of the summary target video V, the multiple-time scale setting unit 11 sets the time scale, for example, from the start point of the summary target video V. Similarly, if the end point of the time scale is after the end point of the summary target video V, the multiple-time scale setting unit 11 sets the time scale, for example, from the end point of the summary target video V. This allows the multiple time scale setting section 11 to set multiple time scales that include video section k for video section k. Returning to FIG. 2, the internal configuration of the feature vector calculation unit 10 will be further described.

[0021] The modal feature vector calculation unit 12 calculates, for each modality, feature vectors of a plurality of time scales for each video section set by the multiple time scale setting unit 11 . Modal feature vector calculation units 121, 122, ..., 12 NM is a set of multiple time scales T1,…,T NTDifferent modal features are calculated for each of the videos.

[0022] For example, the modal feature vector calculation unit 121 can calculate feature vectors relating to the types of subjects appearing in the video using a general object recognition network such as VGG (Visual Geometry Group) 19. Furthermore, for example, the modal feature vector calculation unit 122 can calculate a feature vector relating to a face area shown in the video. Also, for example, the modal feature vector calculation unit 12 NM can calculate the motion information of the camera in the video as a feature vector.

[0023] Of course, the modal feature vector calculation unit 12 may calculate a feature vector from the speech content (text) recognized from the audio corresponding to the video, or may calculate the pitch and volume of the audio corresponding to the video as a feature vector. In addition, the modal feature vector calculation units 121, 122, ..., 12 NM The number of elements of the feature vector calculated by may differ depending on the modal feature.

[0024] The modal feature vector calculation unit 121 calculates the number of time scales N as the first modal feature for the video of the video section k. T The feature vector v s [k,1,1],v s [k,1,2],…,v s [k,1,N T ] is calculated. The modal feature vector calculation unit 122 calculates the number of time scales N as the second modal feature for the video of the video section k. T The feature vector v s [k,2,1],v s [k,2,2],…,v s [k,2,N T ] is calculated. Similarly, the modal feature vector calculation unit 12 NMis the video of video section k, N M The number of time scales as the Nth modal feature T The feature vector v s [k,N M ,1],v s [k,N M ,2],…,v s [k,N M ,N T ] is calculated.

[0025] The feature vector calculation unit 10 includes modal feature vector calculation units 121, 122, . . . , 12 NM The feature vector v calculated by s [k,m,n](m=1,…,N M , n=1,…,N T ) is defined as a feature vector set V for video segment k. SET [k] to the video section importance calculation unit 20. Returning to FIG. 1, the description of the configuration of the summary video generation device 1 will be continued.

[0026] The video section importance calculation unit 20 uses a pre-trained neural network to calculate the importance of each video section from a plurality of feature vectors (feature vector set) for that video section. The video section importance calculation unit 20 calculates the feature vector set V calculated by the feature vector calculation unit 10. SET For [k] (k=1, 2, ..., K; K is the number of video segments set for the video V to be summarized), the importance of the video segment is calculated by integrating feature vectors of multiple time scales for each video segment using the internal parameters of the trained neural network stored in the parameter storage unit 30, and further integrating feature vectors across multiple modalities.

[0027] Here, the internal configuration of the video section importance calculation unit 20 will be described with reference to FIG. As shown in FIG. 4, the video section importance calculation unit 20 includes a plurality of multiple time scale feature integration units 21 (211, 212, . . . , 21 NM [NM〔=N M] is the modal number]), and multiple multi-layer fully connected NN calculation units 22 (221, 222, ..., 22 NM ), a multi-modal feature integration unit 23, and a multi-layer fully connected NN calculation unit 24.

[0028] The multiple time scale feature integration unit 21 uses a trained one-dimensional convolutional neural network (1DCNN) to convert, for each modality, feature vectors of multiple time scales corresponding to video segments into vectors of predetermined multiple channels, and generates a time scale integrated feature vector that integrates the feature vectors of the multiple time scales by extracting the maximum value for each corresponding element. For example, the multiple time scale feature integration unit 211 may SET Among [k], the first modal feature is the feature vector v s [k,1,1],v s [k,1,2],…,v s [k,1,N T ] to generate the time-scale integrated feature vector v′[k,1]. Similarly, the multiple time scale feature integration unit 212 integrates the feature vector set V SET Among [k], the second modal feature is the feature vector v s [k,2,1],v s [k,2,2],…,v s [k,2,N T ] to generate the time-scale integrated feature vector v′[k,2]. In addition, the multiple time scale feature integration unit 21 NM is the feature vector set V SET Among [k], N M The feature vector v is the th modal feature. s [k,N M ,1],v s [k,N M ,2],…,v s [k,N M ,N T ] to obtain the time-scale integrated feature vector v′[k,N M ] is generated.

[0029] Here, the internal configuration of the multiple-time scale feature integration unit 21 will be further described with reference to FIG. As shown in FIG. 5, the multiple-time scale feature integration unit 21 includes a one-dimensional CNN calculation unit 210 and an element-specific maximum value extraction unit 211.

[0030] The one-dimensional CNN calculation unit 210 performs calculations of a first one-dimensional convolutional neural network (1DCNN) using trained parameters of the first one-dimensional convolutional neural network (1DCNN) stored in the parameter storage unit 30 (FIG. 1) for multiple feature vectors for each modal. The one-dimensional CNN calculation unit 210 calculates the m-th modal feature, which is a D[m]-dimensional feature vector v s [k,m,1],v s [k,m,2],…,v s [k,m,N T ] is input. Note that the D[m] dimension is the number of dimensions of the m-th modal feature (the number of elements in the feature vector).

[0031] The one-dimensional CNN calculation unit 210 calculates the input vector length D[m] and the number of input channels N T , filter size is 1, and the number of output channels is C T As, v s [k,m,1],v s [k,m,2],…,v s [k,m,N T ], and then perform 1DCNN calculations. The number of output channels is C T is a predetermined number equal to or greater than 2, such as 16, 32, etc. In this way, by providing multiple output channels, multiple vectors are generated in which elements are multiplied by different weights. Here, the one-dimensional CNN calculation unit 210 calculates a D[m]-dimensional C T vectors v s ′[k,m,1],v s ′[k,m,2],…,v s ′[k,m,C T ] is generated and output to the element-specific maximum value extraction unit 211.

[0032] The element-specific maximum value extraction unit 211 extracts the maximum value of the elements from the multiple (C T It extracts the maximum value for each element from a vector of (items). The element-specific maximum value extraction unit 211 extracts the D[m]-dimensional C T vectors v s ′[k,m,1],v s ′[k,m,2],…,v s ′[k,m,C T ], a D[m]-dimensional time scale integrated feature vector v′[k,m] for one channel is generated by extracting the maximum value for each element. The time scale integrated feature vector v′[k,m] generated in this way is a feature vector that reflects the importance of each element. The element-specific maximum value extraction unit 211 outputs the generated time scale integrated feature vector to the multi-layer fully connected NN operation unit 22. Returning to FIG. 4, the internal configuration of the video section importance calculation unit 20 will be further described.

[0033] The multi-layer fully-connected NN calculation unit (first multi-layer fully-connected NN calculation unit) 22 performs multi-layer fully-connected NN calculations on the time scale integrated feature vector v'[k,m] for video section k integrated by the multiple time scale feature integration unit 21, using the trained parameters of the first multi-layer fully-connected neural network (multi-layer fully-connected NN) stored in the parameter storage unit 30 (FIG. 1), to convert the vector into a predetermined number of dimensions.

[0034] A multi-layer fully connected NN is a network that consists of an input layer, multiple hidden layers, and an output layer, with the nodes in each layer fully connected between layers. Note that the number of hidden layers and the number of nodes can be any number, but considering the learning process time described below, it is desirable to limit the number of hidden layers to 2 to 3, and the number of nodes in each hidden layer to around 500 to 1000. The multilayer fully-connected NN in the multilayer fully-connected NN computing unit 22 has the number of input nodes D[m] (the number of dimensions of the time scale integrated feature vector v'[k,m]) and the number of output nodes is a predetermined number (D).

[0035] The multi-layer fully connected NN calculation unit 22 includes multiple time scale feature integration units 211, 212, ..., 21 NM Corresponding to the above, multi-layer fully connected NN calculation units 221, 222, ..., 22 NM It consists of: The multi-layer fully connected NN operation unit 221 converts the time scale integrated feature vector v'[k,1] generated by the multiple time scale feature integration unit 211 into a D-dimensional time scale integrated feature vector v[k,1]. Similarly, the multi-layer fully connected NN operation unit 222 converts the time scale integrated feature vector v′[k,2] generated by the multiple time scale feature integration unit 212 into a D-dimensional time scale integrated feature vector v[k,2]. In addition, the multi-layer fully connected NN calculation unit 22 NM is a multiple time scale feature integration unit 21 NM The time scale integrated feature vector v′[k,N M ] is a D-dimensional time-scale integrated feature vector v[k,N M ].

[0036] As a result, the multi-layer fully connected NN calculation units 221, 222, ..., 22 NM can convert the number of dimensions of all time-scale integrated feature vectors into D dimensions. Multilayer fully connected NN calculation unit 221, 222,…, 22 NM and output the converted D-dimensional time scale integrated feature vectors to the multi-modal feature integration unit 23.

[0037] The multi-modal feature integration unit 23 uses a trained one-dimensional convolutional neural network (1DCNN) to convert the time scale integrated feature vectors for each modality into vectors of predetermined multiple channels, and generates an integrated feature vector by integrating the time scale integrated feature vectors for each modality by extracting the maximum value for each corresponding element. The multi-modal feature integration unit 23 includes multi-layer fully connected NN calculation units 221, 222, ..., 22 NM From the above, we obtain D-dimensional time scale integrated feature vectors v[k,1],v[k,2],…,v[k,N M] is input, and an integrated feature vector V[k] is generated by integrating the features for each time scale and modality.

[0038] Here, the internal configuration of the multi-modal feature integrator 23 will be further described with reference to FIG. As shown in FIG. 6, the multi-modal feature integration unit 23 includes a one-dimensional CNN calculation unit 230 and an element-specific maximum value extraction unit 231.

[0039] The one-dimensional CNN calculation unit 230 performs calculations of a second one-dimensional convolutional neural network (1DCNN) using trained parameters of the 1DCNN stored in the parameter storage unit 30 (FIG. 1) for multiple time scale integrated feature vectors for each modal. The one-dimensional CNN calculation unit 230 calculates D-dimensional time-scale integrated feature vectors v[k,1], v[k,2], ..., v[k,N M ]. The one-dimensional CNN calculation unit 230 calculates the input vector length D and the number of input channels N. M , filter size is 1, and the number of output channels is C M Let v[k,1],v[k,2],…,v[k,N M ], and then perform 1DCNN calculations. The number of output channels is C M is a predetermined number equal to or greater than 2, such as 16, 32, etc. In this way, by providing multiple output channels, multiple vectors are generated in which elements are multiplied by different weights. Here, the one-dimensional CNN calculation unit 230 calculates a D-dimensional C M Vectors v′[k,1],v′[k,2],…,v′[k,C M ] is generated and output to the element-specific maximum value extraction unit 231.

[0040] The element-specific maximum value extraction unit 231 extracts the maximum value of the elements from the multiple (C M It extracts the maximum value for each element from a vector of (items). The element-specific maximum value extraction unit 231 extracts the D-dimensional C Mvectors v s ′[k,1],v s ′[k,2],…,v s ′[k,C M ], a D-dimensional integrated feature vector V[k] for one channel is generated by extracting the maximum value for each element. The integrated feature vector V[k] generated in this way is a feature vector that reflects the importance of each element. The element-specific maximum value extraction unit 231 outputs the generated integrated feature vector to the multi-layer fully connected NN operation unit 24. Returning to FIG. 4, the internal configuration of the video section importance calculation unit 20 will be further described.

[0041] The multi-layer fully-connected NN calculation unit (second multi-layer fully-connected NN calculation unit) 24 performs multi-layer fully-connected NN calculations on the integrated feature vector V[k] for video section k integrated by the multi-modal feature integration unit 23, using the trained second multi-layer fully-connected neural network (multi-layer fully-connected NN) parameters stored in the parameter storage unit 30 (Figure 1).

[0042] A multi-layer fully connected NN is a network that consists of an input layer, multiple hidden layers, and an output layer, with the nodes in each layer fully connected between layers. Note that the number of hidden layers and the number of nodes can be any number, but considering the learning process time described below, it is desirable to limit the number of hidden layers to 2 to 3, and the number of nodes in each hidden layer to around 500 to 1000. The multilayer fully-connected NN in the multilayer fully-connected NN computing unit 24 has the number of input nodes D (the number of dimensions of the integrated feature vector V[k]) and the number of output nodes 1. The multi-layer fully-connected NN calculation unit 24 outputs the value of the output node calculated by the multi-layer fully-connected NN to the video summarization unit 40 as the importance S[k] of the video segment k. Returning to FIG. 1, the description of the configuration of the summary video generation device 1 will be continued.

[0043] The parameter storage unit 30 stores pre-trained internal parameters of the one-dimensional convolutional neural network (1DCNN) and the multi-layer fully connected neural network (multi-layer fully connected NN) used in the video section importance calculation unit 20. The parameter storage unit 30 can be configured with a general storage medium such as a semiconductor memory. The learning of the internal parameters to be stored in the parameter storage unit 30 will be described later with reference to FIGS.

[0044] The video summarization unit 40 extracts video sections up to a predetermined total length from the video to be summarized in order of importance calculated by the video section importance calculation unit 20, and connects them in chronological order to generate a summarized video.

[0045] Here, the internal configuration of the video summarizing section 40 will be described with reference to FIG. As shown in FIG. 7, the video summarizing unit 40 includes an important section sorting unit 41, a video clipping unit 42, and a video connection unit 43.

[0046] The important section sorting unit 41 sorts the video section identification numbers k (k=1, 2, ..., K) in descending order based on the importance S[k] of the video section k calculated by the video section importance calculation unit 20. Here, the important section sorting unit 41 sorts the video section identification numbers k (k=1, 2, ..., K) in descending order of importance, as follows: k1, k2, ..., k K and rearrange them. The important section sorting unit 41 sorts the video sections by their identification numbers k1, k2, . . . , k K is output to the video extraction unit 311.

[0047] The video cutting unit 42 cuts out video of highly important video sections from the video V to be summarized, based on the identification numbers of the video sections sorted by the important section sorting unit 41, until the total length exceeds a predetermined threshold. The video clipping section 42 clips out videos V[k1], V[k2], . . . corresponding to video segments k1, k2, .

[0048] The video linking unit 43 links the videos cut out by the video cutting unit 42 by rearranging them in time series. The video connection unit 43 outputs the connected video as a summary video SV.

[0049] As described above, the summarized video generation device 1 can identify the importance of a video section based on a plurality of time scale features and a plurality of modal features corresponding to the video section. This allows the summary video generation device 1 to extract video of a video section from the video to be summarized in relation to the video section and neighboring sections or the entire video, thereby realizing processing similar to the manual process of generating a summary video. The summary video generating device 1 can be operated by a computer (not shown) running a summary video generating program that causes the computer to function as each of the above-mentioned units.

[0050] <Operation of summary video generation device> Next, with reference to FIG. 8 (for the configuration, refer to FIGS. 1, 2, and 4 to 7 as appropriate), the operation of the summary video generation device according to the embodiment of the present invention will be described.

[0051] In step S1, the multiple time scale setting unit 11 sets multiple time scales including each video section of the video V to be summarized (see FIG. 3). Hereinafter, steps S2 to S5 are assumed to be performed in parallel for each modality, but they may be performed in order for each modality.

[0052] In step S2, the modal feature vector calculation unit 12 calculates feature vectors of the video of the plurality of time scales set in step S1 for each video section. NM [NM〔=N M ] is the number of modals, and ] is the number of modals. The same applies to steps S3 to S5 below. As a result, for each video section k, a feature vector set VSET [k] (see Figure 2) is calculated.

[0053] In step S3, the multiple time scale feature integration unit 21 performs a one-dimensional convolutional neural network (1DCNN) operation on the feature vectors of the multiple time scales corresponding to the modal among the feature vector set calculated in step S2 using the trained parameters, thereby generating a number of output channels C T A vector is generated according to the following (see Figure 5). This results in a C T vectors are generated.

[0054] In step S4, the multiple time scale feature integration unit 21 extracts the maximum value of each element from the multiple (C T Extract the maximum value element by element from a vector of n (items). As a result, feature vectors of a plurality of time scales are integrated as modal features, and furthermore, an integrated time scale feature vector is generated in which the importance of each element is reflected.

[0055] In step S5, the multi-layer fully-connected NN calculation unit 22 performs calculations of a multi-layer fully-connected neural network (multi-layer fully-connected NN) on the time scale integrated feature vector generated in step S4 using the trained parameters, thereby converting the time scale integrated feature vectors, which have different numbers of dimensions for each modal, into vectors with the same number of dimensions D.

[0056] In step S6, the multi-modal feature integrating unit 23 integrates the time scale integrated feature vectors for each modality generated in step S5 to generate an integrated feature vector. Although not shown in the figure, step S6 performs the same processing as steps S3 and S4. That is, the multi-modal feature integration unit 23 performs a one-dimensional convolutional neural network (1DCNN) operation on the time scale integrated feature vector for each modality generated in step S5 using the trained parameters by the one-dimensional CNN operation unit 230, thereby obtaining the number of output channels C M A vector is generated according to the following (see Figure 6). This results in a C M vectors are generated. Then, the multi-modal feature integration unit 23 extracts multiple (C M By extracting the maximum value for each element from the vector (number of vectors), an integrated feature vector V[k] corresponding to video segment k is generated.

[0057] In step S7, the multi-layer fully-connected NN calculation unit 24 calculates the importance S[k] corresponding to the video section k by performing a multi-layer fully-connected neural network (multi-layer fully-connected NN) calculation on the integrated feature vector generated in step S6 using the learned parameters.

[0058] In step S8, the important section sorting unit 41 sorts the identification numbers k (k=1, 2, ..., K) of the video sections in descending order of importance based on the importance S[k] of the video sections k calculated in step S7. Here, the identification numbers k1, k2, ..., k are sorted in descending order of importance. K Let's say. In step S9, the video clipping unit 42 initializes a variable i (i=1). In step S10, the video cutout unit 42 extracts a video section k from the video V to be summarized. i Image V[k i ] is cut out.

[0059] In step S11, the video clipping unit 42 determines whether the total length of the video sections clipped in sequence in step S10 exceeds a predetermined threshold L or not. Here, if the total length is less than the threshold L (Yes in step S11), the video clipping unit 42 adds 1 to the variable i in step S12, and returns to step S10. On the other hand, if the total length is equal to or greater than the threshold L (No in step S11), the video clipping unit 42 ends the clipping of the video and proceeds to step S13.

[0060] In step S13, the video connection unit 43 generates a summary video SV by rearranging and connecting the videos cut out in step S10 in chronological order. Through the above operations, the video summary generation device 1 can generate a video summary by processing similar to that of a manual video summary.

[0061] <Configuration of a Video Section Importance Calculation Model Learning Device> Next, the configuration of the video section importance calculation model learning device will be described with reference to FIG. The video section importance calculation model learning device 2 learns the internal parameters of a neural network (1DCNN, multi-layer fully connected NN) to be stored in the parameter storage unit 30 of the summarized video generation device 1 (see FIG. 1).

[0062] The video section importance calculation model learning device 2 uses, as learning data, a video V including a video section used in a summarized video. Pos and video V including the video section not used in the summary video. Neg and are used plurally. Video V Pos The summary video was used in Pos video segments (hereinafter referred to as positive example segments). Neg In the summary video, Neg video segments (hereinafter referred to as negative example segments).

[0063] In this learning data, for example, the video sections used when generating a summary video from a self-produced video (video to be summarized) can be set as positive example sections, and other video sections can be set as negative example sections. The video to be summarized and the summary video may be a program video acquired from a broadcast and a summary video of the program video distributed over a network. In this case, the video sections used in the summary video can be matched with the video sections of the program video based on the similarity of the frame images in the summary video. Note that the positive example sections and negative example sections are video sections divided by cut points, etc., just like the video input to the summary video generation device 1.

[0064] As shown in FIG. 9, the video section importance calculation model learning device 2 includes a feature vector calculation unit 50, a feature vector set storage unit 60, a parameter storage unit 70, a video section importance calculation unit 80, and a parameter update unit 90.

[0065] The feature vector calculation unit 50 calculates the feature vector of the video (V Pos ,V Neg For each video section (positive example section, negative example section) of the video, feature vectors for multiple modalities are calculated for multiple time scales that include that video section, and multiple feature vectors for each time scale and modality are calculated. This feature vector calculation unit 50 has the same configuration as the feature vector calculation unit 10 of the summary video generation device 1 (see FIGS. 1 and 2).

[0066] The feature vector calculation unit 50 calculates the feature vector of the image V Pos and the positive example interval p(p=1,2,…,N Pos ;N Pos is the number of positive example intervals) and the IN / OUT points of the positive example interval p. SET Calculate [p]. Furthermore, the feature vector calculation unit 50 calculates the feature vector of the image V Neg and the negative example interval q(q=1,2,…,N Neg ;N Neg is the number of negative example intervals) and the IN / OUT points of the negative example interval q. SET Calculate [q]. The feature vector calculation unit 50 stores in the feature vector set storage unit 60 a feature vector set for the positive example section and a feature vector set for the negative example section, both of which are generated from the training data.

[0067] The feature vector set storage unit 60 stores a plurality of feature vector sets for positive example sections and feature vector sets for negative example sections calculated by the feature vector calculation unit 50. The feature vector set storage unit 60 can be configured with a general storage medium such as a semiconductor memory.

[0068] The parameter storage unit 70 stores the internal parameters of the 1DCNN and multi-layer fully connected NN that are the learning targets. The parameter storage unit 70 can be configured with a general storage medium such as a semiconductor memory.

[0069] The video section importance calculation unit 80 calculates the importance of a video section from the feature vector set calculated by the feature vector calculation unit 50 and stored in the feature vector set storage unit 60 . This video section importance calculation unit 80 has the same configuration as the video section importance calculation unit 20 of the summarized video generation device 1 (see FIGS. 1 and 4). The video section importance calculation unit 80 calculates the N feature vector sets stored in the feature vector set storage unit 60. Pos Randomly select P feature vector sets V from the feature vector sets of positive example intervals. SET [p1],…,V SET [p P ], and calculates the importance S[p1], ..., S[p P ] is calculated.

[0070] Furthermore, the video section importance calculation unit 80 calculates the N feature vector set stored in the feature vector set storage unit 60. Neg Randomly select Q feature vector sets V from the feature vector sets of negative example intervals. SET [q1],…,V SET [q Q ], and calculates the importance S[q1], ..., S[q Q ] is calculated. The video section importance calculation unit 80 outputs the calculated importance of the positive example section and the calculated importance of the negative example section to the parameter update unit 90. It should be noted that every time the video section importance calculation unit 80 is instructed by the parameter update unit 90 to calculate the importance, it repeats the process of randomly selecting a feature vector set and calculating the importance.

[0071] The parameter update unit 90 updates the internal parameters stored in the parameter storage unit 70 based on the importance of the positive example section and the importance of the negative example section calculated by the video section importance calculation unit 80. The parameter update unit 90 initializes the internal parameters stored in the parameter storage unit 70 with random numbers only once at startup.

[0072] The parameter update unit 90 calculates the importance of the positive example sections S[p1], ..., S[p P ], the importance of the negative example interval [q1],…,S[q Q The internal parameters stored in the parameter storage unit 70 are updated so that the difference value d obtained by subtracting the average of [ ] becomes larger than the difference value d calculated in the previous iteration.

[0073]

number

[0074] If the number of updates is less than a predetermined number and the difference value d is less than a predetermined threshold, the parameter update unit 90 instructs the video section importance calculation unit 80 to calculate importance using a new feature vector set. On the other hand, when the number of updates reaches a predetermined number, or when the difference value d becomes equal to or greater than a predetermined threshold, the parameter update unit 90 ends the internal parameter update process. As a result, the internal parameters are learned so that the importance is high for the positive example section and low for the negative example section.

[0075] With the above configuration, the video section importance calculation model learning device 2 can learn the internal parameters of the neural network (1DCNN, multi-layer fully connected NN) used in the summarized video generation device 1. The video section importance calculation model learning device 2 can be operated by a computer (not shown) running a video section importance calculation model learning program that causes the computer to function as each of the above-mentioned units.

[0076] <Operation of the video section importance calculation model learning device> Next, the operation of the video section importance calculation model learning device will be described with reference to FIG. 10 (and as appropriate, with reference to FIG. 9 for the configuration).

[0077] In step S20, the feature vector calculation unit 50 calculates the feature vector V including the video section (positive example section) used in the summary video. Pos Input the video (V Pos ), feature vectors (feature vector sets) for multiple modals are calculated from videos of multiple time scales that include the positive example section, and stored in a feature vector set storage unit 60. In step S21, the feature vector calculation unit 50 calculates a video V including a video section (negative example section) that was not used in the summary video. Neg Input the video (V Neg ), feature vectors (feature vector sets) for a plurality of modals are calculated from videos of a plurality of time scales including the negative example section, and stored in a feature vector set storage unit 60. These steps S20 and S21 may be processed in parallel as shown in FIG.

[0078] In step S22, the parameter update unit 90 initializes the internal parameters stored in the parameter storage unit 70 with random numbers. In step S23, the parameter update unit 90 updates the variable n TR Initialize (n TR =0).

[0079] In step S24, the video section importance calculation unit 80 randomly selects P feature vector sets V from the feature vector sets of the positive example sections calculated in step S20. SET [p1],…,V SET [p P ]Select. In step S25, the video section importance calculation unit 80 calculates importance S[p1], ..., S[p P ] is calculated.

[0080] In step S26, the video section importance calculation unit 80 randomly selects Q feature vector sets V from the feature vector sets of the negative example sections calculated in step S21. SET [q1],…,V SET [q Q ]Select. In step S27, the video section importance calculation unit 80 calculates importance S[q1], ..., S[q Q ] is calculated. Note that steps S24 and S25 and steps S26 and S27 may be processed in parallel as shown in FIG.

[0081] In step S28, the parameter update unit 90 calculates the importance of the positive example sections S[p1], ..., S[p P ], the importance of the negative example interval [q1],…,S[q Q The internal parameters stored in the parameter storage unit 70 are updated so that the difference value d obtained by subtracting the average of [ ] becomes larger than the previous value.

[0082] In step S29, the parameter update unit 90 updates the variable n TR is less than a predetermined number of times (n TR <N TR ), and the difference value d is less than a predetermined threshold value (d <D TH ) and determine whether it is true or false. where n TR <N TR And d <D TH If so (Yes in step S29), in step S30, the parameter update unit 90 updates the variable n TR Then, the process returns to step S24. TR ≧N TR or d≧D TH In this case (No in step S29), the parameter update unit 90 does not update the parameters, and the video section importance calculation model learning device 2 ends the operation.

[0083] This allows the video section importance calculation model learning device 2 to learn the parameters of the neural network model (1DCNN, multi-layer fully connected NN) used in the summarized video generation device 1. [Explanation of symbols]

[0084] 1. Summary video generation device 10 Feature vector calculation unit 11 Multiple time scale setting section 12. Modal feature vector calculation section 20 Video section importance calculation unit 21 Multiple time scale feature integration unit 210 1D CNN calculation unit 211 Element-specific maximum value extraction section 22 Multilayer fully connected NN calculation unit (first multilayer fully connected NN calculation unit) 23 Multimodal feature integration unit 230 1D CNN calculation unit 231 Element-specific maximum value extraction part 24 Multilayer fully connected NN calculation unit (second multilayer fully connected NN calculation unit) 30 Parameter storage section 40 Video Summary Section 41 Important section sorting section 42 Image cutting section 43 Video connection section 5 Video section importance calculation model learning device 50 Feature vector calculation unit 60 Feature vector set storage unit 70 Parameter storage section 80 Video section importance calculation unit 90 Parameter Update Section

Claims

1. A video summary generation device that generates a video summary from a video to be summarized, the video summary generation device comprising: a feature vector calculation unit that calculates, for each video section, feature vectors of a plurality of modals for a plurality of time scales including the video section; a video section importance calculation unit that calculates the importance of each video section from the plurality of feature vectors calculated for each video section by the feature vector calculation unit using a pre-trained neural network; a video summarization unit that extracts video sections from the video to be summarized in descending order of importance up to a predetermined total length and connects them in chronological order to generate the summarized video; The video section importance calculation unit a multiple-time scale feature integration unit that uses a trained first one-dimensional convolutional neural network to convert, for each modality, feature vectors of multiple time scales corresponding to the video segments into vectors of predetermined multiple channels, and extracts a maximum value for each corresponding element to generate an integrated time scale feature vector by integrating the feature vectors of the multiple time scales; a first multi-layer fully-connected NN calculation unit that converts the time scale integrated feature vector into a predetermined number of dimensions for each modality using a trained first multi-layer fully-connected neural network; a multi-modal feature integration unit that uses a trained second one-dimensional convolutional neural network to convert the per-modal time scale integrated feature vector, which has been converted into a predetermined number of dimensions by the first multi-layer fully-connected NN operation unit, into a vector of a predetermined number of channels, and generates an integrated feature vector by integrating the per-modal time scale integrated feature vector by extracting a maximum value for each corresponding element; a second multilayer fully-connected NN calculation unit that calculates the importance from the integrated feature vector using a trained second multilayer fully-connected neural network; A summary video generation device comprising:

2. The video summary generation device according to claim 1, characterized in that the feature vector calculation unit sets a plurality of time scales including the video section as the minimum time scale and the entire video to be summarized as the maximum time scale, and calculates feature vectors of the plurality of modals.

3. A video summary generation program for causing a computer to function as the video summary generation device according to claim 1 or 2.

Images