Video splitting device and its program

Abstract

To provide a video division device which can accurately divide a news program video forwarded for each item by an announcer into items.SOLUTION: A video division device 1 comprises: cut point detection means 12 which detects a cut point from a video; image feature calculation means 13 which calculates an image included in the cuts as input to a CNN for image classification and output of an intermediate layer of the CNN as an image feature for each of the cuts divided at the cut point; cut score calculation means 14 which calculates a score indicating a degree being an image of a lead section from the image feature for each image by using an image discrimination model and calculates the average score for each cut as a cut score; and section specification means 15 which determines whether a video section of the cut is the lead section or a non-lead section on the basis of the cut score and specifies the lead section and the non-lead section subsequent to it as the same item section.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a video dividing device and a program for dividing a video into items. [Background technology]

[0002] In order to make effective use of the vast amount of video assets they have accumulated, broadcasting stations are working to add various pieces of information to their videos that can be used as search keywords. News program videos often have information such as summaries and titles added to each "news item," so there is a need to introduce technology that can automatically divide news program videos into items.

[0003] A method has been disclosed in which topic changes are detected from closed captions (text) associated with relative times from the start of a program by using the similarity of segment features that represent the frequency of occurrence of word sequences, and then video is divided (see Patent Document 1). Also, a method has been disclosed for detecting points where topics change and segmenting video based on similarities between frames, where the statistical quantity of pixel values ​​on a frame-by-frame basis is used as an image feature (see Patent Document 2). In addition, a method has been disclosed that uses the similarity of segment features consisting of video features and audio features to detect, for each news item, repetitions from video between segments (video) in which a newscaster (announcer) introduces the content of the item and video segments in which correspondents report from the field (see Patent Document 3). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5231130 [Patent Document 2] International Publication No. 2009 / 031398 [Patent Document 3] Japanese Patent Application Laid-Open No. 2000-285243 Summary of the Invention [Problem to be solved by the invention]

[0005] The technique disclosed in Patent Document 1 requires text corresponding to the program audio, so the audio must be accurately transcribed in advance. The methods disclosed in Patent Documents 2 and 3 detect topic change points using statistical quantities such as histograms as image features. However, because statistical quantities generally lack sufficient information as image features, this method has the problem of sometimes being unable to correctly detect topic change points.

[0006] The present invention has been made in consideration of the problems of the conventional technology, and aims to provide a video division device and a program therefor that can accurately divide video that is presented item by item by a speaker such as an announcer. [Means for solving the problem]

[0007] In order to solve the above problem, the video segmentation device of the present invention is a video segmentation device that segments a video composed of multiple items, including lead sections, which are video sections in which a speaker introducing the content of the item is shown, and non-lead sections, which are video sections that represent the content of the item, by item, and is configured to include a cut point detection means, an image feature calculation means, a cut score calculation means, and a section identification means.

[0008] In such a configuration, the video dividing device detects cut points indicating camera changes or the like from the video by the cut point detection means. The video segmentation device then inputs the images (frame images) included in each cut of the video segmented by the cut points into a pre-trained convolutional neural network for image classification using an image feature calculation means, and calculates the output of the intermediate layer of the convolutional neural network for image classification (CNN) as image features. The output of the intermediate layer of the convolutional neural network for image classification contains a variety of information such as the color and shape of the image.

[0009] The video segmentation device then uses the image discrimination model to calculate an image score from the image features using the cut score calculation means, and calculates the average image score for each cut as the cut score. The image discrimination model is a learning model such as a pre-trained support vector machine or neural network that outputs the degree to which an image is an image of a lead section as an image score. In this way, the cut score calculation means calculates a score for each image and averages it within a cut, so that the score (cut score) becomes an index indicating the degree to which the cut is an image of a lead section.

[0010] Then, the video division device uses a section identification means to determine whether the video section of the cut is a lead section or a non-lead section based on the cut score, and identifies the lead section and the subsequent non-lead section as the same item section. In addition, the section identification means detects a non-audio section longer than a predetermined time length from the audio corresponding to the video of the lead section, and identifies the lead section immediately before the non-audio section as an item section not accompanied by a non-lead section. This allows the video segmentation device to accurately determine the type of cut and identify the section of the item based on the various image features of the images that make up the cut using CNN. The video division device can be operated by a video division program that causes a computer to function as each of the above-mentioned means. [Effects of the Invention]

[0011] According to the present invention, video composed of various items such as news programs and information programs can be accurately divided into item sections each consisting of a video section in which a speaker introduces the content of an item and a video section that follows that showing the content of the item. As a result, the present invention can improve the efficiency of the task of adding information for each item to video of a news program or the like. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a configuration of a video division device according to an embodiment of the present invention; [Figure 2] FIG. 2 is an explanatory diagram for explaining the processing of the cut point detection means in FIG. [Figure 3] FIG. 2 is an explanatory diagram for explaining the processing of the image feature calculation means and the cut score calculation means of FIG. [Figure 4] FIG. 2 is an explanatory diagram for explaining the processing of the section identification means in FIG. [Figure 5] 3 is a flowchart illustrating an operation of the video division device according to the embodiment of the present invention. [Figure 6] 1 is a block diagram showing the configuration of an image discrimination model generation device that generates an image discrimination model used in a video segmentation device according to an embodiment of the present invention. [Figure 7] 7 is a flowchart showing the operation of the image discrimination model generating device of FIG. 6. DETAILED DESCRIPTION OF THE INVENTION

[0013] <Configuration of video splitter> First, with reference to FIG. 1, the configuration of a video dividing device according to an embodiment of the present invention will be described.

[0014] The video division device 1 divides a video made up of multiple items, including a video section showing a speaker such as an announcer or presenter introducing the content of the item, and a video section showing the content of the item, into individual items. The video input to the video segmentation device 1 is video made up of various items such as news programs, information programs, etc. For example, in the case of news program video, the items are made up of individual news stories, and in the case of information program video, the items are made up of individual topics. Here, the input video is assumed to be a news program video, and the speaker is assumed to be an announcer. As shown in Figure 1, the video segmentation device 1 includes an image classification CNN storage means 10, an image discrimination model storage means 11, a cut point detection means 12, an image feature calculation means 13, a cut score calculation means 14, and a section identification means 15.

[0015] The image classification CNN storage means 10 stores a CNN (convolutional neural network) for image classification. The image classification CNN storage means 10 can be configured with a general storage medium such as a semiconductor memory. This image classification CNN storage means 10 stores an image classification CNN that has been trained in advance to classify the content of images. This image classification CNN can be one trained using a known method. For example, the image classification CNN can be the CNN described in Reference 1 below, which is trained using ImageNet (http: / / www.image-net.org / ), an open data set. A detailed explanation will be omitted here.

[0016] Reference 1: C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the Inception Architecture for Computer vision,” Proc. 2016 IEEE Conf. on Computer Vision and Pattern Recognition, pp.2818-2826, 2016

[0017] The image classification CNN outputs image features from intermediate layers (multiple convolution layers, pooling layers, etc.). Here, the image classification CNN is referenced when calculating image features in the CNN calculation means 131 described later.

[0018] The image discrimination model storage means 11 stores a learning model for image discrimination (image discrimination model). The image discrimination model storage means 11 can be configured with a general storage medium such as a semiconductor memory. The image discrimination model is, for example, a support vector machine, a neural network, etc. The image discrimination model storage means 11 stores a pre-trained image discrimination model that discriminates, from image features, whether or not an image is an image in which an announcer introduces the content of an item.

[0019] The images in which the announcer introduces the content of an item are individual frame images that make up the video section in which the announcer reads the news while on screen. Hereinafter, the video section in which the announcer introduces the content of an item will be referred to as the "lead section," and the images that make up the lead section will be referred to as "lead section images," while other video sections will be referred to as "non-lead sections," and the images that make up the non-lead sections will be referred to as "non-lead section images." The image features input to the image discrimination model are the image features output from the intermediate layer of the image classification CNN.

[0020] The output of the image discrimination model is a score indicating the degree (likelihood) that an image having the input image features is a lead section image. For example, the image discrimination model outputs a score in the range of "0" to "1" so that the value approaches "1" when the degree of likelihood that the image is a lead section image is high, and the value approaches "0" when the degree of likelihood that the image is a non-lead section image is high. The method for generating this image discrimination model will be described later with reference to FIGS.

[0021] The cut point detection means 12 detects cut points, such as camera changes and editing points, from the input video (news program video). Note that a general method can be used to detect cut points from video. For example, the method disclosed in Japanese Patent Application Laid-Open No. 2008-33749 can be used. Here, the cut point detection means 12 divides the input image into a cut sequence V(1), V(2), V(3), ..., V(Nc) for each cut point. Note that a cut is a video divided by a cut point, and Nc indicates the number of cuts.

[0022] That is, as shown in FIG. 2, the cut point detection means 12 detects cut points c from the news program video V, and divides the news program video V into a cut sequence V(1), V(2), . . . at the cut points c. The cut point detection means 12 outputs the time code at the beginning of each of the cuts V(1), V(2), . . . divided into sections for each cut point c to the image feature calculation means 13.

[0023] The image feature calculation means 13 calculates the features of an image. Here, for each cut of a video segmented by cut points, the image feature calculation means 13 inputs the images included in the cut into a convolutional neural network for image classification, and calculates the output of the intermediate layer of the convolutional neural network as the image feature. The image feature calculation means 13 includes an image sampling means 130 and a CNN calculation means 131.

[0024] The image sampling means 130 samples the image of each cut of the input news program image at predetermined time intervals in the time direction. The image sampling means 130 samples images at equal intervals (for example, 1 second intervals) from the first frame identified by the time code at the beginning of each cut for each cut V(1), V(2), .... In this way, the image sampling means 130 generates a sequence of sampled images P(k,1), P(k,2), ..., P(k,Np(k)) for each cut V(k) (k=1, 2, ..., Nc). Note that Np(k) indicates the number of images sampled from cut V(k).

[0025] That is, as shown in Figure 3(a), the image sampling means 130 generates an image sequence P(1,1), P(1,2), ... in cut V(1), and an image sequence P(2,1), P(2,2), ... in cut V(2). The same applies to the other cuts. Note that Figure 3 shows only the first image of the cut out of the image sequence P(1,1), P(1,2), ..., etc. The image sampling means 130 outputs an image sequence of sampled images for each cut to the CNN calculation means 131.

[0026] The CNN calculation means 131 extracts image features from the sampled images for each cut sampled by the image sampling means 130. The CNN calculation means 131 inputs image data for each image into the image classification CNN stored in the image classification CNN storage means 10, and calculates the output of the intermediate layer of the image classification CNN as image features (feature vectors). In this way, the CNN calculation means 131 extracts image features from the intermediate layer of the image classification CNN, thereby making it possible to extract features that include various types of information, such as color and shape within the image. The CNN calculation means 131 inputs an image sequence P(k,1), P(k,2), ..., P(k,Np(k)) of sampled images for each cut, and sequentially extracts a feature sequence f(k,1), f(k,2), ..., f(k,Np(k)) of image features from the intermediate layer of the image classification CNN.

[0027] That is, as shown in Fig. 3(b), the CNN calculation means 131 sequentially inputs the image sequence P(1,1), P(1,2), ... of the sampled images of cut V(1) into the image classification CNN, and sequentially extracts the feature sequence f(1,1), f(1,2), ... from the intermediate layer. The same applies to cut V(2) and onwards. The CNN calculation means 131 outputs the image features of the extracted sampled images for each cut to the cut score calculation means 14.

[0028] The cut score calculation means 14 calculates a score (cut score) for determining whether or not a cut is a lead section image. Here, the cut score calculation means 14 uses an image discrimination model to calculate an image score, which is the degree to which an image is a lead section image, from image features, and calculates the average score of the image scores for each cut as the cut score. The cut score calculation means 14 includes an image score calculation means 140 and an image score averaging means 141 .

[0029] The image score calculation means 140 uses the image discrimination model to calculate, from the image features calculated by the image feature calculation means 13, the image scores of the sampled images having the image features. The image score calculation means 140 inputs the image features into the image discrimination model stored in the image discrimination model storage means 11, and calculates the image score as the output value of the image discrimination model. The image score calculation means 140 inputs the feature series f(k,1), f(k,2), ..., f(k,Np(k)) of image features into an image discrimination model and calculates the score series s(k,1), s(k,2), ..., s(k,Np(k)) of image scores.

[0030] 3(c), the image score calculation means 140 sequentially inputs the feature sequence f(1,1), f(1,2), ... of cut V(1) into the image discrimination model, and sequentially calculates the score sequence s(1,1), s(1,2), ... as output values ​​of the image discrimination model. The same applies to cut V(2) and onwards. The image score calculation means 140 outputs a score series of image scores calculated from the image features of the sampled images for each cut to the image score averaging means 141.

[0031] The image score averaging means 141 averages the image scores of the sampled images calculated by the image score calculation means 140 for each cut. The image score averaging means 141 averages the image score series s(k,1), s(k,2), ..., s(k,Np(k)) to calculate the kth cut score S(k), as shown in the following equation (1).

[0032]

number

[0033] That is, the image score averaging means 141 calculates the cut score S(1) by averaging the score series s(1,1), s(1,2), ... of cut V(1), as shown in Fig. 3(d). The same applies to cut V(2) and onwards.

[0034] In this way, the cut score is an average of the image scores within a cut, and therefore serves as an index showing the degree to which the cut is an image of the lead section. Furthermore, by averaging the image scores, the cut score becomes a stable index for determining that the image is a lead section, even if the frame image suddenly changes due to noise or the like. The image score averaging means 141 outputs the time code at the beginning of each cut and the cut score to the section identification means 15.

[0035] The section identification means 15 determines whether the video section of a cut is a lead section or a non-lead section based on the cut score, and identifies the lead section and the subsequent non-lead section as the same item section. The section specifying means 15 includes a lead candidate section / non-lead section specifying means 150 , a lead candidate section dividing means 151 , and an item section specifying means 152 .

[0036] The lead candidate section / non-lead section identifying means 150 identifies lead candidate sections that are candidates for lead sections in news program video and non-lead sections based on the cut score. The lead candidate section / non-lead section identification means 150 compares the cut score with a predetermined threshold value, starting from the first cut, to determine whether the cut corresponds to a lead section or a non-lead section. This threshold value is, for example, the midpoint between the maximum and minimum values ​​of the output of the image discrimination model.

[0037] When the lead candidate section / non-lead section specifying means 150 determines that a cut is a cut that corresponds to a lead section, it sets the cut as a lead candidate section. Furthermore, when the lead candidate section / non-lead section identifying means 150 determines that a cut corresponds to a non-lead section, it further determines whether the subsequent cuts correspond to non-lead sections, and identifies one or more consecutive non-lead sections as one non-lead section. Note that when the input video is news program video, this non-lead section can be considered as a video section in which VTR video or the like without an announcer is played (hereinafter referred to as a VTR section).

[0038] Furthermore, the lead candidate section / non-lead section identifying means 150 determines that the cuts before the cut that was initially determined to be a lead section are unnecessary sections at the beginning of the program. These unnecessary sections correspond to sections that do not correspond to news items, such as the title of the program.

[0039] For example, as shown in Figure 4(a), if S(2) and S(6) are greater than the threshold value among the cut scores S(1), S(2), S(3), ..., S(6), S(7), ..., the lead candidate section / non-lead section identification means 150 determines cuts V(2) and V(6) corresponding to S(2) and S(6) as lead candidate sections. Furthermore, the lead candidate section / non-lead section identification means 150 determines that the cuts V(1), V(3), V(4), V(5), V(7), ... corresponding to cut scores S(1), S(3), S(4), S(5), S(7), ... below the threshold are non-lead sections. However, the cut V(1) before the first lead section (lead candidate section) is determined to be an unnecessary section.

[0040] In addition, the lead candidate section / non-lead section identification means 150 identifies cuts V(3), V(4), V(5), V(7)... as non-lead sections, but identifies consecutive non-lead sections, cuts V(3), V(4), V(5), as a single non-lead section. As a result, the lead candidate section / non-read section identifying means 150 identifies lead candidate sections LC(1), LC(2), . . . and non-read sections (VTR sections) VT(1), VT(2), . The lead candidate section / non-lead section identifying means 150 outputs the time codes of the beginning and end of the lead candidate section to the lead candidate section dividing means 151 .

[0041] The lead candidate section dividing means 151 divides the lead candidate section at non-utterance sections that are longer than a preset threshold value in the lead candidate section. The lead candidate section division means 151 detects non-speech sections from the audio corresponding to the input video (news program video) in the lead candidate section, and if the non-speech section is longer than a preset threshold (e.g., 2 seconds), divides the lead candidate section into individual lead sections within the non-speech section (e.g., at the midpoint of the non-speech section). Furthermore, if no non-speech section longer than the threshold is detected in the lead candidate section, the lead candidate section dividing means 151 identifies the lead candidate section as one lead section. Note that a general method may be used to detect non-speech segments from audio. For example, the speech segment detection method disclosed in Japanese Patent Laid-Open Publication No. 2007-233148 may be used to identify segments other than speech segments as non-speech segments.

[0042] For example, as shown in FIG. 4(b), if no non-speech section longer than the threshold is detected from the lead candidate section LC(1), the lead candidate section division means 151 divides the lead candidate section LC(1) into one lead section L(1,1). Furthermore, if a non-speech section longer than the threshold is detected from the lead candidate section LC(2), the lead candidate section dividing means 151 divides the lead candidate section LC(6) into lead sections L(2,1), L(2,2). As a result, the lead candidate section division means 151 can divide even a unique item in which there is only a lead section and no non-lead section (VTR section) as a single item in the basic configuration in which a lead section and a non-lead section (VTR section) are a single item as a single item. The lead candidate section dividing means 151 outputs the time codes at the beginning of the identified lead sections and non-read sections (VTR sections) to the item section identifying means 152.

[0043] The item section specifying means 152 specifies an item section based on the lead section and non-lead section specified by the lead candidate section dividing means 151 . The item section identification means 152 identifies a pair of a lead section and a non-lead section (VTR section) that is consecutive thereto as one item section. If a lead section is not followed by a non-lead section, the item section identification means 152 identifies the lead section as an item section without a non-lead section. This allows the item section identification means 152 to identify an item section, for example, even if a news program video continues to show the announcer and moves on to the next item without a VTR.

[0044] For example, as shown in FIG. 4(c), in the lead section L(1,1) and the VTR section VT(1) that follows it, the item section identification means 152 identifies the pair of L(1,1) and VT(1) as one item section I(1). Furthermore, when a lead section L(2,1) is followed by a lead section L(2,2), the item section identification means 152 identifies the lead section L(2,1) as one item section I(2). The item section identification means 152 outputs information that identifies an item section, for example, the time codes of the start and end points of the item section, as item information.

[0045] With the above-described configuration, the video dividing device 1 can divide a video (news program video) made up of various items into each item. It is possible. The video division device 1 can be operated by a program (video division program) that causes a computer to function as each of the above-mentioned means.

[0046] <Video Splitter Operation> Next, the configuration of the video segmentation device according to the embodiment of the present invention will be described with reference to Fig. 5 (see Fig. 1 for the configuration as appropriate). It is assumed that the image classification CNN storage means 10 stores a pre-trained image classification CNN, and the image discrimination model storage means 11 stores a pre-trained image discrimination model.

[0047] In step S1, the cut point detection means 12 detects cut points, which are video switching points, from the input video (news program video). In step S2, the image sampling means 130 of the image feature calculation means 13 samples images at equal intervals from the leading frame for each cut divided by the cut points detected in step S1. In step S3, the CNN calculation means 131 of the image feature calculation means 13 inputs image data to the image classification CNN stored in the image classification CNN storage means 10 for each image (sampled image) in the cut sampled in step S2, and calculates the output of the intermediate layer of the image classification CNN as an image feature (feature vector).

[0048] In step S4, the image score calculation means 140 of the cut score calculation means 14 uses the image discrimination model stored in the image discrimination model storage means 11 to calculate a score (image score) from the image features calculated in step S3 indicating the degree to which the sampling image having the image features is a lead section image. In step S5, the image score averaging means 141 of the cut score calculation means 14 calculates a cut score indicating the degree to which the cut is an image of the lead section by averaging the image scores of the sampled images for each cut calculated in step S4.

[0049] In step S6, the lead candidate section / non-lead section identification means 150 of the section identification means 15 identifies lead candidate sections that are candidates for lead sections based on the cut score calculated in step S5, and identifies one or more consecutive non-lead sections as one non-lead section (VTR section). In step S7, the lead candidate section division means 151 of the section identification means 15 detects a non-speech section longer than a preset threshold from the audio corresponding to the input video (news program video) in the lead candidate section identified in step S6.

[0050] Here, if the lead candidate section includes a non-speech section longer than the threshold (Yes in step S8), in step S9, the lead candidate section dividing means 151 divides the lead candidate section into non-speech sections and sets them as multiple lead sections. On the other hand, if the lead candidate section does not include a non-utterance section longer than the threshold value (No in step S8), in step S10, the lead candidate section dividing means 151 sets the lead candidate section as one lead section.

[0051] In step S11, the item section identification means 152 of the section identification means 15 identifies an item section from the lead section set in steps S9 and S10 and the non-lead section (VTR section) identified in step S6. That is, the item section identification means 152 identifies a pair of a lead section and its adjacent VTR section as one item section. Alternatively, if a VTR section is not adjacent to the lead section, the item section identification means 152 identifies the lead section as an item section without a VTR section. Through the above operations, the video dividing device 1 can divide a video (news program video) made up of various items into each item.

[0052] <Configuration of the image discrimination model generation device> Next, the configuration of the image discrimination model generating device will be described with reference to FIG. The image discrimination model generating device 2 generates an image discrimination model to be used in the video segmenting device 1 (see FIG. 1). The image discrimination model generating device 2 uses a large amount of lead section images P P (P P (1), P P (2),…,P P (N P )) and non-lead section image PN (P N (1), P N (2),…,P N (N N )) are used as training data to generate an image discrimination model through training.

[0053] Lead section image P P is an image in which an announcer introduces the content of an item. For example, when generating an image discrimination model to be used for a news program video, the lead section image P P The model can use frame images that make up video segments in which a single announcer is present reading the news. By using images with multiple announcers presenting the content of an item as training data, it is possible to generate an image discrimination model that can handle cases in which multiple announcers are presenting the news. Non-lead section image P N is the lead section image P P Any image other than the above will suffice, and when generating an image discrimination model to be used in news program video, frame images constituting video sections other than the video section in which the announcer is on screen reading the news can be used.

[0054] As shown in FIG. 6, the image discrimination model generating device 2 includes an image classification CNN storage means 10, an image discrimination model storage means 11, an image feature calculation means 20, and an image discrimination model learning means 21.

[0055] The image classification CNN storage means 10 stores a CNN (convolutional neural network) for image classification. The image classification CNN storage means 10 can be configured with a general storage medium such as a semiconductor memory. The image classification CNN stored in this image classification CNN storage means 10 is the same as the image classification CNN described with reference to FIG.

[0056] The image discrimination model storage means 11 stores a model for image discrimination, for example, an SVM (Support Vector Machine). The image discrimination model storage means 11 can be configured with a general storage medium such as a semiconductor memory. The image discrimination model stored in the image discrimination model storage means 11 is learned by the image discrimination model learning means 21 .

[0057] The image feature calculation means 20 calculates the lead section image P P and non-lead section image P N The image features are calculated from the The image feature calculation means 20 inputs image data for each image into the image classification CNN stored in the image classification CNN storage means 10, and calculates the output of the intermediate layer of the image classification CNN as an image feature (feature vector). That is, the image feature calculation means 20 performs the same processing as the CNN operation means 131 described in FIG. The image feature calculation means 20 calculates the lead section image P P From the lead section image feature f P (f P (1), f P (2),…,f P (N P )) and output it to the image discrimination model learning means 21. Furthermore, the image feature calculation means 20 calculates the non-lead section image P N From the non-lead section image feature f N (f N (1), f N (2),…,f N (N N )) and output it to the image discrimination model learning means 21.

[0058] The image discrimination model learning means 21 uses the lead section image feature f calculated by the image feature calculation means 20. P is a positive example, and the non-lead section image feature f N The image classification model is trained using the negative examples. Here, the image discrimination model learning means 21 uses the lead section image feature f P When input, the score is "1" and the non-lead section image feature f N The parameters of the image discrimination model are trained so that when the input is "0", the score is output. This allows the image discrimination model to learn to output a score in the range of "0" to "1" for the input image features. With the above-described configuration, the image discrimination model generating device 2 can generate an image discrimination model to be used in the video segmenting device 1.

[0059] <Operation of the image discrimination model generation device> Next, the operation of the image discrimination model generation device will be described with reference to Fig. 7 (see Fig. 6 for the configuration as appropriate). It is assumed that the image classification CNN storage means 10 stores a trained image classification CNN.

[0060] In step S20, the image feature calculation means 20 calculates the lead section image P P and non-lead section image P N Enter the following in order. In step S21, the image feature calculation means 20 calculates the learning data (the lead section image P P or non-lead section image P N ) is input to the image classification CNN stored in the image classification CNN storage means 10, and the output of the intermediate layer of the image classification CNN is calculated as the image feature (feature vector).

[0061] Here, the learning data input in step S20 is the lead section image P P If so (Yes in step S22), in step S23, the image discrimination model learning means 21 P is input to the image discrimination model. Then, the image discrimination model learning means 21 updates the parameters of the image discrimination model in the image discrimination model storage means 11 so that the discrimination result of the image discrimination model becomes a lead section image (score of, for example, "1").

[0062] On the other hand, the learning data input in step S20 is the non-lead section image P N If so (No in step S22), in step S24, the image discrimination model learning means 21N is input to the image discrimination model. Then, the image discrimination model learning means 21 updates the parameters of the image discrimination model in the image discrimination model storage means 11 so that the discrimination result of the image discrimination model becomes a non-lead section image (score, for example, "0").

[0063] In step S25, the image discrimination model learning means 21 determines whether the input of learning data has been completed or whether learning has been completed based on whether the amount of change in the parameter has become equal to or less than a predetermined threshold value. If the learning has not yet finished (No in step S25), the image discrimination model generating device 2 returns to step S20 and continues the operation. On the other hand, if the learning is completed (Yes in step S25), the image discrimination model generation device 2 ends its operation.

[0064] This enables the image discrimination model generating device 2 to generate a learning discrimination model that can discriminate whether an image is a lead section image or a non-lead section image based on the image features output from the intermediate layer of the image classification CNN.

[0065] The above describes the configuration and operation of the video segmentation device 1 according to an embodiment of the present invention, as well as the configuration and operation of the image discrimination model generation device 2 that generates the image discrimination model used in the video segmentation device 1, but the present invention is not limited to this embodiment. For example, in this case, it is possible to divide an item that is made up of only lead sections and does not have non-lead sections (VTR sections). However, if it is determined that the items of the input video are always composed of pairs of lead sections and non-lead sections, the video division device 1 may be configured so that the lead candidate section / non-lead section identification means 150 functions as a means for identifying lead sections and non-lead sections (lead section / non-lead section identification means), and the lead candidate section division means 151 may be omitted.

[0066] Also, here, the video segmentation device 1 and the image discrimination model generation device 2 are configured separately, but the video segmentation device 1 may be configured to include the image discrimination model generation device 2 inside. [Explanation of symbols]

[0067] 1. Video splitter 10 Image classification CNN memory means 11 Image discrimination model storage means 12 Cut point detection means 13 Image feature calculation method 130 Image Sampling Method 131 CNN calculation method 14 Cut score calculation method 140 Image score calculation method 141 Image Score Averaging Method 15 Section Identification Method 150 Lead candidate section / non-lead section identification means 151 Lead candidate section division method 152 Item section identification means 2. Image discrimination model generation device 20 Image feature calculation method 21 Image discrimination model learning method

Claims

1. A video dividing device that divides a video made up of a plurality of items, including a lead section that is a video section in which a speaker introducing the content of the item is shown and a non-lead section that is a video section that shows the content of the item, by item, a cut point detection means for detecting a cut point from the video; an image feature calculation means for inputting, for each cut of the video divided by the cut points, an image included in the cut into a convolutional neural network for image classification, and calculating an output of an intermediate layer of the convolutional neural network as an image feature; a cut score calculation means for calculating an image score indicating the degree to which the image is an image of a lead section from the image features using an image discrimination model, and calculating an average score for each cut of the image scores as a cut score; a section specifying means for determining whether a video section of the cut is the lead section or the non-lead section based on the cut score, and specifying the lead section and the subsequent non-lead section as the same item section; The video division device is characterized in that the section identification means detects a non-audio section longer than a predetermined length of time from the audio corresponding to the video of the lead section, and identifies the lead section immediately before the non-audio section as an item section that does not include the non-lead section.

2. 2. The video division device according to claim 1, wherein the video is a news program video, the lead section is a video section in the video in which an announcer introduces the news, and the non-lead section is a video section of a VTR after the announcer introduces the news in the video.

3. 3. The video division device according to claim 1, wherein the section specifying means specifies consecutive non-read sections as a non-read section of one item.

4. The image feature calculation means an image sampling means for sampling frame images at predetermined time intervals for each cut; a CNN calculation means for inputting the sampled frame image, performing calculations on the convolutional neural network, and setting the output of a predetermined intermediate layer as the image feature; 4. The video dividing device according to claim 1, further comprising:

5. The cut score calculation means an image score calculation means for inputting the image features calculated by the image feature calculation means into the image discrimination model and calculating the image score; an image score averaging means for averaging the image scores for each cut to calculate the cut score; 5. The video dividing device according to claim 1, further comprising:

6. The image discrimination model is trained by inputting images constituting a video section in which one or more speakers introducing the content of an item are shown into the convolutional neural network, and outputting image features from the intermediate layer as positive examples, and inputting images constituting a video section that represents the content of the item into the convolutional neural network, and outputting image features from the intermediate layer as negative examples.

7. A video division program for causing a computer to function as the video division device according to any one of claims 1 to 6.

Images