Scene division device and program

Abstract

To provide a scene dividing device capable of automatically detecting scene dividing points (timings) of an appropriate number and appropriate positions.SOLUTION: A scene dividing device includes a text joining portion, a dividing point detection model, and a scene dividing point detection portion. The text joining portion joins two adjacent sentences of text data divided into sentence units to generate text join data. The dividing point detection model determines whether a scene dividing point exists between two sentences included in the input text join data. The scene dividing point detection portion outputs information on the position of the scene dividing point on the basis of the determination result output by the dividing point detection model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a scene division device and a program. [Background technology]

[0002] There are technologies that divide video content into scenes. One method for automatically determining the timing of scene division is to detect the time when the video changes (cut point) and divide the scene at that cut point. Another method for determining the timing of scene division is to automatically detect silent sections in the video and divide the scene at that point if the silent section continues for a predetermined period of time or more.

[0003] Patent Document 1 describes a technology for automatically detecting the timing of video transitions. Video transitions can be detected based on whether the sum of the absolute values ​​of changes in pixel values ​​between one frame and the next frame for all pixels is equal to or greater than a predetermined threshold.

[0004] Patent Document 2 describes a technique for detecting scene changes by detecting silence.

[0005] Non-Patent Document 1 describes a neural machine translation technique as one of the methods for converting input text into output text. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-105119 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-209958 [Non-patent literature]

[0007] [Non-Patent Document 1] Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, “Neural Machine Translation By Jointly Learning To Align and Translate.”, ICLR 2015 (International Conference on Learning Representations), 2015. Summary of the Invention [Problem to be solved by the invention]

[0008] However, the above-described conventional techniques each have problems.

[0009] When using the method for detecting video cut points described in Patent Document 1, there is a problem in that too many cut points are detected. In a broadcast program, when the video camera switches, all of those points are detected as cut points, but the number of such cut points is far greater than the number of scene division points. In other words, video frequently switches even at times that cannot be considered scene divisions. For example, in the case of a broadcast news program, the part where, for a certain news item, the video captured by a camera in the news studio switches to the video tape footage of that news is detected as a cut point. However, it is often better not to regard such points as scene division points.

[0010] A similar problem occurs when using the silent section detection method described in Patent Document 2. In other words, there are many cases where it is better not to regard a silent state that continues for a predetermined time or longer as a scene division point. For example, this would be the case when a silent section is intentionally included in a scene for the purpose of creating a content.

[0011] The present invention has been made in consideration of the above circumstances, and aims to provide a scene division device and program that can automatically detect an appropriate number of scene division points (timing) at appropriate positions without relying on video. [Means for solving the problem]

[0012] [1] In order to solve the above problem, a scene segmentation device according to one embodiment of the present invention comprises: a text combining unit that combines two adjacent sentences of text data that has been divided into sentence units to generate combined text data; a segmentation point detection model that determines whether a scene segmentation point exists between the two sentences in the input combined text data; and a scene segmentation point detection unit that outputs information on the location of the scene segmentation point based on the determination result output by the segmentation point detection model.

[0013] [2] Furthermore, in one aspect of the present invention, in the above-mentioned scene division device, when a scene division point exists between two sentences in the text-combined data, the division point detection model inserts a symbol indicating that the scene division point exists between the two sentences and outputs the symbol, and the scene division point detection unit outputs information about the position of the scene division point based on the presence or absence of the symbol indicating that the scene division point exists.

[0014] [3] Furthermore, one aspect of the present invention is that, in the above-mentioned scene segmentation device, the text combination unit generates the text combination data so as to include information on the time corresponding to the position between the two sentences, and the scene division point detection unit obtains and outputs information on the time corresponding to the position of the scene division point from the text combination data.

[0015] [4] Furthermore, one aspect of the present invention is that the above-mentioned scene division device further comprises a video division point detection unit that detects video division points for the video content based on the video content associated with the text data, and the scene division point detection unit corrects the time corresponding to the position of the scene division point based on the output from the division point detection model so that it matches the video division point detected by the video division point detection unit, and outputs the time of the corrected scene division point.

[0016] [5] In addition, one aspect of the present invention is a scene segmentation device further comprising a learning unit that performs machine learning of the segmentation point detection model using learning data that includes text consisting of two sentences combined together and information indicating whether the space between the two sentences is a scene segmentation point.

[0017] [6] Another aspect of the present invention is a program that causes a computer to function as a scene division device, which includes a text combining unit that combines two adjacent sentences of text data divided into sentence units to generate text-combined data, a scene division detection model that determines whether the space between the two sentences in the input text-combined data is a scene division point, and a scene division detection unit that outputs information about the location of the scene division point based on the determination result output by the scene division detection model. [Effects of the Invention]

[0018] According to the present invention, the scene division device can automatically detect an appropriate number of scene division points (timing) at appropriate positions without relying on the video, that is, without relying on cut points or the like in the video. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a scene division device according to a first embodiment of the present invention. [Figure 2] 3 is a schematic diagram showing a configuration of time-stamped text data supplied by a time-stamped text supply unit according to the first embodiment and an example of the data. FIG. [Figure 3] FIG. 4 is a schematic diagram showing an example of data resulting from text combination performed by a text combination unit according to the first embodiment based on time-stamped text data. [Figure 4] 3 is a schematic diagram showing the relationship between data that is input to a scene boundary detection model by a scene boundary detection unit according to the first embodiment and data that is output by the boundary detection model. FIG. [Figure 5] 3 is a schematic diagram showing a configuration of learning data supplied by a learning data supplying unit according to the first embodiment and an example of the data. FIG. [Figure 6] 5 is a flowchart showing a processing procedure for performing scene division on one piece of video content by the scene division device according to the first embodiment. [Figure 7] FIG. 10 is a block diagram showing a schematic functional configuration of a scene division device according to a second embodiment. [Figure 8] FIG. 10 is a schematic diagram showing an example of a method for a scene division time output unit according to the second embodiment to determine scene division times. [Figure 9] 1 is a block diagram showing an example of the internal configuration of a scene division device according to first and second embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0020] Next, embodiments of the present invention will be described with reference to the drawings. The scene segmentation devices of the first and second embodiments described below determine points (timing, time) for dividing scenes of video content, regardless of the video or whether or not silent intervals exist. Specifically, the scene segmentation device determines scene division points by analyzing text associated with the video content. This allows the scene segmentation device to detect only more appropriate scene division points, whereas conventional techniques detect too many scene division points.

[0021] The scene division device outputs the time of the scene division point, thereby making it possible to divide the scenes of the video content. For example, it becomes possible to automatically divide the video content into chapters. Dividing the video content into chapters is achieved in two stages: (1) identifying the timing of the scene division, and (2) assigning titles and metadata to the divided scenes. In other words, the scene division device of the first or second embodiment makes it possible to automate the first stage of processing. In other words, it becomes possible to significantly reduce the amount of human work required compared to when scene division points are determined by human judgment.

[0022] [First embodiment] FIG. 1 is a block diagram showing a schematic functional configuration of a scene division device according to this embodiment. The scene division device 1 is a device that divides video content into a plurality of scenes. Specifically, the scene division device 1 outputs time information of scene division points, which are points that divide scenes. The scene division device 1 outputs time information of zero or more scene division points as a result of processing. The time information may be absolute time or relative time (time in which the start time of the content is set to zero (00:00:00)). The scene division device 1 of this embodiment finds scene division points based on text data related to the video content.

[0023] As shown in the figure, the scene segmentation device 1 includes a time-stamped text supply unit 11, a text combination unit 12, a scene segmentation point detection unit 13, a segmentation point detection model 14, a scene segmentation time output unit 15, a learning data supply unit 21, and a learning unit 22. Each of these functional units can be realized, for example, by a computer and a program. Each functional unit also has a storage unit as needed. The storage unit is, for example, a program variable or memory allocated by program execution. Non-volatile storage units such as a magnetic hard disk drive or solid-state drive (SSD) may also be used as needed. At least some of the functions of each functional unit may also be realized as a dedicated electronic circuit rather than a program. The functions of each unit are described below.

[0024] The time-attached text supply unit 11 supplies text data associated with a time. The data supplied by the time-attached text supply unit 11 is, for example, subtitle text data. The subtitle text data is associated with video content data and includes data on the subtitle text itself and data on its presentation time. The presentation time includes at least information on the presentation start time. The presentation time may also include information on the presentation end time. Note that the text includes sentences written in a natural language (Japanese, English, Chinese, etc.).

[0025] The data supplied by the time-attached text supply unit 11 may be data other than subtitle text. For example, it may be text of a recognition result obtained by performing a speech recognition process on a speech (speech) included in video content. The text of the speech recognition result is also associated with the time of the original speech. The data supplied by the time-attached text supply unit 11 may also be data in other forms.

[0026] The text combining unit 12 combines two adjacent sentences of text data divided into sentence units to generate combined text data. Here, the data that the text combining unit 12 processes is the data supplied by the time-attached text supply unit 11. When the text of each sentence is associated with a time, the text combining unit 12 sets the start time of the latter of the two sentences (the second sentence) as the time of the combined data. The start time of the second sentence is a time that corresponds to (or is close to) the time of the connection point between the two sentences to be combined. In other words, the text combining unit 12 generates combined text data so as to include information on the time that corresponds to the position between the two combined sentences.

[0027] The scene division point detection unit 13 outputs information about the positions of scene division points based on the determination results output by the division point detection model 14. Specifically, the scene division point detection unit 13 inputs the combined text data (data in which the text of two sentences is combined) passed from the text combination unit 12 to the division point detection model 14. Then, based on the data output by the division point detection model 14, the scene division point detection unit 13 determines whether or not a scene division point exists between the two sentences that have been input to the division point detection model 14. The scene division point detection unit 13 obtains information about the time corresponding to the positions of the scene division points from the combined text data passed from the text combination unit 12 and outputs it.

[0028] The segmentation detection model 14 is a model that receives input of text consisting of two concatenated sentences and outputs information indicating whether the space between the two sentences is a scene segmentation point. The segmentation detection model 14 is realized, for example, using a neural network and is configured to be able to perform learning using training data. Specifically, the segmentation detection model 14 can be realized using a Transformer or an RNN (Recurrent Neural Network). The segmentation detection model 14 receives input of a string of symbols and outputs a string of symbols. Here, each symbol corresponds to a word, punctuation mark, etc. The string of symbols output by the segmentation detection model 14 may include a symbol indicating a scene segmentation point. The segmentation detection model 14 can be trained in advance using training data. Alternatively, the training unit 22 may perform machine learning on the segmentation detection model using training data supplied by a training data supply unit 21, which will be described later.

[0029] The segmentation point detection model 14 has internal parameters that represent the connection weights of the neural network. The values ​​of these internal parameters can be updated through machine learning. In other words, by performing machine learning using appropriate training data, the segmentation point detection model 14 becomes able to determine whether or not a scene segmentation point exists between two given sentences.

[0030] Specifically, when a scene division point exists between two sentences in the combined text data, the division point detection model 14 inserts a symbol indicating that the scene division point exists between the two sentences and outputs the result. The symbol indicating that the scene division point exists can be, for example, It is a tag that reads: "." If the space between the above two sentences is not a scene division point, no symbol is inserted between the two sentences. When the division point detection model 14 is realized in this way, the scene division point detection unit 13 can output information about the position of a scene division point based on the presence or absence of a symbol indicating a scene division point.

[0031] The scene division time output unit 15 outputs to the outside the time information of the scene division points detected by the scene division point detection unit 13. The time information of the scene division points is based on the time provided by the text combination unit 12. In other words, the time information of the scene division points is based on the time information provided by the time-attached text supply unit 11. The time information of the scene division points may be expressed as absolute time or as relative time based on the start position of the video content with which the text is associated.

[0032] The learning data supply unit 21 supplies learning data for training the segmentation boundary detection model 14. The data supplied by the learning data supply unit 21 is a set of pairs of input data to the segmentation boundary detection model 14 and output data (correct answer data) corresponding to the input data. In other words, the learning data includes a large amount of example data when a scene segmentation boundary exists between two sentences, and a large amount of example data when no scene segmentation boundary exists between the two sentences.

[0033] The learning unit 22 controls the segmentation detection model 14 to perform machine learning using the training data supplied by the training data supply unit 21. As described above, the training data is data that includes text formed by combining two sentences and information indicating whether the space between those two sentences is a scene division point. Specifically, the learning unit 22 inputs input data (text formed by combining two sentences) to the segmentation detection model 14. The learning unit 22 also calculates the difference (loss) between output data calculated by the segmentation detection model 14 based on the input data and information indicating whether the space between those two sentences is a scene division point (correct output data). Then, based on the difference (loss), the learning unit 22 adjusts the values ​​of internal parameters of the segmentation detection model 14 using an error backpropagation algorithm.

[0034] FIG. 2 is a schematic diagram showing the structure and example of time-stamped text data supplied by the time-stamped text supply unit 11. The example of time-stamped text data shown here is subtitle text data associated with a presentation start time. As shown in the figure, the time-stamped text data may be expressed, for example, as data in a table format. The table shown has items for start time and subtitle text. The start time is the presentation start time of the subtitle text in the video content. The start time is expressed, for example, in the format HH:MM:SS (hours, minutes, seconds). The start time may contain information on times less than a second. The start time may be an absolute time or a relative time. The subtitle text is data of the subtitle text to be presented on a screen, etc. Note that in the example shown, the time-stamped text only has a presentation start time, but the time-stamped text may also contain data on a presentation end time.

[0035] Each row in the table in Figure 2 represents one piece of time-stamped text data. In this figure, a row number is assigned to each row of data for convenience. That is, in the data in the first row of the figure, the start time is "00:00:10" (00 hours, 00 minutes, 10 seconds, and so on), and the subtitle text is "That's all for sports." In the data in the second row, the start time is "00:00:15," and the subtitle text is "Next is the weather information." And in the data in the third row, the start time is "00:00:23," and the subtitle text is "It will rain in Tokyo tomorrow."

[0036] In this embodiment, subtitle text is used as the data of the time-stamped text, but data other than subtitle text may also be used. Also, in the example shown in Figure 2, the data of the time-stamped text only has information about the start time, but it may also have both the start time (presentation start time) and the end time (presentation end time).

[0037] FIG. 3 is a schematic diagram showing an example of data resulting from text combining performed by the text combining unit 12 based on the time-stamped text data (FIG. 2). For convenience, each line of this data is assigned a symbol (C1, C2, . . . ). Each line of this data stores time information and combined text data in association with each other. The combined text is obtained by combining (concatenating) two adjacent sentences in the original text data. Specifically, the data C1 in FIG. 3 is obtained by concatenating the text in the first and second lines in FIG. 2. That is, the combined text data in C1 is "That was the sports information up to this point. Next is the weather information." Similarly, the data C2 in FIG. 3 is obtained by concatenating the text in the second and third lines in FIG. 2. Similarly, the data C3 in FIG. 3 is obtained by concatenating the text in the third and fourth lines in FIG. 2. In this way, the text combining unit 12 generates text by combining the n-th sentence and the (n+1)-th sentence of the original text.

[0038] The time in the data in Figure 3 is the start time of the latter sentence (the second sentence) of the two combined sentences. For example, the time in the data in C1 is "00:00:15." This is the same as the start time of the second line in the data in Figure 2.

[0039] 4 is a schematic diagram showing the relationship between the data (input data, before scene division) that the scene division detection unit 13 inputs to the division point detection model 14, and the data (output data, after scene division) that the division point detection model 14 outputs. The same symbols (C1, C2, ...) as in FIG. 3 are assigned to each row of data in FIG. 4. The input data side of the data shown in FIG. 4 is the same data as the data shown in FIG. 3. The division point detection model 14 inputs this input data and outputs output data. Since the division point detection model 14 has been trained in advance, the output data is symbols that represent scene division points. In the illustrated example, the output data in the row indicated by C1 includes a symbol indicating a scene division point, and may include "This is the end of sports. Next is the weather information." On the other hand, the data in lines C2 and C3 do not contain scene division points. In this way, the division point detection model 14, by learning in advance, determines whether or not a scene division point is contained between input sentences, and outputs the determination result. The scene division point detection unit 13 detects scene division points based on the data output by the division point detection model 14.

[0040] The scene division time output unit 15 outputs scene division time information based on the data passed from the scene division point detection unit 13. For example, when the data shown in FIG. 4 is passed, the scene division time output unit 15 recognizes that a scene division point exists in line C1, but that no scene division points exist in lines C2 and C3. The data in line C1 is given the start time of its second sentence, "00:00:15." Therefore, the scene division time output unit 15 outputs this "00:00:15" as the time of the scene division point. The scene division time output unit 15 outputs zero or more scene division point times for one video content.

[0041] Next, an example of training the segmentation detection model 14 will be described. As described above, the training data supply unit 21 supplies training data. The training unit 22 controls the segmentation detection model 14 to perform machine learning using the training data. When the segmentation detection model 14 is implemented using a neural network, the segmentation detection model 14 can be trained using backpropagation. That is, the training unit 22 provides input data contained in the training data to the segmentation detection model 14, and calculates the difference (loss) between the output calculated by the segmentation detection model 14 based on the input data and the correct data contained in the training data. Furthermore, the training unit 22 adjusts the values ​​of the internal parameters of the segmentation detection model 14 using backpropagation based on the calculated difference (loss).

[0042] FIG. 5 is a schematic diagram showing the structure of the learning data supplied by the learning data supply unit 21 and an example of the data. As shown in the figure, the learning data is a set of pairs of input data to the segmentation point detection model 14 and output data (correct answer data) from the segmentation point detection model 14. The input data side of this learning data is text data containing two sentences. Each sentence is a string of words (including symbols such as punctuation marks). Each sentence ends with a period. In other words, on the surface of the input data, the break between the two sentences is clear. On the other hand, the output data side of the learning data is text containing the same two sentences as the corresponding input data. However, on the output data side, a symbol representing a scene division point may be inserted between the two sentences. In the example shown in the figure, the symbol representing a scene division point is It is represented as a tag.

[0043] In Figure 5, for convenience, each line of data is assigned a code (L1, L2, ...). In the data shown by L1, the input data is "The festival is scheduled to run until tomorrow. Here is the next news.", and the output data is "The festival is scheduled to run until tomorrow. "The next news item is..." In other words, there is a division point between these two sentences as the correct answer for the L1 data. In the data shown in L2, the input data is "The festival was held, and fireworks were set off. The festival is scheduled to run until tomorrow.", and the output data is also "The festival was held, and fireworks were set off. The festival is scheduled to run until tomorrow." In other words, there is no division point between these two sentences as the correct answer for the L2 data. Although only two data examples are shown in the figure, the training data contains many such pairs of input data and output data.

[0044] By learning using such learning data, the segmentation detection model 14 obtains information for determining whether two consecutive sentences given to it belong to the same scene or should be divided into different scenes (a scene segmentation point should be inserted). In other words, the segmentation detection model 14 adjusts its internal parameters so that it can make such a determination. In other words, by performing machine learning on the segmentation detection model 14 using learning data, the segmentation detection model 14 learns to be able to determine whether it is appropriate to insert a segmentation point between two sentences.

[0045] The segmentation point detection model 14, which is constructed using a neural network, is a black box, so to speak, and the relationship between individual parameter values ​​and the judgment result of whether or not a segmentation point should be inserted is not simple. However, it is expected that the segmentation point detection model 14 will be able to make such judgments appropriately by learning using a sufficient amount of learning data. As mentioned above, the data input to the segmentation point detection model 14 is data that combines (concatenates) the first and second sentences. Each of the first and second sentences is a string of words (which may include symbols such as punctuation marks). Each word that makes up the word string has characteristics as a natural language expression. Each word is represented, for example, as a word distributed representation using a multidimensional vector. Specifically, the trained segmentation point detection model 14 extracts information from the input data to identify symbols corresponding to scene segmentations. Here, the information extracted by the segmentation point detection model 14 may implicitly include, for example, (1) the degree of likelihood that the first sentence of two consecutive sentences is the sentence immediately before the scene segmentation point, (2) the degree of likelihood that the second sentence of two consecutive sentences is the sentence immediately after the scene segmentation point, and (3) the degree to which the relationship between the expressions contained in the first sentence and the expressions contained in the second sentence of two consecutive sentences (for example, the relationship between the sets of vocabulary contained in each of them) indicates that a scene segmentation point should be inserted between those two sentences.

[0046] 6 is a flowchart showing the procedure of processing for dividing a certain piece of video content into scenes by the scene division device 1. The processing procedure will be explained below with reference to this flowchart.

[0047] First, in step S1, the text combining unit 12 acquires time-stamped text data provided by the time-stamped text supplying unit 11. The time-stamped text data is, for example, subtitle data for one piece of video content. The time-stamped text data has relative time information, with the start of the video content being zero (00:00:00).

[0048] Next, in step S2, the text combining unit 12 generates new data obtained by combining two consecutive sentences of the time-stamped text acquired in step S1. An example of the data generated by the text combining unit 12 is as shown in Fig. 3. In other words, the text combining unit 12 generates data shifted by one sentence from the text supplied by the time-stamped text supply unit 11, such as data combining the first and second sentences, data combining the second and third sentences, data combining the third and fourth sentences, ... (same below).

[0049] Next, in step S3, the scene division point detection unit 13 uses the division point detection model 14 to determine whether or not a division point exists for each piece of data (text consisting of two consecutive sentences) passed from the text combining unit 12. In other words, by inputting each of the two sentences into the division point detection model 14, the scene division point detection unit 13 detects a symbol representing a scene division point between the two sentences in the data output from the division point detection model 14. That is, the scene division point detection unit 13 uses the division point detection model 14 to detect division points.

[0050] Next, in step S4, the scene division time output unit 15 outputs the scene division times. The scene division times are times corresponding to the scene division points detected by the scene division point detection unit 13 in step S3. The scene division time output unit 15 outputs a list of all scene division times of the target video content. For example, in the data C1, C2, and C3 shown in FIG. 4, the symbols indicating the scene division points are exists only in the data C1. In other words, the scene division time in these data is "00:00:15", which is the start time of the second sentence in data C1.

[0051] The scene division time output unit 15 outputs a list of zero or more scene division point times, each of which is a time at which a scene in the original video content is divided.

[0052] [Second embodiment] Next, a second embodiment of the present invention will be described. Note that the following description may omit the matters already described in the previous embodiment. Here, the description will focus on matters unique to this embodiment.

[0053] 7 is a block diagram showing a schematic functional configuration of a scene segmentation device according to this embodiment. As shown in the figure, the scene segmentation device 2 includes a time-stamped text supply unit 11, a text combination unit 12, a scene segmentation point detection unit 13, a segmentation point detection model 14, a scene segmentation time output unit 16, a learning data supply unit 21, a learning unit 22, and a video segmentation point detection unit 31. That is, the scene segmentation device 2 of this embodiment includes a scene segmentation time output unit 16 instead of the scene segmentation time output unit 15 included in the scene segmentation device 1 of the first embodiment. The scene segmentation device 2 also includes a video segmentation point detection unit 31.

[0054] In this embodiment, the functions and operations of the time-stamped text supply unit 11, text combining unit 12, scene division point detection unit 13, division point detection model 14, learning data supply unit 21, and learning unit 22 are the same as those in the first embodiment, so a description of the functions of these units will be omitted here.

[0055] The video division point detection unit 31, which is one of the features of this embodiment, detects video division points for video content based on video content associated with text data. Video division points are also called cut points. Video division points are determined, for example, by determining whether the total absolute value of changes in pixel values ​​between frames is equal to or greater than a predetermined threshold. Note that the technology for determining these video division points is an existing technology.

[0056] Furthermore, the scene division time output unit 16 of this embodiment corrects the time corresponding to the position of a scene division point based on the output from the division point detection model 14 so that it matches the video division point detected by the video division point detection unit 31. The scene division time output unit 16 outputs the time of this corrected scene division point. As a method of correction, the scene division time output unit 16 corrects the time corresponding to the position of a scene division point based on the output from the division point detection model 14 so that it matches the time of the previous video division point that is closest to that time. In other words, the scene division time output unit 16 may correct the time corresponding to the position of a scene division point based on the output from the division point detection model 14 to the time of the previous video division point.

[0057] 8 is a schematic diagram showing an example of a method by which the scene division time output unit 16 of this embodiment determines scene division times. The scene division time output unit 16 determines scene division times, for example, by one of the methods shown in (A) or (B) of this figure. In FIG. 8, the direction of the horizontal axis corresponds to time. Each of these methods (A) and (B) will be described below.

[0058] In FIG. 8(A), the times of the video division points detected by the video division point detection unit 31 are "00:14:30" and "00:14:44." On the other hand, the time of the scene detection point detected by the division point detection model 14 based on the time-stamped text and output by the scene division point detection unit 13 is "00:14:32." In this case, the scene division time output unit 16 outputs the time of the video division point closest to the time of the scene detection point output by the scene division point detection unit 13 as the scene division time. That is, in this example, the scene division time output unit 16 determines the scene division time to be "00:14:30," which is the time of the video division point immediately before "00:14:32." Even when there are multiple scene division points in a single video content, the scene division time output unit 16 determines the scene division times for each of them using the same process.

[0059] In FIG. 8(B), the times of the video division points detected by the video division point detection unit 31 are also "00:14:30" and "00:14:44." On the other hand, the time of the scene detection point detected by the division point detection model 14 based on the time-stamped text and output by the scene division point detection unit 13 is "00:14:40." In this example as well, the scene division time output unit 16 outputs the time of the video division point immediately preceding the scene detection point output by the scene division point detection unit 13 as the scene division time. That is, in this example, the scene division time output unit 16 determines the scene division time to be "00:14:30," which is the time of the video division point immediately following "00:14:40." Even when there are multiple scene division points in a single video content, the scene division time output unit 16 determines the scene division times for each of them using the same process.

[0060] That is, the scene division time output unit 16 may output the time of the video division point immediately preceding the time of the scene detection point output by the scene division point detection unit 13 as the scene division time.

[0061] FIG. 9 is a block diagram showing an example of the internal configuration of the scene division device 1 of the first embodiment or the scene division device 2 of the second embodiment. The scene division device 1 or 2 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices, etc. Input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from / to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.

[0062] At least some of the functions of the scene division device (1 or 2) in each of the above-described embodiments can be realized by a computer and a program. In this case, the functions can be realized by recording a program for realizing the functions on a computer-readable recording medium and loading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within a computer system that serves as a server or client in such cases. The program may be designed to realize some of the functions described above, or may be capable of realizing the functions described above in combination with a program already stored in the computer system.

[0063] Although a number of embodiments have been described above, the present invention can also be embodied in the following modified examples. Note that a number of modified examples may be combined insofar as they are combinable.

[0064] [Variation 1] The time-stamped text data does not necessarily have to be associated with video content, but may be associated with content consisting of only audio, for example.

[0065] [Variation 2] In the above embodiments, tags (symbols, ) is used to indicate the location of a scene division point in the data. Other methods may be used to indicate whether or not a scene division point exists between two sentences. For example, a tag indicating that a scene division point is not present may be used instead. Alternatively, both a tag indicating a scene division point and a tag indicating that a scene division point is not present may be used simultaneously.

[0066] [Variation 3] In all of the above embodiments, the scene division device (1, 2) has a learning data supply unit 21 and a learning unit 22, and is configured to be able to learn the segmentation detection model 14. As a modified example, the scene division device may be configured without the learning data supply unit 21 or the learning unit 22. Even in this case, scene segmentation points can be detected by using a segmentation detection model 14 that has been trained in advance.

[0067] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Industrial Applicability]

[0068] The present invention can be used in industries that handle video content or related information, such as the production of video content, the distribution of video content (broadcasting or communications, etc.), the creation of video content metadata, etc. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0069] 1,2 Scene dividing device 11 Timed text supply unit 12 Text Joiner 13 Scene segmentation detection unit 14 Segmentation point detection model 15,16 Scene division time output section 21 Learning data supply unit 22 Learning Department 31 Video division point detection unit 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. A text combining unit that combines two adjacent sentences of text data that is time-stamped text data associated with video content and is divided into sentence units to generate text combined data; a segmentation point detection model that determines whether a scene segmentation point exists between two sentences in the input text combination data; a scene division point detection unit that outputs information on the positions of scene division points for dividing scenes of the video content based on the determination result output by the division point detection model; Equipped with The segmentation point detection model is a model that has been machine-learned using learning data in which text data consisting of two sentences joined together is input and information indicating whether the space between the two sentences is a scene segmentation point or not is used as a correct answer. Scene dividing device.

2. the segmentation point detection model, when a scene segmentation point exists between two sentences in the text combination data, inserts a symbol indicating a scene segmentation point between the two sentences and outputs the symbol; the scene division point detection unit outputs information about the position of the scene division point based on the presence or absence of the symbol indicating the scene division point. The scene dividing device according to claim 1 .

3. the text combining unit generates the text combining data so as to include information on a time corresponding to a position between the two sentences; the scene division point detection unit acquires and outputs time information corresponding to the positions of the scene division points from the text combined data; 3. A scene dividing device according to claim 1 or 2.

4. a video division point detection unit that detects video division points of the video content based on the video content associated with the text data; Furthermore, the scene division point detection unit corrects the time corresponding to the position of the scene division point based on the output from the division point detection model so that it coincides with the video division point detected by the video division point detection unit, and outputs the time of the corrected scene division point. A scene splitting device according to any one of claims 1 to 3.

5. a learning unit that performs machine learning of the segmentation point detection model using the learning data; The scene splitting device according to any one of claims 1 to 4, further comprising:

6. A program that causes a computer to function as a scene division device described in any one of claims 1 to 5.

Images