Image processing device, image processing method, and program

Abstract

Provided is a video processing device, wherein an information acquisition means acquires raw video and information of sections of the raw video which were used for a digest video. A video searching means searches in the raw video for elements similar to elements included in the sections which were used for the digest video. An importance calculating means calculates the importance of each element, on the basis of the frequency of appearance of each element in the raw video and in the sections which were used for the digest video.

Description

[Technical field]

[0001] The present disclosure relates to processing video data. [Background technology]

[0002] A technology for generating a video digest from a video has been proposed. Patent Document 1 discloses a highlight extraction device that creates a learning data file from training videos prepared in advance and important scene videos specified by a user, and detects important scenes from a target video based on the learning data file. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-022103 A Summary of the Invention [Problem to be solved by the invention]

[0004] Videos included in digest videos of sports and the like include videos that are characteristic of digest videos and other general videos. In the learning of conventional digest video generating devices, learning data was generated based on whether or not a video was used in the digest video, and learning was performed. In other words, learning data was generated and learning was performed by regarding all parts of the raw video that were used in the digest video as correct answers. In this case, general videos were included in both the correct and incorrect classes of the learning data, which was one of the factors that reduced the accuracy of detecting important events from raw video.

[0005] One object of the present disclosure is to enable the generation of a digest video device capable of detecting important events from raw video with high accuracy. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, there is provided a video processing device comprising: an information acquisition means for acquiring a raw video and information on a section of the raw video used for a digest video; a video search means for searching the raw video for an element similar to an element included in the section used in the digest video; an importance calculation means for calculating the importance of each element based on the frequency of appearance of each element in the raw video and in a section used in the digest video; Equipped with 、 The importance level has a higher value as the frequency of appearance of the element in the section used in the digest video is higher than the frequency of appearance of other elements, and the importance level has a higher value as the frequency of appearance of the element in the raw video is lower. .

[0007] In another aspect of the present disclosure, a video processing method includes: Acquire a raw video and information on a section of the raw video used in the digest video; Searching for an element similar to an element included in the section used in the digest video in the raw video; Calculating the importance of each element based on the frequency of appearance of each element in the raw video and in the section used in the digest video death, The importance level has a higher value as the frequency of appearance of the element in the section used in the digest video is higher than the frequency of appearance of other elements, and the importance level has a higher value as the frequency of appearance of the element in the raw video is lower. .

[0008] In yet another aspect of the invention, a program comprising: Acquire a raw video and information on a section of the raw video used in the digest video; Searching for an element similar to an element included in the section used in the digest video in the raw video; Calculating the importance of each element based on the frequency of appearance of each element in the raw video and in the section used in the digest video death, The importance level has a higher value as the frequency of appearance of the element in the section used in the digest video is higher than the frequency of appearance of other elements, and the importance level has a higher value as the frequency of appearance of the element in the raw video is lower. The processing is executed by a computer. Effect of the Invention

[0009] According to the present disclosure, it is possible to generate a digest video device capable of detecting important events from raw video with high accuracy. [Brief description of the drawings]

[0010] [Figure 1] 1 shows a schematic configuration of a digest video generating device. [Diagram 2] An example of a digest video is shown below. [Diagram 3] An example of labeling raw footage when generating training data is shown below. [Figure 4] 1 shows an overall configuration of a training data generating device according to the present embodiment. [Diagram 5] FIG. 2 is a block diagram showing a hardware configuration of the training data generating device. [Figure 6] FIG. 2 is a block diagram showing a functional configuration of the training data generating device. [Figure 7] An example of a calculation formula for importance is shown below. [Figure 8] 13 is a flowchart of a learning data generation process performed by the learning data generation device. [Figure 9] FIG. 11 is a diagram illustrating a specific example of a learning data generation process. [Figure 10] FIG. 11 is a diagram illustrating a specific example of a learning data generation process. [Figure 11] FIG. 11 is a diagram illustrating a specific example of a learning data generation process. [Figure 12] An example of generated training data is shown below. [Figure 13] 1 shows a configuration for learning a digest video generation device. [Figure 14] FIG. 13 is a block diagram showing a configuration of a modified example of the digest video generation device. [Figure 15] FIG. 11 is a block diagram showing a functional configuration of a video processing device according to a second embodiment. [Figure 16] 10 is a flowchart of a process performed by a video processing device according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. First Embodiment [Digest video generating device] First, as a premise, the digest video generation device will be described. FIG. 1 shows a schematic configuration of the digest video generation device. The digest video generation device 200 is connected to a material video database (hereinafter, "database" will also be referred to as "DB") 2. The material video DB2 stores various material videos, that is, moving images. The material videos may be, for example, videos of television programs broadcast by a broadcasting station, or videos distributed via the Internet or the like. Note that the material videos may or may not include audio.

[0012] The digest video generation device 200 generates and outputs a digest video using a portion of the material video stored in the material video DB2. The digest video is a video in which scenes in which some event occurs in the material video are linked in chronological order. The digest video generation device 200 detects an event scene from the material video using a digest video generation model trained by machine learning, and links a plurality of event scenes in chronological order to generate a digest video. The digest video generation model is a model that detects event scenes from the material video, and for example, a deep learning model using a neural network can be used.

[0013] FIG. 2 shows an example of a digest video. In the example of FIG. 2, digest video generating device 200 extracts event scenes A to D included in the material video, and connects them in chronological order to generate a digest video. Event scenes extracted from the material video may be used repeatedly in the digest video depending on their contents. An event scene is composed of one or more frame images corresponding to a scene in which some event occurs in the material video. An event scene is defined by its start point and end point. An event scene may be defined by the length of the event scene instead of the end point.

[0014] [Learning data generation device] (Basic principle) Next, the basic principle of the training data generation device according to this embodiment will be described. The training data generation device is a device that generates training data used for training the digest video generation device 200 (digest video generation model) described above.

[0015] In sports digest videos, important events that are detected as event scenes are often composed of multiple short videos that are implicitly determined. For example, in a soccer goal scene, a video of the player's shot from the moment he scored the goal is often shown with a camera that shoots the entire field from above (hereafter referred to as a "bird's-eye view camera"), followed by a video of the player's goal celebration, or a video of the goal scene shot with a camera at a different angle.

[0016] Moreover, the images contained in the digest video include images that are characteristic of the digest video and other general images. General images are images that are not only contained in the digest video but also commonly contained in scenes other than the digest video.

[0017] Normally, in the training of a digest video generation device, training data is generated based on whether or not a part was used in the digest video, and a digest video generation model is trained. In other words, training data is generated based on all parts of the raw video that were used in the digest video as correct answers, and the digest video generation model is trained. In this case, common images are included in both the correct and incorrect classes of the training data, which is one of the factors that reduces the accuracy of detecting important event scenes, such as soccer goal scenes, from raw video of the entire game.

[0018] Therefore, in this embodiment, the learning data generation device assumes that a video that appears frequently in a portion used in a digest video but appears relatively infrequently in a portion not used in the digest video is a video characteristic of an event scene, and determines a "characteristic element" that defines this characteristic video. A characteristic element is an element included in a section used in a digest video, i.e., an event scene. Here, the characteristic element may be, for example, a single frame image of a video, or multiple consecutive frame images. Also, the characteristic element may be one type, or two or more types.

[0019] FIG. 3 shows an example of labeling of a material video when training data is generated. When training a digest video generation model, a correct answer label is added to the material video to generate training data. In FIG. 3, an event scene 7 indicated by diagonal lines is a scene included in an existing digest video among the material videos. That is, the event scene 7 is a portion where the material video and the existing digest video used in the news and the like match. In the case of the soccer material video shown in FIG. 3, a goal scene shot by an overhead camera, a goal scene shot by a camera behind the goal, a goal performance scene of a player, and the like are detected as event scenes. Specifically, the event scene 7a in FIG. 3 is a goal scene shot by an overhead camera, the event scene 7b is a goal scene shot by a camera behind the goal, and the event scene 7c is a goal performance scene of a player. In addition, scenes 8 before and after the event scenes 7a to 7c are scenes not included in the digest video.

[0020] When generating training data, correct labels are assigned to event scenes detected from video footage. As shown in Figure 3, the usual labeling method assigns the label "1" to frame images of event scenes and the label "0" to frame images of other scenes.

[0021] In contrast, in the labeling method 1 of the present embodiment, a value between "0" and "1" is assigned to each event scene according to the degree to which the event scene is distinctive. This "degree of distinctiveness" corresponds to the "importance" described below. In the example of FIG. 3, the image captured by the camera behind the goal is considered to be a particularly distinctive scene of a soccer goal, so the label "1" is assigned to the event scene 7b, which is a goal scene captured by the camera behind the goal. The label "0.5" is assigned to the event scene 7a, which is a goal scene captured by the overhead camera, and the event scene 7c, which is a goal performance scene of a player. The label "0" is assigned to the other scenes.

[0022] Moreover, in labeling method 2 of the present embodiment, the label "1" is assigned to event scene 7b, which is a goal scene captured by a camera behind the goal, and the label "0" is assigned to other scenes.

[0023] In this way, in this embodiment, the learning data is labeled not simply based on whether the scene is included in the digest video, but based on the degree to which the scene is characteristic of the digest video. This makes it possible to reduce the influence of generic images in the sections used in the digest video. Therefore, by learning using this learning data, it is possible to obtain a highly accurate digest video generation device.

[0024] (Overall composition) Fig. 4 shows the overall configuration of the training data generation device of this embodiment. The training data generation device 100 generates training data from raw footage and digest footage. The raw footage is footage that is the source of the training data. The digest footage is existing digest footage used in news programs and the like. Meanwhile, the training data is data in which a correct answer label is added to the raw footage, as exemplified in Fig. 3.

[0025] (Hardware configuration) 5 is a block diagram showing a hardware configuration of the training data generation device 100. As shown in the figure, the training data generation device 100 includes an interface (IF) 11, a processor 12, a memory 13, a recording medium 14, and a database (DB) 15.

[0026] The IF 11 inputs and outputs data to and from an external device. Specifically, the IF 11 receives raw video that is the source of learning data and existing digest video, and outputs the generated learning data.

[0027] The processor 12 is a computer such as a CPU (Central Processing Unit), and executes a prepared program to control the entire training data generation device 100. Specifically, the processor 12 executes a training data generation process, which will be described later.

[0028] The memory 13 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 13 is also used as a working memory while the processor 12 is executing various processes.

[0029] The recording medium 14 is a non-volatile and non-temporary recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the training data generation device 100. The recording medium 14 records various programs executed by the processor 12. When the training data generation device 100 executes various processes, the programs recorded on the recording medium 14 are loaded into the memory 13 and executed by the processor 12.

[0030] The database 15 temporarily stores raw footage, digest footage, and the like input through the IF 11. The learning data generation device 100 may also include an input unit such as a keyboard and a mouse for an operator to give instructions or input, and a display unit such as a liquid crystal display.

[0031] (Functional configuration) 6 is a block diagram showing a functional configuration of the training data generation device 100. The training data generation device 100 includes a video acquisition unit 21, a video search unit 22, an importance calculation unit 23, and a training data generation unit 24.

[0032] The video acquisition unit 21 acquires the material video and the digest video from the outside. The video acquisition unit 21 compares the material video with the digest video, and generates information on a section in the material video that matches the digest video, i.e., a section used in the digest video (hereinafter also referred to as a "digest video using section"). The video acquisition unit 21 then outputs the material video and information on the digest video using section in the material video to the video search unit 22. The video acquisition unit 21 also outputs the material video to the learning data generation unit 24.

[0033] When the information on the digest video using section itself can be acquired from an external source, the video acquiring unit 21 may acquire the raw video and the information on the digest video using section from an external source and output the information to the video searching unit 22.

[0034] The video search unit 22 searches the raw video for elements similar to elements included in the digest video using section. The "element" here may be one frame image or multiple consecutive frame images. For example, the video search unit 22 extracts multiple elements by clustering the frame images included in the digest video using section. In this case, each element may be a frame image included in each cluster obtained by clustering one frame image or multiple frame images. Furthermore, each element may be the central feature of each cluster obtained by clustering one frame image or multiple frame images.

[0035] Alternatively, the video search section 22 may detect shot change positions from the digest video and extract the video footage captured by each camera as elements.

[0036] Then, video search unit 22 searches for elements similar to each element from the material video, and outputs the elements to importance calculation unit 23. Note that video search unit 22 may detect each element by searching the entire material video from start to finish, or may detect each element by searching a section of the material video other than the section in which the digest video is used.

[0037] Instead of searching for each element from the material video, video search unit 22 may classify the material video into each element. The method of searching and classifying each element may be a method that requires learning, or a method that does not require learning (for example, unsupervised learning or a method that uses an existing model).

[0038] The importance calculation unit 23 calculates the importance of each element based on the frequency of appearance of each element in the raw video and in the digest video usage section. Here, "importance" is a value reflecting whether each element is an element that characterizes a scene included in the digest video, and corresponds to the previously mentioned "degree to which each event scene is distinctive." The larger the importance value, the higher the degree to which the scene is distinctive. The importance calculation unit 23 then calculates the importance of each element and outputs it to the learning data generation unit 24.

[0039] Figure 7 shows an example of a formula for calculating the importance. The importance s(x) can be defined by, for example, formula (1) or formula (2) shown in Figure 7(A) using functions a(x) and b(x). By using formula (1) or formula (2), the importance s(x) becomes a larger value when both functions a(x) and b(x) become larger.

[0040] FIG. 7(B) shows examples of functions a(x) and b(x). Function a(x) is the ratio (called the "first ratio") of the number of elements x in the section used for the digest video to the total number of elements in the section used for the digest video. The function a(x) takes a larger value the greater the frequency of occurrence of element x in the section used for the digest video compared to the frequency of occurrence of other elements. Function b(x) is the ratio (called the "second ratio") of the number of elements x in the section used for the digest video to the number of elements x in the material video. Function b(x) takes a larger value the greater the frequency of occurrence of element x in the section used for the digest video and the less frequently it appears in the material video.

[0041] FIG. 7(C) shows another example of a function. Function a(x) is the same as in FIG. 7(B). Meanwhile, function b'(x) is used in place of function b(x). Function b'(x) is a value obtained by subtracting from 1 the ratio of the number of elements x in the source video to the total number of elements in the source video (called the "third ratio"). The lower the frequency of appearance of element x in the source video, the larger the value of function b'(x).

[0042] The learning data generating unit 24 generates and outputs learning data by labeling the material video based on the importance of each element. Specifically, the learning data generating unit 24 determines the value of the correct label for each element similar to an element included in the digest video using section in the material video as shown in FIG. 3 based on the importance calculated by the importance calculating unit 23, and performs labeling. In one example, the learning data generating unit 24 assigns a label to the element with the highest importance among the multiple elements as "1" and normalizes the importance of the other elements. In another example, the learning data generating unit 24 assigns a label to a predetermined number of elements in descending order of importance as "1" and labels the other elements as "0". In this way, as shown in FIG. 3, learning data is generated in which an appropriate label is assigned to each element similar to an element included in the digest video using section.

[0043] In the above example, the learning data generation unit 24 generates learning data by labeling the raw video based on the importance of each element calculated by the importance calculation unit 23. Alternatively, the importance calculation unit 23 may output the calculated importance of each element directly to an external device. In this case, the external device may generate learning data in the same way as the learning data generation unit 24, or may use the importance of each element for other purposes.

[0044] In the above configuration, the video acquisition unit 21 is an example of an information acquisition means, the video search unit 22 is an example of a video search means, the importance calculation unit 23 is an example of an importance calculation means and an output means, and the learning data generation unit 24 is an example of a learning data generation means.

[0045] (Learning data generation process) 8 is a flowchart of the learning data generation process by the learning data generation device 100. This process is realized by the processor 12 shown in FIG. 5 executing a prepared program and operating as each element shown in FIG.

[0046] First, the video acquisition unit 21 acquires information on the material video and the digest video-using section in the material video (step S21). In one example, the video acquisition unit 21 acquires the material video and the digest video, detects a section where the material video and the digest video match, and acquires information on the digest video-using section. In another example, the video acquisition unit 21 acquires the material video and the information on the digest video-using section from an external source.

[0047] Next, the video search unit 22 determines whether all elements included in the digest video using section have been processed (step S22). If not all elements have been processed (step S22: No), the video search unit 22 determines whether the end of the material video has been processed (step S23). If the end of the material video has not been processed (step S23: No), the video search unit 22 searches for elements in the material video that are similar to elements included in the digest video (step S24). Specifically, the video search unit 22 first detects multiple elements included in the digest video using section by clustering the videos in the digest video using section, and searches the material video for an element similar to one of the elements.

[0048] When the video search unit 22 performs this search up to the end of the material video (step S23: Yes), it executes steps S23 to S24 for other elements included in the digest video using section. When all elements included in the digest video using section have been processed in this manner (step S22: Yes), the importance calculation unit 23 calculates the importance of each element based on the frequency of appearance of each element in the material video and the digest video using section (step S25). Specifically, the importance calculation unit 23 calculates the importance of each element using a formula such as that shown in FIG.

[0049] Next, the learning data generation unit 24 assigns a correct label to the section corresponding to each element of the material video based on the importance of each element, and generates learning data (step S26), and the learning data generation process then ends.

[0050] (Specific examples) Next, a specific example of the learning data generation process will be described. Assume that material videos A and B and a digest video are given as shown in Fig. 9(A). The video acquisition unit 21 matches the material videos A and B with the material videos, and detects a matching section M in the material videos as shown in Fig. 9(B). The matching section corresponds to a section in which the digest video is used.

[0051] Next, as shown in Fig. 10(A), the video search unit 22 clusters the videos of each matching section M to obtain a plurality of clusters. In the example of Fig. 10(A), videos P1 to P9 of the matching section are obtained from material videos A and B, and the video search unit 22 clusters the videos P1 to P9 to obtain five clusters as shown in Fig. 10(B). Each cluster corresponds to an individual element. In Fig. 10(B), cluster 1 includes videos P1 and P3, cluster 2 includes videos P4 and P7, cluster 3 includes video P6, cluster 4 includes video P8, and cluster 5 includes videos P2, P5, and P9.

[0052] Next, video search unit 22 trains a classification model with each cluster shown in Fig. 10(B) as one class. This classification model is a model that classifies input material video into classes 1 to 5.

[0053] Next, as shown in Fig. 11, the video search unit 22 inputs the material video A to the trained classification model 31 and classifies the material video A into five classes. Note that the classification result by the classification model 31 may include videos that do not resemble any of the classes. Note that the material video B is also classified using the classification model 31 in the same manner.

[0054] Next, importance calculation unit 23 calculates the frequency of appearance of each element in the material video and the digest video using section based on the classification result, and calculates the importance of each element using the calculation formula shown in Fig. 7. Learning data generation unit 24 then labels the section of each element in the material video using the importance of each element, and generates learning data. As a result, learning data is generated in which correct labels are assigned to the sections of each element in material video A, as shown in Fig. 12, for example.

[0055] [Learning of digest video generation device] Next, a method for training the digest video generation device (digest video generation model) using the training data generated by the training data generation device 100 will be described. FIG. 13 shows a configuration for training the digest video generation device 200. The digest video generation device 200 is connected to a training unit 41. The training data generated by the training data generation device 100 is input to the digest video generation device 200 and the training unit 41. Specifically, the training data includes material video and correct answer labels assigned to sections of each element of the material video, as exemplified in FIG. 12.

[0056] During learning, the raw video of the learning data is input to the digest video generation device 200, and the correct answer label is input to the learning unit 41. The digest video generation device 200 detects event scenes from the raw video using the digest video generation model, and outputs the detected event scenes to the learning unit 41. The learning unit 41 optimizes the digest video generation model based on the event scenes detected by the digest video generation model and the correct answer labels. In this way, the digest video generation device 200 learns using the learning data generated by the learning data generation device 100.

[0057] The learning data generated by the learning data generation device 100 of this embodiment is labeled not simply based on whether it is used in the digest video, but based on the importance indicating whether it is an element that characterizes the scene used in the digest video. Therefore, the digest video generation device 200 that learns using the learning data reduces the influence of general images included in the sections used in the digest video, making it possible to generate an appropriate digest video.

[0058] [Modification of digest video generating device] As described above, instead of training the digest video generation device 200 using the training data generated by the training data generation device 100, the digest video generation device may be configured using the classification model 31 generated by the video search unit 22 of the training data generation device 100.

[0059] 14 is a block diagram showing a configuration of a digest video generation device 200x according to a modified example. The digest video generation device 200x includes a classification model 31, an importance calculation unit 32, an event scene selection unit 33, and a video combination unit 34. A material video X is input to the classification model 31. The material video X is a video that is a source for creating a digest video. The classification model 31 classifies the material video X into each element included in the digest video using section, and outputs the result to the importance calculation unit 32. The importance calculation unit 32 calculates the importance of each element included in the material video X, and outputs the result to the event scene selection unit 33.

[0060] The event scene selection unit 33 selects event scenes that compose the digest video based on the importance of each element, and outputs the selected event scenes to the video combination unit 34. For example, the event scene selection unit 33 selects, from the material video X, a section that corresponds to an element whose importance is equal to or greater than a predetermined value, as an event scene. The video combination unit 34 combines the scenes selected by the event scene selection unit 33 in chronological order to generate the digest video.

[0061] <Second embodiment> Next, a second embodiment of the present disclosure will be described. Fig. 15 is a block diagram showing a functional configuration of a video processing device according to the second embodiment. As shown in the figure, a video processing device 70 includes an information acquisition unit 71, a video search unit 72, and an importance calculation unit 73.

[0062] 16 is a flowchart of the process performed by the video processing device 70. The information acquisition means 71 acquires the material video and information on the section of the material video used in the digest video (step S71). The video search means 72 searches the material video for elements similar to elements included in the section used in the digest video (step S72). The importance calculation means 73 calculates the importance of each element based on the frequency of appearance of each element in the material video and in the section used in the digest video (step S73).

[0063] According to the video processing device 70 of the second embodiment, it is possible to calculate the importance of elements in the material video that are similar to elements included in the section used in the digest video.

[0064] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.

[0065] (Appendix 1) an information acquisition means for acquiring a raw video and information on a section of the raw video used for a digest video; a video search means for searching the raw video for an element similar to an element included in the section used in the digest video; an importance calculation means for calculating the importance of each element based on the frequency of appearance of each element in the raw video and in a section used in the digest video; A video processing device comprising:

[0066] (Appendix 2) 2. The video processing device according to claim 1, further comprising an output unit for outputting the importance of each element.

[0067] (Appendix 3) 2. The video processing device according to claim 1, further comprising a learning data generating means for labeling a section of each element in the raw video based on the importance of each element, and generating learning data.

[0068] (Appendix 4) 4. The image processing device according to claim 1, wherein the element is one frame image or a plurality of frame images.

[0069] (Appendix 5) 4. The image processing device according to claim 1, wherein the elements are frame images included in each cluster obtained by clustering one or more frame images.

[0070] (Appendix 6) The image processing device according to any one of claims 1 to 3, wherein the element is a central feature in each cluster obtained by clustering one or more frame images.

[0071] (Appendix 7) The video processing device according to any one of claims 1 to 6, wherein the importance calculation means calculates the importance based on a first ratio which is a ratio of the number of elements in the section used for the digest video to the number of all elements in the section used for the digest video, and a second ratio which is a ratio of the number of elements in the section used for the digest video to the number of elements in the raw video.

[0072] (Appendix 8) The video processing device according to any one of claims 1 to 6, wherein the importance calculation means calculates the importance based on a value obtained by subtracting from 1 a first ratio, which is a ratio of the number of one element in a section used in the digest video to the number of all elements in the section used in the digest video, and a third ratio, which is a ratio of the number of the one element in the raw video to the number of all elements in the raw video.

[0073] (Appendix 9) Acquire a raw video and information on a section of the raw video used in the digest video; Searching for an element similar to an element included in the section used in the digest video in the raw video; A video processing method for calculating the importance of each element based on the frequency of appearance of each element in the raw video and in a section used in the digest video.

[0074] (Appendix 10) Acquire a raw video and information on a section of the raw video used in the digest video; Searching for an element similar to an element included in the section used in the digest video in the raw video; A recording medium having recorded thereon a program for causing a computer to execute a process of calculating the importance of each element based on the frequency of appearance of each element in the raw video and in the section used in the digest video.

[0075] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-mentioned embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. [Explanation of symbols]

[0076] 12 processors 21 Video Acquisition Unit 22 Video Search Department 23 Importance calculation part 24 Learning data generation unit 31 Classification Models 32 Importance calculation part 33 Event scene selection section 34 Video Combination Section 41 Learning Department 100 Learning data generation device 200 Digest video generation device

Claims

1. an information acquisition means for acquiring a raw video and information on a section of the raw video used for a digest video; a video search means for searching the raw video for an element similar to an element included in the section used in the digest video; an importance calculation means for calculating the importance of each element based on the frequency of appearance of each element in the raw video and in a section used in the digest video; Equipped with The importance level has a higher value when the frequency of appearance of the element in the section used in the digest video is higher than the frequency of appearance of other elements, and the importance level has a higher value when the frequency of appearance of the element in the raw video is lower.

2. 2. The video processing apparatus according to claim 1, further comprising an output unit that outputs the importance of each element.

3. 2. The video processing device according to claim 1, further comprising: learning data generating means for labeling a section of each element in the raw video based on the importance of each element, and generating learning data.

4. The image processing device according to claim 1 , wherein the element is one frame image or a plurality of frame images.

5. The image processing device according to claim 1 , wherein the elements are frame images included in each cluster obtained by clustering one or more frame images.

6. The image processing device according to claim 1 , wherein the element is a central feature amount in each cluster obtained by clustering one or more frame images.

7. 7. The video processing device according to claim 1, wherein the importance calculation means calculates the importance based on a first ratio which is a ratio of the number of elements in the section used for the digest video to the number of all elements in the section used for the digest video, and a second ratio which is a ratio of the number of elements in the section used for the digest video to the number of elements in the raw video.

8. 7. The video processing device according to claim 1, wherein the importance calculation means calculates the importance based on a value obtained by subtracting from 1 a first ratio, which is a ratio of the number of one element in the section used in the digest video to the number of all elements in the section used in the digest video, and a third ratio, which is a ratio of the number of the one element in the raw video to the number of all elements in the raw video.

9. Acquire a raw video and information on a section of the raw video used in the digest video; Searching for an element similar to an element included in the section used in the digest video in the raw video; Calculating the importance of each element based on the frequency of appearance of each element in the raw video and in the section used in the digest video; a video processing method in which the importance level has a higher value when the frequency of appearance of the element in the section used in the digest video is higher than the frequency of appearance of other elements, and the importance level has a higher value when the frequency of appearance of the element in the raw video is lower.

10. Acquire a raw video and information on a section of the raw video used in the digest video; Searching for an element similar to an element included in the section used in the digest video in the raw video; Calculating the importance of each element based on the frequency of appearance of each element in the raw video and in the section used in the digest video; The program causes a computer to execute a process in which the importance level is increased when the frequency of appearance of the element in the section used in the digest video is higher than the frequency of appearance of other elements, and is increased when the frequency of appearance of the element in the raw video is lower.

Images