Video analysis device, video analysis system, editing device, learning device, video analysis method, and video analysis program

Abstract

To allow an improvement of reliability of an analysis of a scene and an object.SOLUTION: A moving image analyzer divides a moving image in each scene from the moving image and genre information indicating genre of the moving image, uses a first learning model for determining a scene name of a classification of the scene to each scene, divides the moving image into each scene, determines the scene name to each scene, sets a label as information related to an object imaged to a video image frame on the basis of the scene name to each video image frame contained in the scene, and outputs the scene name of each scene containing the video image frame and the label in each video image frame.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a video analysis device and the like.

Background Art

[0002] When creating a new video, a video created in the past may be used. At this time, the video creator searches for the video to be used for creating a new video from the archive in which the videos created in the past are stored.

[0003] Information such as a program name and a program genre may be attached to a video as metadata. In this case, the video creator searches for the video to be used based on this metadata. However, when a large number of videos are stored in the archive, even if the videos are searched based on the metadata, many videos may be extracted. Then, the creator visually searches for the scene to be used from these many videos. In addition, the creator manually cuts out the scene to be used and creates a new video using the cut-out video. Here, a scene refers to a continuous scene in time series.

[0004] As related methods, there are the methods described in Patent Document 1 to Patent Document 2. In these methods, the scenes and subjects captured in the video are analyzed.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the case of videos used by broadcasting stations, for example, a high level of reliability is required for the analysis results of scenes and subjects. If the reliability of the analysis results is low, it will take time for the workers to check and correct them, which will increase the burden on the workers.

[0007] In view of the above-mentioned problems, the object of the present invention is to provide a video analysis device, etc., that can further improve the reliability of scene and subject analysis. [Means for solving the problem]

[0008] In one aspect of the present invention, the video analysis device includes a scene division unit that divides the video into scenes based on genre information indicating the genre of the video, and determines a scene name indicating the classification of each scene for each of the scenes; a label setting unit that sets a label which is information relating to the subject being imaged in the video frame for each of the video frames included in the scene; and an output unit that outputs the scene name of the scene in which the video frame is included and the label for each of the video frames. The scene division unit divides the video into scenes from the video and the genre information of the video, and determines the scene name for each of the scenes using a first learning model.

[0009] Furthermore, in another aspect of the present invention, the video analysis method divides the video into scenes from the video and genre information indicating the genre of the video, and for each of the scenes, it uses a first learning model to determine a scene name indicating the classification of the scene to divide the video into scenes, determines a scene name for each of the scenes, sets a label for each of the video frames included in the scene based on the scene name, and outputs the scene name of the scene in which the video frame is included and the label for each of the video frames.

[0010] In another aspect of the present invention, the video analysis program provides a computer with a scene division function that divides the video into scenes based on genre information indicating the genre of the video, and determines a scene name indicating the classification of each scene; a label setting function that sets a label, which is information about the subject being filmed in the video frame, based on the scene name, for each video frame included in the scene; and an output function that outputs the scene name of the scene in which the video frame is included, and the label, for each video frame. The scene division function divides the video into scenes from the video and the genre information of the video, and determines the scene name using a first learning model that determines the scene name for each scene. [Effects of the Invention]

[0011] According to the present invention, it becomes possible to further improve the reliability of scene and subject analysis. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows an example configuration of a video analysis device according to the first embodiment of the present invention. [Figure 2] This figure shows an example of the operation flow of the video analysis device according to the first embodiment of the present invention. [Figure 3] This figure shows an example configuration of a system including a video analysis device according to a second embodiment of the present invention. [Figure 4] This figure shows an example configuration of a learning device according to a second embodiment of the present invention. [Figure 5] This figure shows an example configuration of a video analysis device according to a second embodiment of the present invention. [Figure 6] This figure shows an example configuration of an editing device according to a second embodiment of the present invention. [Figure 7] This figure shows an example of a scene editing image according to a second embodiment of the present invention. [Figure 8] This figure shows an example of a label editing image according to a second embodiment of the present invention. [Figure 9] It is a diagram showing an example of an editing menu image of the second embodiment of the present invention. [Figure 10] It is a diagram showing an example of an operation flow of a learning device of the second embodiment of the present invention. [Figure 11] It is a diagram showing an example of an operation flow of a video analysis device of the second embodiment of the present invention. [Figure 12] It is a diagram showing an example of an operation flow of an editing device of the second embodiment of the present invention. [Figure 13] It is a diagram showing an example of a hardware configuration of each embodiment of the present invention. [Embodiments for Carrying Out the Invention]

[0013] [First Embodiment] The first embodiment of the present invention will be described. A specific example of the video analysis device 10 in the first embodiment is the video analysis device 20 in the second embodiment described later.

[0014] Fig. 1 shows a configuration example of the video analysis device 10 of this embodiment. The video analysis device 10 of this embodiment includes a scene division unit 11, a label setting unit 12, and an output unit 13.

[0015] The scene division unit 11 divides the video into scenes based on genre information. The genre information indicates the genre of the video. Further, the scene division unit 11 determines a scene name for each of the scenes. The scene name indicates the classification of the scene.

[0016] The scene division unit 11 uses the first learning model to perform the above-described scene division and scene name determination. The first learning model divides the video into scenes based on the video and the genre information of the video, and determines a scene name for each of the scenes.

[0017] The label setting unit 12 sets a label for each of the video frames included in the scene based on the scene name. The label is information regarding the subject imaged in the video frame.

[0018] The output unit 13 outputs, for each video frame, the scene name of the scene in which the video frame is contained, and a label.

[0019] Next, Figure 2 shows an example of the operation flow of the video analysis device 10 of this embodiment.

[0020] The scene division unit 11 divides the video into scenes based on genre information. The scene division unit 11 also determines a scene name for each of the scenes (step S101).

[0021] The label setting unit 12 sets a label for each video frame included in the scene (step S102).

[0022] The output unit 13 outputs the scene name of the scene containing the video frame and a label for each video frame (step S103).

[0023] As described above, in the first embodiment of the present invention, the video analysis device 10 includes a scene division unit 11, a label setting unit 12, and an output unit 13. The scene division unit 11 divides the video into scenes based on genre information. The scene division unit 11 also determines a scene name for each scene. The scene division unit 11 performs scene division and scene name determination using a first learning model. The first learning model divides the video into scenes based on the video and the genre information of the video, and determines a scene name for each scene. The label setting unit 12 sets a label for each video frame included in a scene based on the scene name. The output unit 13 outputs the scene name of the scene in which the video frame is included, and a label for each video frame.

[0024] In this way, the video analysis device 10 divides scenes and determines scene names based on genre information. This improves the likelihood that the video analysis device 10 will analyze a scene as a scene frequently occurring in the genre indicated by the genre information. As a result, the video analysis device 10 can reduce the possibility that a scene with similar video features will be incorrectly analyzed. Therefore, the reliability of scene analysis can be improved. Furthermore, since the video analysis device 10 sets labels based on scene names, the reliability of subject analysis is also improved by the improved reliability of scene analysis. Therefore, it becomes possible to further improve the reliability of scene and subject analysis.

[0025] [Second Embodiment] Next, the video analysis device 20 in the second embodiment of the present invention will be described. The video analysis device 20 in the second embodiment is a specific example of the video analysis device 10 in the first embodiment.

[0026] First, Figure 3 shows an example configuration of a video analysis system 80 including the video analysis device 20 of this embodiment. The video analysis system 80 includes a learning device 60, a video analysis device 20, and an editing device 30. The video analysis device 20 is connected to the video storage device 40, the learning device 60, and the editing device 30. The learning device 60 is also connected to the video storage device 40 and the video analysis device 20. The editing device 30 is also connected to the video storage device 40 and the user terminal 50.

[0027] The video storage device 40 stores video. The video includes image information. The video may also include audio information. The video storage device 40 also stores metadata. The metadata is data related to the video.

[0028] Metadata includes genre information. Genre information indicates the genre of the video. A genre is the type of video. Examples of genres include sports, news / current affairs, and variety.

[0029] Furthermore, metadata can include scene names for each video frame contained within the video. A scene name is a name that indicates the classification of a scene. A scene name is determined for video frames that represent the same scene. For example, if the video is a soccer video, then dribbling, goal, and counter-attack might be scene names.

[0030] Furthermore, metadata may include labels for each video frame contained within the video. Labels provide information about the subject being captured in the video frame. The subject may be a person. One or more labels may be assigned to a single video frame. Additionally, some video frames may not have any labels assigned.

[0031] A label may be, for example, the name of the subject. In this case, the label may be, for example, "soccer player" or "soccer ball." A label may also be a person's name. Alternatively, a label may be information indicating the subject's action. In this case, the label may be, for example, "kicking the ball," "pitching," or "batting." Furthermore, one or more labels may be assigned to a single subject. For example, a video frame labeled "batting" may also be assigned labels that provide more detailed information than "batting," such as "no-step batting" or "one-legged batting."

[0032] Furthermore, the metadata may include region information for each label, corresponding to the subject in the image. Region information indicates which region of the video frame the subject is in.

[0033] The video storage device 40 can store videos that are the target of analysis by the video analysis device 20 and videos used for learning by the learning device 60. The videos to be analyzed may include videos that have not yet been analyzed and videos that have been analyzed. Furthermore, analyzed videos may have edited or unedited analysis results included in their metadata. The analysis results refer to the scene names and labels of each video frame. Learning, analysis, and editing will be described later. The metadata of videos that have not yet been analyzed does not include scene names or labels. The metadata of analyzed videos and videos used for learning includes scene names and labels.

[0034] Furthermore, the video storage device 40 transmits the video and its metadata requested by the video analysis device 20 to the video analysis device 20. The video storage device 40 also transmits the video and its metadata requested by the learning device 60 to the learning device 60. Finally, the video storage device 40 transmits the video and its metadata requested by the editing device 30 to the editing device 30.

[0035] The user terminal 50 is a terminal used by a user who edits metadata. The user terminal 50 is, for example, an information processing device such as a personal computer. The user terminal 50 is equipped with a human-machine interface, such as a keyboard as an input device and a display as an output device. The user terminal 50 gives instructions to the editing device 30 in response to operations entered by the user into the input device. The user terminal 50 also displays images on its display means under control from the editing device 30.

[0036] The learning device 60 generates a learning model for analyzing videos. Details of the learning device 60 will be described later.

[0037] The video analysis device 20 performs video analysis. Details of the video analysis device 20 will be described later.

[0038] The editing device 30 performs metadata editing. Details of the editing device 30 will be described later.

[0039] Next, Figure 4 shows an example of the configuration of the learning device 60 of this embodiment. The learning device 60 includes a learning information input unit 61, a learning information storage unit 62, and a model generation unit 63.

[0040] In this embodiment, the case where the learning device 60 and the video analysis device 20 are different devices is described, but the video analysis device 20 may also have the functions of the learning device 60.

[0041] The learning device 60 of this embodiment generates a first learning model for scene segmentation and a second learning model for label setting. Scene segmentation and label setting will be described later. The process of generating the first learning model and the process of generating the second learning model are different processes. The learning device 60 may be divided into a device for generating the first learning model and a device for generating the second learning model.

[0042] The learning information input unit 61 receives learning information and stores it in the learning information storage unit 62. The learning information storage unit 62 stores the learning information. The model generation unit 63 generates and outputs a learning model using the learning information.

[0043] First, let's explain the case where the learning device 60 generates a first learning model. The first learning model is a learning model for scene segmentation. The input to the first learning model is a video and the genre information of that video. The output of the first learning model is the scene name for each video frame contained in the video. Based on the genre information of the video, the first learning model divides the video into scenes and determines a scene name for each scene.

[0044] In this case, the training information for generating the first learning model includes the scene name, the video frame corresponding to that scene name, and the genre information of the video containing that video frame. The video storage device 40 stores the training video and its metadata. The metadata of the training video includes genre information and the scene name for each video frame. Therefore, the training information input unit 61 can obtain the training information by acquiring the training video and metadata from the video storage device 40.

[0045] Furthermore, among the analyzed videos, those whose metadata has been edited by the editing device 30 may be used as training videos. In this way, by using the editing results from the editing device 30 for training, the accuracy of the training model can be improved.

[0046] Furthermore, genre information is pre-set for the video by another device (not shown), and metadata containing the set genre information is stored in the video storage device 40. If the metadata does not contain genre information, the video analysis device 20 or editing device 30 may set genre information in the metadata in response to operation input from the user terminal 50. Also, as described later, the editing device 30 can change the genre information when editing the metadata.

[0047] The model generation unit 63 generates a first learning model based on the learning information. The model generation unit 63 also transmits the generated first learning model to the scene division unit 21 of the video analysis device 20.

[0048] The learning information input unit 61 can acquire learning videos from the video storage device 40 at predetermined timings. These predetermined timings include, for example, when a new learning video is stored in the video storage device 40 or when metadata is edited by the editing device 30. The model generation unit 63 may also generate a new first learning model at predetermined timings. These predetermined timings include, for example, at regular intervals, when a new learning video is stored in the learning information storage unit 62, or when a learning instruction is input from the user terminal 50.

[0049] Next, we will describe the case in which the learning device 60 generates a second learning model. The second learning model is a learning model for label setting. The second learning model sets a label name for each video frame contained in the same scene frame, based on the same scene frame which is a video frame with the same scene name, the scene name of the same scene frame, and the genre information of the video containing the same scene frame.

[0050] In this case, the training information for generating the second learning model includes a label, the video frame to which the label is set, the region information of the label, the scene name of the video frame to which the label is set, and the genre information of the video containing the video frame to which the label is set. The video storage device 40 stores the training video and its metadata. The metadata of the training video includes genre information, the scene name and label for each video frame, and region information for each label. Therefore, the training information input unit 61 can obtain training information by acquiring the training video and its metadata from the video storage device 40. Note that among the analyzed videos, the video whose metadata has been edited by the editing device 30 may be used as the training video.

[0051] The model generation unit 63 generates a second learning model based on the learning information. The model generation unit 63 also transmits the generated second learning model to the label setting unit 22 of the video analysis device 20.

[0052] The learning information input unit 61 can acquire a learning video from the video storage device 40 at predetermined timings, such as when a new learning video is stored in the video storage device 40 or when metadata is edited by the editing device 30. The model generation unit 63 may also generate a new second learning model at predetermined timings. These predetermined timings may be, for example, every predetermined time, when a new learning video is stored in the learning information storage unit 62, or when a learning instruction is input from the user terminal 50.

[0053] Next, Figure 5 shows an example of the configuration of the video analysis device 20 of this embodiment. The video analysis device 20 includes a scene division unit 21, a label setting unit 22, and an output unit 23. The series of operations performed on the video by the video analysis device 20 is called analysis.

[0054] The scene division unit 21 acquires the video to be analyzed and its metadata from the video storage device 40. The scene division unit 21 also divides the video to be analyzed into scenes based on genre information. The scene division unit 21 also determines a scene name for each scene. The scene division unit 21 performs scene division and scene name determination using a first learning model generated by the learning device 60. The scene division unit 21 inputs the video to be analyzed and its genre information into the first learning model. The first learning model outputs a scene name for each video frame contained in the video.

[0055] The label setting unit 22 sets a label for each video frame included in the video to be analyzed. The label setting unit 22 sets the labels using a second learning model generated by the learning device 60. The label setting unit 22 also uses the second learning model to determine the region information for each label. The label setting unit 22 inputs the same scene frame, the scene name of the same scene frame, and the genre information of the video containing the same scene frame into the second learning model. A same scene frame is a video frame with the same scene name. The second learning model outputs the label for each video frame included in the same scene frame, and the region information for the label.

[0056] The output unit 23 outputs the scene name determined by the scene division unit 21 and the label set by the label setting unit 22. More specifically, the output unit 23 includes the scene name and label for each video frame, as well as region information for each label, in the metadata and stores it in the video storage device 40.

[0057] Next, Figure 6 shows an example of the configuration of the editing device 30 of this embodiment. The editing device 30 of this embodiment includes an editing unit 34 and a storage unit 35. The editing device 30 may be included in the video analysis device 20.

[0058] The editing unit 34 retrieves the video to be edited and its metadata from the video storage device 40 in response to a video acquisition instruction from the user terminal 50. The editing unit 34 also stores the acquired video and metadata in the storage unit 35. A video acquisition instruction is an instruction to acquire the video to be edited. The video acquisition instruction includes identification information of the video to be edited, such as the file name.

[0059] The editing unit 34 edits the metadata of the video to be edited according to scene editing instructions. Scene editing instructions are instructions for editing the scene name for each video frame. The editing unit 34 also edits the metadata of the video to be edited according to label editing instructions. Label editing instructions are instructions for editing the label for each video frame. Scene editing instructions and label editing instructions are input to the editing unit 34 from the user terminal 50. The editing unit 34 also stores the edited metadata in the video storage device 40.

[0060] Furthermore, the editorial department 34 displays the scene editing image 110 on the user terminal 50 in response to the scene editing image display instruction. The scene editing image display instruction is an instruction to display the scene editing image 110. The scene editing image 110 is an image for editing the scene name. The editorial department 34 also generates the scene editing image 110 to be displayed on the user terminal 50 based on the acquired video and metadata.

[0061] Figure 7 shows an example of a scene edited image 110.

[0062] The scene editing image 110 includes the original video display image 111. The original video display image 111 displays the video to be edited. The video displayed on the original video display image 111 can be played and stopped by user input from the user terminal 50. The editing unit 34 displays the video to be edited on the original video display image 111. The editing unit 34 also starts and stops video playback in response to user input from the user terminal 50.

[0063] Furthermore, the scene editing image 110 includes the scene display image 112. The scene display image 112 shows how the video being edited is divided into scenes. In the scene display image 112, the horizontal axis represents the playback time from the start of the video. In the scene display image 112, video frames with the same scene name may be shown in the same color. The metadata includes the scene name for each video frame. Based on the metadata of the video being edited, the editing unit 34 can determine that video frames with the same scene name belong to the same scene. The editing unit 34 displays the boundaries between video frames with different scene names in the scene display image 112.

[0064] Furthermore, the scene editing image 110 includes a cropped image 113. The cropped image 113 is an image (still image) of the video frame indicated by the cursor displayed on the scene display image 112. The cursor displayed on the scene display image 112 moves according to the operation input from the user terminal 50. The editing unit 34 displays the image of the video frame indicated by the cursor displayed on the scene display image 112 as the cropped image 113, according to the operation input from the user terminal 50. The time code displayed above the cropped image 113 represents the start and end positions of the scene containing the video frame indicated by the cursor.

[0065] Furthermore, the scene editing image 110 includes a genre image 114. The genre image 114 contains genre information of the video being edited. The editing unit 34 displays the genre information contained in the metadata of the video being edited in the genre image 114.

[0066] Furthermore, the scene editing image 110 includes the scene name editing image 115. The scene name editing image 115 is an image for editing the scene name. The scene name editing image 115 displays the scene name of the scene indicated by the cursor displayed on the scene display image 112. In addition, when "Edit" is selected in the scene name editing image 115 through operation input from the user terminal 50, the scene name can be edited. When a string is entered into the scene name editing image 115 through operation input from the user terminal 50, a scene editing instruction is input to the editing unit 34. In response to the input scene editing instruction, the editing unit 34 changes the scene name of the video frame included in the scene by editing the metadata stored in the storage unit 35, and stores the edited metadata in the user terminal 50.

[0067] The scene editing image 110 also includes a scene list image 116. The scene list image 116 displays a list of scenes. The scene list image 116 includes the scene names of the scenes included in the video being edited. The scene list image 116 also displays a thumbnail of the corresponding scene for each scene name. The scene list image 116 may also include a timecode for each scene. The timecode indicates the position of each scene in the video based on the playback time from the start of the video.

[0068] The scene list image 116 may include a scene name input area 117 and a scene search image 118. The scene name input area 117 is an area for entering a scene name. The scene search image 118 is an image for searching for scenes with the scene name entered in the scene name input area 117. When the scene search image 118 is selected, the scenes with the scene name entered in the scene name input area 117 are displayed in the scene list image 116.

[0069] Furthermore, the thumbnails displayed in the scene list image 116 can be moved to the display area of ​​a thumbnail with a different scene name by input from the user terminal 50. Moving a thumbnail corresponds to moving a scene. When a scene editing instruction indicating a scene move is input, the editing unit 34 changes the scene name of the video frame corresponding to the moved scene by editing the metadata stored in the storage unit 35. The editing unit 34 then stores the edited metadata in the video storage device 40.

[0070] Furthermore, the scene editing image 110 may also include a time-width editing image 119. The time-width editing image 119 is an image that makes the start and end positions of a scene editable. When the time-width editing image 119 is selected, the start and end positions of the scene in the scene display image 112 become editable. When a scene editing instruction indicating editing of the start or end position of a scene is input to the editing unit 34, the editing unit 34 changes the scene name of the video frame whose scene name has been changed by editing the metadata stored in the storage unit 35. The editing unit 34 then stores the edited metadata in the video storage device 40.

[0071] Furthermore, the editorial department 34 displays the label editing image 120 on the user terminal 50 in response to the label editing image display instruction. The label editing image display instruction is an instruction to display the label editing image 120. The label editing image 120 is an image for editing labels. The editorial department 34 also generates the label editing image 120 to be displayed on the user terminal 50 based on the acquired video and metadata.

[0072] Figure 8 shows an example of label editing image 120.

[0073] Label editing image 120 includes the original video display image 121. Since the original video display image 121 is the same as the original video display image 111, its explanation is omitted.

[0074] Furthermore, label editing image 120 includes scene display image 122. Since scene display image 122 is the same as scene display image 112, its explanation is omitted.

[0075] Furthermore, label editing image 120 includes cropped image 123. Since cropped image 123 is the same as cropped image 113, its explanation is omitted.

[0076] Furthermore, the label editing image 120 includes a genre image 124. The genre image 124 displays the genre information of the video being edited. The editing unit 34 displays the genre information contained in the metadata of the video being edited on the genre image 124.

[0077] Furthermore, in genre image 124, the displayed genre information changes in response to user input from the user terminal 50. When the user selects "Register" for genre image 124, the editorial department 34 edits the metadata stored in the memory unit 35 to change the genre information to the genre information displayed in genre image 124.

[0078] The first and second learning models are learning models based on genre information. Therefore, if the genre information is changed, the scene names and labels of the video frames may change when reanalysis is performed. For this reason, the editing device 30 may instruct the video analysis device 20 to reanalyze the video to be edited when the genre information is changed. The editing unit 34 may then obtain the metadata after reanalysis from the video storage device 40 and display the scene edited image 110 and label edited image 120 based on the new metadata on the user terminal 50.

[0079] Furthermore, the label editing image 120 includes the label display image 125. The label display image 125 is an image for editing labels. The label display image 125 displays the label of the video frame selected by the cursor displayed in the scene display image 122. The editing unit 34 refers to the metadata of the video to be edited and displays the label set for that video frame in the label display image 125. In addition, the labels displayed in the label display image 125 can be deleted by operation input from the user terminal 50.

[0080] Additionally, label display image 125 includes similar labels. These similar labels are candidates for additional labels.

[0081] Similar labels may be, for example, words similar to the label that is set. The editorial unit 34, for example, refers to a similar word dictionary and displays words similar to the label set in the video frame as similar labels in the label display image 125. In this case, the similar word dictionary is stored in the storage unit 35.

[0082] Furthermore, similar labels may be, for example, labels set for similar videos. The editorial department 34 searches for videos with the same genre information as the video to be edited from among the videos stored in the video storage device 40, and obtains metadata for the searched videos. The editorial department 34 then displays the labels included in the metadata as similar labels on the label display image 125. Alternatively, the editorial department 34 may search for videos with similar characteristics from among the videos stored in the video storage device 40. In this case, the output unit 23 of the video analysis device 20 calculates the feature quantities of the video during analysis and stores them in the video storage device 40. The editorial department 34 then searches the video storage device 40 for videos with feature quantities similar to the feature quantities of the video to be edited.

[0083] When a label editing instruction is entered by the editorial department 34 to add similar labels to the video frame by editing the metadata, the editorial department 34 adds the selected similar labels as labels to the video frame. More specifically, when the operation input to select "Add" on the label addition image 127 is entered by the editorial department 34, the similar labels for which the checkbox next to the similar label is checked are added as labels set for the video frame.

[0084] Furthermore, the label editing image 120 includes the unlabeled scene image 126. The unlabeled scene image 126 displays thumbnails of scenes for which no labels have been set. Scenes for which no labels have been set are scenes for which there are no video frames with labels. Based on the metadata of the video to be edited, the editorial department 34 displays thumbnails of scenes for which no labels have been set in the unlabeled scene image 126. Also, when the operation input to select "Add" on the label addition image 127 is made and a label is set for a scene that was displayed as an unlabeled scene, the editorial department 34 deletes that scene from the unlabeled scene image 126. The editorial department 34 may also delete scenes with labels set from the unlabeled scene image 126 when the operation input to select the save image 129 or the registration image 130 is made.

[0085] Furthermore, the label editing image 120 includes the label addition image 127. The label addition image 127 is an image for adding labels. The editing unit 34 displays the label addition image 127 when the user selects the "+" displayed on the label display image 125. The editing unit 34 also displays the label addition image 127 when the user selects a thumbnail displayed on the unlabeled scene image 126. In this case, the editing unit 34 moves the cursor displayed on the scene display image 122 to the right or left to the position of the scene corresponding to the selected thumbnail.

[0086] When the editor 34 receives an input and enters text into the label addition image 127, and selects "Add," the editor displays the entered text as the label set for the video frame in the label display image 125. At this time, if there are similar labels with a checkbox checked, those similar labels are also added to the label display image 125 as labels set for the video frame.

[0087] Furthermore, the label editing image 120 includes a label box image 128. The label box image 128 displays boxes indicating region information for the label. The boxes indicate region information, that is, in which region of the video frame the subject corresponding to the label is captured. The label box image 128 displays the boxes of the labels set for the video frame indicated by the cursor displayed in the scene display image 122. At this time, the label box image 128 displays the boxes superimposed on the image of the video frame indicated by the cursor displayed in the scene display image 122. The label box image 128 also displays the labels corresponding to the boxes. The editing unit 34 displays the boxes for each label in the label box image 128 based on the region information contained in the metadata. The boxes can also be moved and transformed by operation input from the user terminal 50.

[0088] Furthermore, if a label set on a video frame is added to the label display image 125, the editorial unit 34 displays a box for the added label in the label box image 128. The initial position and size of the box displayed at this time are arbitrary. Also, if a label set on a video frame is removed from the label display image 125, the editorial unit 34 removes the box for the removed label from the label box image 128.

[0089] Furthermore, the label editing image 120 includes a save image 129. The save image 129 is an image for temporarily saving labels. When an operation input is made to select the save image 129, the editing unit 34 edits the metadata stored in the storage unit 35 to change the label of the video frame indicated by the cursor displayed in the scene display image 122 to the label displayed in the label display image 125. At this time, the editing unit 34 also edits the metadata stored in the storage unit 35 to change the label area information of the video frame indicated by the cursor displayed in the scene display image 122 to the label box area information displayed in the label box image 128. The editing unit 34 may also perform the same label change on video frames that have the same scene name as the video frame in question and whose labels were the same before the change.

[0090] Furthermore, the label editing image 120 includes the registration image 130. The registration image 130 is an image used for registering metadata. When the editing unit 34 receives an input to select the registration image 130, it first edits the metadata stored in the storage unit 35, similar to when the operation input to select the save image 129 is received. Then, the editing unit 34 stores the metadata stored in the storage unit 35 in the video storage device 40.

[0091] The scene editing image 110 and the label editing image 120 are displayed on the user terminal 50, for example, by inputting an operation on the editing menu image 140. Figure 9 shows an example of the editing menu image 140.

[0092] The editing menu image 140 includes a video information display image 141. The video information display image 141 displays identification information of the video being edited, such as the file name.

[0093] Furthermore, the editing menu image 140 includes the video selection image 142, which is used to select the video to be edited. When the editing unit 34 receives input to select the video selection image 142, it displays a list of videos, etc. When the editing unit 34 selects the video to be edited, it displays the identification information of the selected video on the video information display image 141. The editing unit 34 also acquires the selected video and its metadata, and stores the acquired video and metadata in the storage unit 35.

[0094] Furthermore, the editing menu image 140 includes the scene editing image display image 143. The scene editing image display image 143 is an image for displaying the scene editing image 110. When the scene editing image display image 143 is selected, a scene editing image display instruction is input to the editing unit 34. When the editing unit 34 receives the scene editing image display instruction, it displays the scene editing image 110 on the user terminal 50.

[0095] Furthermore, the editing menu image 140 includes an image 144 for displaying the label editing image. The image 144 for displaying the label editing image 120 is an image for displaying the label editing image 120. When the image 144 for displaying the label editing image is selected, a label editing image display instruction is input to the editing unit 34. When the editing unit 34 receives the label editing image display instruction, it displays the label editing image 120 on the user terminal 50.

[0096] Thus, the editing device 30 of this embodiment can edit metadata to change scene names and labels. Furthermore, the learning device 60 can perform retraining using the edited metadata. In this way, by performing retraining using learning information based on the edited metadata, it becomes possible to achieve scene segmentation with the accuracy expected by the user and labeling with the granularity expected by the user.

[0097] For example, suppose the label displayed in the label display image 125 for a video frame is "Batting". In this case, the user can check the cropped image 123, etc., and add more detailed labels (for example, "No-Step Batting" or "One-Legged Batting"). The learning device 60 can also perform retraining using the learning information that includes the added labels. When retraining is performed using the learning information that includes the added labels and a new second learning model is generated, the added labels will be automatically set by the second learning model. In this way, by performing retraining using the learning information that includes the edited labels, it becomes possible to achieve labeling with the granularity expected by the user.

[0098] Next, Figure 10 shows an example of the operation flow of the learning device 60 in this embodiment. The learning device 60 performs the operations shown in Figure 10 for each of the generation processes: the generation of the first learning model and the generation of the second learning model. The learning device 60 also performs the operations shown in Figure 10 at predetermined intervals or when learning is instructed from the user terminal 50.

[0099] The learning information input unit 61 receives the learning information and stores it in the learning information storage unit 62 (step S201). The model generation unit 63 generates and outputs a learning model using the learning information (step S202).

[0100] Next, Figure 11 shows an example of the operation flow of the video analysis device 20 of this embodiment. The video analysis device 20 performs the operations shown in Figure 11 at predetermined intervals or when analysis is instructed from the user terminal 50.

[0101] Furthermore, the video analysis device 20 may perform the operation shown in Figure 11 when a new video to be analyzed is added to the video storage device 40. For example, the video analysis device 20 can detect that a new video has been added to the video storage device 40 by obtaining a list of videos from the video storage device 40 at predetermined intervals and comparing the old and new video lists. Alternatively, the device that added the new video to the video storage device 40 may send a notification to the video analysis device 20 indicating that a video has been added.

[0102] The scene division unit 21 acquires the video to be analyzed and its metadata from the video storage device 40 (step S301). The scene division unit 21 also divides the video to be analyzed into scenes based on genre information. The scene division unit 21 also determines a scene name for each video frame included in the video (step S302). The scene division unit 21 performs scene division and scene name determination using a first learning model generated by the learning device 60. The scene division unit 21 inputs the video to be analyzed and its genre information into the first learning model. The first learning model outputs a scene name for each video frame included in the video.

[0103] The label setting unit 22 sets a label for each video frame included in the video to be analyzed (step S303). The label setting unit 22 sets the labels using a second learning model generated by the learning device 60. The label setting unit 22 also uses the second learning model to determine region information for each label. The label setting unit 22 inputs the same scene frame, the scene name of the same scene frame, and the genre information of the video containing the same scene frame into the second learning model. The second learning model outputs the label for each video frame included in the same scene frame, and the region information for the label.

[0104] The output unit 23 outputs the scene name determined by the scene division unit 21 and the label set by the label setting unit 22. More specifically, the output unit 23 includes the scene name and label for each video frame, as well as the region information for each label, in the metadata and stores it in the video storage device 40 (step S304).

[0105] Next, Figure 12 shows an example of the operation flow of the editing device 30 in this embodiment.

[0106] The editing unit 34, in response to a video acquisition instruction from the user terminal 50, acquires the video to be edited and its metadata from the video storage device 40 (step S401). The editing unit 34 also stores the acquired video and metadata in the storage unit 35.

[0107] The editorial unit 34 displays the scene editing image 110 on the user terminal 50 in response to a scene editing image display instruction. The editorial unit 34 also displays the label editing image 120 on the user terminal 50 in response to a label editing image display instruction (step S402).

[0108] Then, the editing unit 34 updates the scene editing image 110 in accordance with the scene editing instructions. The editing unit 34 also edits the metadata stored in the storage unit 35 in accordance with the scene editing instructions. The editing unit 34 also updates the label editing image 120 in accordance with the label editing instructions. The editing unit 34 also edits the metadata stored in the storage unit 35 in accordance with the label editing instructions. The editing unit 34 also stores the metadata stored in the storage unit 35 in the video storage device 40 in accordance with the instructions (step S403).

[0109] As described above, in the second embodiment of the present invention, the video analysis device 20 includes a scene division unit 21, a label setting unit 22, and an output unit 23. The scene division unit 21 divides the video into scenes based on genre information. The scene division unit 21 also determines a scene name for each scene. The scene division unit 21 performs scene division and scene name determination using a first learning model. The first learning model divides the video into scenes based on the video and the genre information of the video, and determines a scene name for each scene. The label setting unit 22 sets a label for each video frame included in a scene based on the scene name. The output unit 23 outputs the scene name of the scene in which the video frame is included, and a label for each video frame.

[0110] In this way, the video analysis device 20 divides scenes and determines scene names based on genre information. This improves the likelihood that the video analysis device 20 will analyze a scene as a scene frequently occurring in the genre indicated by the genre information. As a result, the video analysis device 20 can reduce the possibility that a scene with similar video features will be incorrectly analyzed. Therefore, the reliability of scene analysis can be improved. Furthermore, since the video analysis device 20 sets labels based on scene names, the reliability of subject analysis is also improved by the improved reliability of scene analysis. Therefore, it becomes possible to further improve the reliability of scene and subject analysis.

[0111] The label setting unit 22 sets labels using a second learning model. The second learning model sets labels for each video frame contained within a given scene frame, based on the given scene frame, the scene name of that scene frame, and the genre information of the video containing that scene frame. A given scene frame is a video frame with the same scene name. As a result, genre information is also used for label setting, which improves the reliability of subject analysis. Furthermore, by using a second learning model for label setting, the variability of the labels set can be reduced compared to when labels are set manually.

[0112] Furthermore, the second learning model outputs region information for each label. Region information indicates which region of the video frame contains the subject corresponding to the label. The label setting unit 22 uses the second learning model to determine the region information for each label. The output unit 23 further outputs the region information. This makes it possible to visualize the region information, improving convenience for the user.

[0113] Furthermore, the output unit 23 stores the scene name in the video storage device, including it in the metadata. The metadata is information about the video. The video storage device stores both the video and the metadata. This allows the user to access the metadata at the time they need it.

[0114] The editing device 30 also includes an editing unit 34. The editing unit 34 acquires the video to be edited and metadata related to that video from the video storage device 40. The editing unit 34 also displays a scene edited image on the user terminal 50 based on the acquired video and metadata, in response to a scene edited image display instruction. The scene edited image is an instruction to display the scene edited image. The scene edited image is an image for editing the scene name. The editing unit 34 also edits the metadata in response to the scene edit instruction. The scene edit instruction is an instruction related to editing the scene name. The editing unit 34 also edits the scene name included in the metadata in response to the scene edit instruction and stores the edited metadata in the video storage device 40. This makes it possible for the user to edit the scene name.

[0115] Furthermore, the scene editing image includes the scene name and thumbnail for each scene in the video being edited. This makes it easy for users to see the results of the scene division.

[0116] Furthermore, when the thumbnail is moved to a display area for thumbnails with different scene names, and a scene editing instruction indicating a scene change is entered, the editorial department 34 changes the scene name of the video frame corresponding to the moved scene by editing the metadata. This makes it possible to easily change scene names.

[0117] Furthermore, the editing unit 34 obtains the video to be edited and metadata related to that video from the video storage device 40. In response to a label editing image display instruction, the editing unit 34 displays a label editing image on the user terminal 50 based on the obtained video and metadata. The label editing image display instruction is an instruction to display the label editing image. The label editing image is an image for editing labels. Furthermore, the editing unit 34 edits the metadata in response to the label editing instruction. The label editing instruction is an instruction related to editing labels. Furthermore, the editing unit 34 edits the labels included in the metadata in response to the label editing instruction and stores the edited metadata in the video storage device 40. This makes it possible for users to edit labels.

[0118] Furthermore, the label editing image includes similar labels that are candidates for the label to be added. When a label editing instruction is entered that instructs the editor 34 to add a similar label, it edits the metadata to add the selected similar label as a label to the video frame. This makes it easy to add labels.

[0119] Furthermore, similar labels are words similar to the labels set for the video frames. This makes it easy to add similar words to the labels.

[0120] Furthermore, similar labels are labels set for similar videos. Similar videos are videos with the same genre information as the video being edited, or videos with similar feature sets. The editing unit 34 searches for similar videos among the videos stored in the video storage device 40, and sets the labels included in the metadata of the found similar videos as similar labels. This makes it easy to add labels set for similar videos.

[0121] Additionally, the label editing images include thumbnails of scenes that do not have labels assigned. This makes it easy for users to see the existence of scenes that do not have labels.

[0122] Furthermore, the label setting unit 22 determines region information for each label. Region information indicates which region of the video frame contains the subject corresponding to the label. The output unit 23 further includes the region information as metadata and stores it in the video storage device 40. The label editing image includes a box that shows the region information for the label. This makes it easy for the user to understand the region corresponding to the label.

[0123] Furthermore, the learning device 60 uses the metadata edited by the editing device 30 as learning information to generate a first learning model. This further improves the reliability of the first learning model.

[0124] The label setting unit 22 sets labels using a second learning model. The second learning model sets labels for each video frame contained in a given scene frame, based on the given scene frame, the scene name of that given scene frame, and the genre information of the video containing that given scene frame. A given scene frame is a video frame with the same scene name. The learning device 60 generates a second learning model using the metadata edited by the editing device 30 as learning information. As a result, the results of editing by the user are used for learning, making it possible to bring the automatically assigned labels closer to the granularity desired by the user.

[0125] [Example Hardware Configuration] This section describes an example of hardware resource configurations for realizing the video analysis device (10, 20), editing device 30, or learning device 60 (hereinafter referred to as "video analysis device, etc.") in each embodiment of the present invention described above, using a single information processing device (computer). Note that the video analysis device, etc. may be realized using at least two information processing devices, either physically or functionally. Furthermore, the video analysis device, etc. may be realized as a dedicated device. Also, only some functions of the video analysis device, etc. may be realized using an information processing device.

[0126] Figure 13 is a schematic diagram showing an example of the hardware configuration of an information processing device capable of realizing the video analysis device and the like according to each embodiment of the present invention. The information processing device 90 includes a communication interface 91, an input / output interface 92, an arithmetic unit 93, a storage device 94, a non-volatile storage device 95, and a drive device 96.

[0127] For example, the scene division unit 11 and label setting unit 12 in Figure 1 can be implemented by the arithmetic unit 93. Furthermore, the output unit 13 can be implemented by the communication interface 91 and the arithmetic unit 93.

[0128] The communication interface 91 is a communication means for the video analysis device, etc., of each embodiment to communicate with an external device by wired and / or wireless means. If the video analysis device, etc., is implemented using at least two information processing devices, these devices may be connected via the communication interface 91 to enable mutual communication.

[0129] The input / output interface 92 is a human-machine interface, such as a keyboard as an example of an input device, or a display as an output device.

[0130] The arithmetic unit 93 is implemented by a general-purpose CPU (Central Processing Unit) or microprocessor, as well as multiple electrical circuits. The arithmetic unit 93 can, for example, read various programs stored in the non-volatile memory device 95 into the memory device 94 and execute processing according to the read programs.

[0131] The storage device 94 is a memory device such as RAM (Random Access Memory) that can be accessed by the arithmetic unit 93, and stores programs and various data. The storage device 94 may also be a volatile memory device.

[0132] The non-volatile storage device 95 is a non-volatile storage device such as ROM (Read Only Memory) or flash memory, and is capable of storing various programs and data.

[0133] The drive device 96 is, for example, a device that processes data reading and writing to the recording medium 97, which will be described later.

[0134] The recording medium 97 is any recording medium capable of recording data, such as an optical disc, magneto-optical disc, or semiconductor flash memory.

[0135] Each embodiment of the present invention may be implemented, for example, by configuring a video analysis device, etc., with the information processing device 90 illustrated in Figure 13, and supplying this video analysis device, etc., with a program capable of realizing the functions described in each embodiment above.

[0136] In this case, the embodiment can be realized by having the computing device 93 execute the program supplied to the video analysis device, etc. Furthermore, it is also possible to configure only some, rather than all, of the functions of the video analysis device, etc., in the information processing device 90.

[0137] Furthermore, the above program can also be recorded on the recording medium 97. The program may be configured to be stored in the non-volatile storage device 95 as appropriate during the shipping or operation phase of the video analysis device, etc. In this case, the method of supplying the program may be to install it into the video analysis device, etc. using an appropriate jig during the manufacturing phase before shipping or during the operation phase, etc. Alternatively, the method of supplying the program may be to use a general procedure such as downloading it from an external source via a communication line such as the Internet.

[0138] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0139] (Note 1) A scene division unit divides the video into scenes based on genre information indicating the genre of the video, and determines a scene name for each of the scenes that indicates the classification of the scene, A label setting unit sets a label for each video frame included in the scene, based on the scene name, which is information about the subject being filmed in the video frame. For each of the aforementioned video frames, an output unit outputs the scene name of the scene containing the video frame and the label. Equipped with, The scene division unit performs scene division and scene naming using a first learning model that classifies the video into scenes based on the video and the genre information of the video, and determines the scene name for each of the scenes. Video analysis device.

[0140] (Note 2) The label setting unit sets the labels using a second learning model that sets the labels for each of the video frames included in the same scene frame, based on the same scene frame which is the same video frame having the same scene name, the scene name of the same scene frame, and the genre information of the video which includes the same scene frame. The video analysis device described in Appendix 1.

[0141] (Note 3) The second learning model further outputs region information for each of the labels, which is information indicating which region of the video frame contains the subject corresponding to the label. The label setting unit uses the second learning model to determine the region information for each of the labels, The output unit further outputs the region information. The video analysis device described in Appendix 2.

[0142] (Note 4) The output unit includes the scene name in the metadata, which is information about the video, and stores the video and the metadata in a video storage device. The video analysis device described in Appendix 1.

[0143] (Note 5) The video analysis device described in Appendix 1, A learning device that generates the first learning model and A video analysis system equipped with the following features.

[0144] (Note 6) The video analysis device described in Appendix 2 or Appendix 3, A learning device that generates the first learning model and the second learning model. A video analysis system equipped with the following features.

[0145] (Note 7) The system comprises a video analysis device and an editing device as described in Appendix 4. The editing device is The editorial department edits the metadata in accordance with the scene editing instructions, which are instructions regarding the editing of the aforementioned scene name. Equipped with, The aforementioned editorial department, From the aforementioned video storage device, obtain the video to be edited and the metadata related to that video. In response to a scene editing image display instruction that instructs the display of a scene editing image, which is an image for editing the aforementioned scene name, the scene editing image is displayed on the user terminal based on the acquired video and the metadata. In accordance with the scene editing instruction, the scene name included in the metadata is edited, and the edited metadata is stored in the video storage device. Video analysis system.

[0146] (Note 8) The scene editing image includes the scene name and thumbnail for each of the scenes included in the video to be edited. The video analysis system described in Appendix 7.

[0147] (Note 9) When the editorial department receives a scene editing instruction indicating a scene change by moving the thumbnail to a display area for the thumbnail with a different scene name, it edits the metadata to change the scene name of the video frame corresponding to the moved scene. The video analysis system described in Appendix 8.

[0148] (Note 10) The system comprises a video analysis device and an editing device as described in Appendix 4. The editing device is The video analysis device edits the metadata in accordance with a label editing instruction, which is an instruction regarding the editing of the label. Equipped with, The aforementioned editorial department, From the aforementioned video storage device, the video to be edited and the metadata relating to that video are obtained. In response to a label editing image display instruction, which instructs the display of a label editing image that is an image for editing the label, the label editing image is displayed on the user terminal based on the acquired video and metadata. In response to the label editing instruction, the label included in the metadata is edited, and the edited metadata is stored in the video storage device. Video analysis system.

[0149] (Note 11) The aforementioned label editing image includes similar labels that are candidates for the label to be added. When the editorial department receives the label editing instruction that instructs the addition of the similar label, it edits the metadata to add the selected similar label as the label for the video frame. The video analysis system described in Appendix 10.

[0150] (Note 12) The aforementioned similar label is a word similar to the label set in the video frame. The video analysis system described in Appendix 11.

[0151] (Note 13) The aforementioned similar label is the label set for the similar video, The aforementioned similar videos are videos that have the same genre information as the video to be edited, or videos with similar feature quantities. The editorial department searches for similar videos among the videos stored in the video storage device, and uses the labels included in the metadata of the found similar videos as the similar labels. The video analysis system described in Appendix 11.

[0152] (Note 14) The label editing image includes a thumbnail of the scene for which no label has been set. The video analysis system described in Appendix 10.

[0153] (Note 15) The label setting unit determines, for each of the labels, region information which indicates which region of the video frame the subject corresponding to the label is imaged in, The output unit further includes the region information in the metadata and stores it in the video storage device. The label editing image includes a box that shows the area information for the label, The video analysis system described in Appendix 10.

[0154] (Note 16) Furthermore, the system includes a learning device that generates the first learning model, The learning device uses the metadata edited by the editing device as learning information to generate the first learning model. A video analysis system as described in any of the appendices 10 to 15.

[0155] (Note 17) The label setting unit sets the labels using a second learning model that sets the labels for each of the video frames included in the same scene frame, based on the same scene frame which is the same video frame having the same scene name, the scene name of the same scene frame, and the genre information of the video which includes the same scene frame. Furthermore, the system includes a learning device that generates the second learning model using the metadata edited by the editing device as learning information. A video analysis system as described in any of the appendices 10 to 15.

[0156] (Note 18) An editing device in a video analysis system as described in any of the appendices 7 to 15.

[0157] (Note 19) A learning device in the video analysis system described in Appendix 5.

[0158] (Note 20) A learning device in the video analysis system described in Appendix 6.

[0159] (Note 21) A learning device in the video analysis system described in Appendix 16.

[0160] (Note 22) A learning device in the video analysis system described in Appendix 17.

[0161] (Note 23) Using a first learning model that divides a video into scenes from a video and genre information indicating the genre of the video, and determines a scene name indicating the classification of each scene, the video is divided into scenes, and a scene name is determined for each scene. For each video frame included in the aforementioned scene, a label is set based on the scene name, which is information about the subject being captured in the video frame. For each of the aforementioned video frames, the scene name of the scene containing the video frame and the label are output. Video analysis methods.

[0162] (Note 24) On the computer, A scene division function that divides the video into scenes based on genre information indicating the genre of the video, and determines a scene name for each of the scenes that indicates the classification of the scene, A label setting function that sets a label for each video frame included in the aforementioned scene, based on the scene name, which is information about the subject being captured in the video frame. For each of the aforementioned video frames, an output function outputs the scene name of the scene containing the video frame and the label. To make it happen, The scene division function divides the video into scenes based on the video and the genre information of the video, and performs the scene division and scene naming using a first learning model that determines the scene name for each of the scenes. Video analysis program.

[0163] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made, which can be understood by those skilled in the art within the scope of the present invention. [Explanation of symbols]

[0164] 10, 20 Video analysis device 11, 21 Scene division section 12, 22 Label setting section 13, 23 Output section 30 Editing device 34 Editorial Department 35 Storage section 40 Video storage devices 50 User terminals 60 Learning device 61 Learning Information Input Section 62 Learning Information Storage Unit 63 Model Generation Unit 80 Video Analysis System 90 Information Processing Equipment 91 Communication Interface 92 Input / Output Interfaces 93 Arithmetic unit 94 Storage device 95 Non-volatile memory devices 96 Drive unit 97 Recording media

Claims

1. A scene division unit divides the video into scenes based on genre information indicating the genre of the video, and determines a scene name for each of the scenes that indicates the classification of the scene, A label setting unit sets a label for each video frame included in the scene, based on the scene name, which is information about the subject being filmed in the video frame. For each of the aforementioned video frames, an output unit outputs the scene name of the scene containing the video frame and the label. Equipped with, The scene division unit divides the video into scenes based on the video and the genre information of the video, and performs the scene division and scene naming using a first learning model that determines the scene name for each of the scenes. Video analysis device.

2. The label setting unit sets the labels using a second learning model that sets the labels for each of the video frames included in the same scene frame, based on the same scene frame which is the same video frame having the same scene name, the scene name of the same scene frame, and the genre information of the video which includes the same scene frame. The video analysis device according to claim 1.

3. The output unit includes the scene name in the metadata, which is information about the video, and stores the video and the metadata in a video storage device. The video analysis device according to claim 1.

4. The video analysis device according to claim 1, A learning device that generates the first learning model and A video analysis system equipped with the following features.

5. The video analysis device according to claim 2, A learning device that generates the first learning model and the second learning model. A video analysis system equipped with the following features.

6. The device comprises a video analysis device and an editing device as described in claim 3, The editing device is The editorial department edits the metadata in accordance with the scene editing instructions, which are instructions regarding the editing of the aforementioned scene name. Equipped with, The aforementioned editorial department, From the aforementioned video storage device, obtain the video to be edited and the metadata related to that video. In response to a scene editing image display instruction that instructs the display of a scene editing image, which is an image for editing the aforementioned scene name, the scene editing image is displayed on the user terminal based on the acquired video and the metadata. In accordance with the scene editing instruction, the scene name included in the metadata is edited, and the edited metadata is stored in the video storage device. Video analysis system.

7. Editing device in the video analysis system according to claim 6.

8. A learning device in the video analysis system according to claim 4.

9. The video analysis device is Using a first learning model that divides a video into scenes from a video and genre information indicating the genre of the video, and determines a scene name indicating the classification of each scene, the video is divided into scenes, and a scene name is determined for each scene. For each video frame included in the aforementioned scene, a label is set based on the scene name, which is information about the subject being captured in the video frame. For each of the aforementioned video frames, the scene name of the scene containing the video frame and the label are output. Video analysis methods.

10. On the computer, A scene division function that divides the video into scenes based on genre information indicating the genre of the video, and determines a scene name for each of the scenes that indicates the classification of the scene, A label setting function that sets a label for each video frame included in the aforementioned scene, based on the scene name, which is information about the subject being captured in the video frame. For each of the aforementioned video frames, an output function outputs the scene name of the scene containing the video frame and the label. To make it happen, The scene division function divides the video into scenes based on the video and the genre information of the video, and performs the scene division and scene naming using a first learning model that determines the scene name for each of the scenes. Video analysis program.

Images