Information processing device, information processing method, and program

Abstract

This information processing device is provided with a control unit for performing: a first control process for determining, from a plurality of analysis engines on the basis of scene detection information for scene detection with respect to an input video, an analysis engine for scene detection; and a second control process for determining, from the plurality of analysis engines on the basis of scene-related information pertaining to a scene obtained as first result information by the analysis engine determined in the first control process, an analysis engine for obtaining second result information related to the scene.

Description

[Technical field]

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to a technology for analyzing moving image data using an analysis engine. [Background technology]

[0002] Techniques have been proposed for analyzing video data using an analysis engine. For example, Patent Document 1 below discloses a technique for selecting an analysis engine from a plurality of analysis engines according to the content of an analysis processing request and using the selected analysis engine for analysis processing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO2012 / 108125 publication Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, for example, in creating a highlight video that collects important scenes, it is not enough to simply identify the important scenes; it is also important to appropriately set the cut-out range before and after the identified scenes.

[0005] The present technology has been developed in consideration of these circumstances, and aims to appropriately identify scenes and extraction ranges using an analysis engine. [Means for solving the problem]

[0006] The information processing device of the present technology includes a control unit that performs a first control process of determining an analysis engine for scene detection from a plurality of analysis engines based on scene detection information for scene detection in an input video, and a second control process of determining an analysis engine for obtaining second result information related to a scene from the plurality of analysis engines based on scene-related information about the scene obtained as first result information by the analysis engine determined in the first control process. The input video is assumed to be, for example, content produced based on images captured by one or more cameras for a certain sport. Also assumed is a system having multiple analysis engines for analyzing such videos. In such a system, an information processing device having a function as a control unit determines, in a first control process, one or more analysis engines for detecting a specific scene in the video as first result information, and, in a second control process, determines one or more analysis engines for identifying second result information related to the scene.

[0007] In the above information processing device, the second result information may include time information related to the scene obtained as the first result information. That is, in the second control process, an analysis engine for obtaining second result information including at least time information related to a scene is determined.

[0008] In the above-mentioned information processing device, the second result information includes scene start information regarding the start of the scene obtained as the first result information and scene end information regarding the end of the scene obtained as the first result information, and the second control process may determine an analysis engine for identifying the scene start information and an analysis engine for identifying the scene end information. That is, in the second control process, an analysis engine for obtaining second result information including scene start information and scene end information as the second result information is determined, and analysis engines corresponding to the scene start information and scene end information are determined, respectively.

[0009] In the above information processing device, the scene-related information may include scene type information, and the second control process may determine an analysis engine for obtaining second result information based on the scene type information. That is, in the second control process, an analysis engine for obtaining the second result information is determined, and a corresponding analysis engine is determined depending on the type of the target scene.

[0010] In the above-mentioned information processing device, the second result information includes scene start information regarding the start of the scene obtained as the first result information and scene end information regarding the end of the scene obtained as the first result information, and the second control process may determine an analysis engine for identifying the scene start information and an analysis engine for identifying the scene end information depending on the type of scene obtained as the first result information. That is, in the second control process, an analysis engine is determined for obtaining second result information including scene start information and scene end information as the second result information, and an analysis engine for obtaining scene start information and an analysis engine for obtaining scene end information are determined depending on the type of scene.

[0011] In the above-mentioned information processing device, the second result information may include time information related to a scene obtained as the first result information, and the control unit may perform a third control process to determine an analysis engine from a plurality of analysis engines to obtain third result information analyzing a section in the video identified by the time information. When a section within a video relating to a certain scene is identified as the second result information, an analysis engine that performs a detailed analysis of that section is determined.

[0012] In the above-described information processing device, the third control process may determine an analysis engine for obtaining third result information based on scene type information. That is, in the third control process, an analysis engine for obtaining the third result information is determined, and a corresponding analysis engine is determined depending on the type of the target scene.

[0013] In the above-described information processing device, the scene-related information may be managed in association with the scene detection information, and the scene-related information may be set in response to a setting of the scene detection information. Management is performed so that scene-related information is identified in response to scene detection information for scene detection being identified.

[0014] In the above information processing device, the control unit may set the scene detection information in response to an input of a scene type. For example, scene detection information is set according to a scene type inputted according to a user operation or automatic determination.

[0015] In the above-mentioned information processing device, the control unit may set the scene detection information in response to an input of a sport type. For example, the scene detection information is set according to the type of sport (for example, the type of sport such as a sport) that is input according to a user operation or automatic determination.

[0016] In the above information processing device, the control unit may perform a process of generating metadata based on the first result information and the second result information, and associating the generated metadata with the input moving image. For example, information on the detected scene and its time information are generated as metadata, and are treated as information related to the input video.

[0017] In the above-mentioned information processing device, the control unit may generate metadata based on the first result information, the second result information, and the third result information, and perform a process of associating the generated metadata with the input video. For example, information about the detected scene, its time information, and further detailed information are generated as metadata, and are treated as information related to the input video.

[0018] In the above-mentioned information processing device, the control unit may compare time information obtained as the first result information with time information provided as external data for a scene that is to be detected in the input video, and if they differ, overwrite the time information provided as the external data with the time information obtained as the first result information. For example, external data such as STATS data regarding scene time codes is rewritten according to the analysis results.

[0019] In the above-mentioned information processing device, the control unit may compare, for a scene that is to be detected in the input video, associated information obtained as third result information by the analysis engine determined in the third control process with associated information provided as external data, and if they differ, may perform a process of overwriting the associated information provided as the external data with the associated information obtained as the third result information. For example, external data provided for scene-related information, such as STATS data, is rewritten according to the analysis results.

[0020] In the above information processing device, the control unit may perform a process of selecting or generating image information corresponding to a scene obtained as the first result information, and synthesizing the image information with the input moving image. For example, it generates an image according to the scene, or selects an appropriate image from among prepared images, and synthesizes such an image into the input video.

[0021] In the above-mentioned information processing device, the second result information may include time information related to the scene obtained as the first result information, and the control unit may perform a process of superimposing the image information on the input video based on the time information obtained in the first result information or the second result information. For example, when synthesizing images according to a scene, the section to be synthesized is set according to time information obtained from the analysis result.

[0022] The information processing method related to the present technology is an information processing method performed by an information processing device, and includes a first control process of determining an analysis engine for scene detection from a plurality of analysis engines based on scene detection information for scene detection in an input video, and a second control process of determining, from the plurality of analysis engines, an analysis engine for obtaining second result information related to the scene based on scene-related information about the scene obtained as first result information by the analysis engine determined in the first control process.

[0023] The program related to the present technology is a program to be executed by an information processing device, and includes a first control process of determining an analysis engine for scene detection from a plurality of analysis engines based on scene detection information for scene detection in an input video, and a second control process of determining, from the plurality of analysis engines, an analysis engine for obtaining second result information related to a scene based on scene-related information about the scene obtained as first result information by the analysis engine determined in the first control process. [Brief description of the drawings]

[0024] [Figure 1] 1 is a diagram illustrating an example of an overall configuration including an information processing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a schematic explanatory diagram showing the flow of each process executed on input data. [Diagram 3] FIG. 1 illustrates an example of the configuration of an information processing device. [Figure 4] FIG. 1 is a block diagram of an information processing device. [Diagram 5] FIG. 1 is a diagram showing examples of analysis engines that can be selected in scene detection and parameters to be assigned thereto. [Figure 6] FIG. 13 is a diagram showing examples of analysis engines that can be selected in scene extraction and the parameters to be assigned. [Figure 7] FIG. 13 is a diagram showing examples of analysis engines that can be selected in the detail description and the parameters assigned thereto. [Figure 8]FIG. 13 is a diagram illustrating an example of a generic screen. [Figure 9] FIG. 13 is a diagram showing an example of a detection setting screen. [Figure 10] FIG. 13 is a diagram showing an example of an image analysis setting screen. [Figure 11] FIG. 13 is a diagram showing another example of the image analysis setting screen. [Figure 12] FIG. 13 is a diagram showing an example of an extraction setting screen. [Figure 13] FIG. 13 is a diagram showing another example of the extraction setting screen. [Figure 14] FIG. 13 is a diagram showing yet another example of the extraction setting screen. [Figure 15] FIG. 13 is a diagram showing yet another example of the extraction setting screen. [Figure 16] FIG. 11 is a diagram for explaining a first example of automatic adjustment of a parameter. [Figure 17] 1 is an example of a flowchart for implementing a first example of automatic parameter adjustment. [Figure 18] FIG. 11 is a diagram for explaining a second example of automatic adjustment of parameters. [Figure 19] FIG. 11 is a diagram for explaining a third example of automatic adjustment of parameters. [Figure 20] FIG. 13 is a diagram for explaining a fourth example of automatic adjustment of parameters. [Figure 21] 13 is an example of a flowchart for implementing a fourth example of automatic parameter adjustment. [Figure 22] FIG. 1 is an example of a functional block diagram of an information processing device for generating a highlight movie. [Figure 23] FIG. 2 is an example of a functional block diagram of an information processing device for performing real-time distribution. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] The embodiments will be described below in the following order. <1. Overall composition> <2. Overview of processing by information processing device> <3. Configuration of information processing device> <4. Analysis engines and parameters used in each phase> <4-1.Analysis engine> <4-2. Scene Detection> <4-3. Scene Extraction> <4-4.Detail Description> <5. Parameter specification by operator> <6. Automatic parameter adjustment> <6-1. First Example> <6-2. Second Example> <6-3.Third Example> <6-4. Example 4> <7. Functional blocks for real-time analysis processing> <7-1. Highlight video generation> <7-2. Real-time distribution> <8.Other> <9. Summary> <10. This Technology>

[0026] <1. Overall composition> FIG. 1 shows an overall configuration including an information processing device 1 according to an embodiment. The configuration shown in FIG. 1 is an example of a configuration for creating a highlight video or the like while live broadcasting a sports match held in a stadium. However, the implementation of the present technology is not limited to this form. For example, various forms are conceivable, such as creating a highlight video of uploaded video data by uploading video data shot by a cameraman to the information processing device 1 and setting various parameters, extracting only specific scenes, extracting character data from audio data included in the video data, and so on.

[0027] One or more imaging devices (not shown) are installed in the stadium 100. Video and still image data captured by the imaging devices are transmitted to a broadcast van 101 located near the stadium 100.

[0028] The relay van 101 is equipped with an antenna device used for transmitting and receiving data, a CCU (Camera Control Unit) that controls the imaging device, a switcher that switches between the imaging devices used for broadcasting and recording, a monitor device for checking the images, etc. The video data (relay video data) created by the relay van 101 is transmitted to, for example, a broadcasting system 102 owned by a broadcasting company.

[0029] The broadcasting system 102 distributes video data to playback devices 103 such as television sets and mobile terminals installed in homes, etc., by appropriately switching between live broadcast footage from a broadcast van 101 and studio footage. This allows viewers to watch a live broadcast of a sports game being held in a stadium 100 on the playback devices 103.

[0030] In addition, the broadcast video data is not only used for live broadcasting but is also processed into a highlight video to be used in later sports news, etc. For this purpose, the worker uploads video data as the broadcast video data to the information processing device 1. In addition, the worker transmits to the information processing device 1 various parameters for determining what kind of highlight video he / she wants to create.

[0031] The information processing device 1 executes various processes (described later) based on the received video data and parameters, and transmits the processed results to the broadcasting system 102.

[0032] The information of the processing result transmitted by the information processing device 1 to the broadcasting system 102 may be metadata for editing the live video data, or may be edited video data (e.g., a highlight video). When transmitting metadata, the metadata is transmitted as information linked to the video data input (uploaded) to the information processing device 1. For example, the metadata is transmitted in association with timestamp information in the input video data.

[0033] When the information processing device 1 transmits metadata to the broadcasting system 102, a process of editing the relayed video data into final moving image data is executed in the broadcasting system 102. In the following description, an example in which the information processing device 1 transmits metadata to the broadcasting system 102 will be described.

[0034] The broadcasting system 102 uploads the received highlight video and a highlight video generated based on the received metadata to, for example, a video distribution site. This allows the user to view the highlight video.

[0035] <2. Overview of processing by information processing device> An overview of the processing executed by the control unit 10 included in the information processing device 1 will be described with reference to Fig. 2. Note that the part enclosed by the dashed line in Fig. 2 is the processing executed by the information processing device 1.

[0036] The information processing device 1 receives a media file as moving image data and parameters used for processing from a broadcasting system 102, for example. First, the information processing device 1 performs scene detection. Scene detection is a process of identifying a location where a specific scene (event) occurred from video data and outputting the scene occurrence time as time information. For example, in an American football (hereinafter simply referred to as "American football") game, a specific scene is a "touchdown," a "field goal," a "long run," a "QB (Quarter Back) sack," etc.

[0037] In scene detection, for example, when detecting a "touch down" scene, the time of the moment of touch down is detected as the scene occurrence time as output. Only one piece of scene occurrence time information may be output, or the number of detected touch downs may be output.

[0038] Furthermore, in scene detection, one type of scene may be detected, or multiple types of scenes may be detected.

[0039] Next, the information processing device 1 performs scene extraction. In the scene extraction phase, the width of time information related to the scene identified with respect to the scene occurrence time identified in the scene detection phase is determined. For example, in scene extraction, an in point and an out point for the identified scene are determined. This makes it possible to determine the cut-out range of the video data.

[0040] Finally, the information processing device 1 performs detail description. In the detail description, a process is performed to specify information to be extracted from video data between the in-point and out-point for the specified scene. For example, in the case of a touchdown scene, the name of the player who made the touchdown is specified. Or, if there is a player who made a pass before that, the name of that player is specified.

[0041] In this way, scene detection identifies the time when a scene occurs, scene extraction identifies the width of time information related to the scene (the range to be cut out), and detail description extracts important matters in the scene. Note that detailed description is not necessarily executed, and the time of scene occurrence and the width of time information related to a scene (extraction range) may be determined by executing only scene detection and scene extraction.

[0042] The information thus obtained in scene detection, scene extraction, and detail description is transmitted, for example, as metadata to the broadcasting system 102 that is the transmission source of the video data. Alternatively, the information processing device 1 may perform the editing process of the video data and transmit the resulting edited video data to the broadcasting system 102. Moreover, instead of transmitting the edited video data to the transmission source, the information processing device 1 may transmit the edited video data to another information processing device, such as a video distribution site.

[0043] <3. Configuration of information processing device> An example of the configuration of the information processing device 1 is shown in FIG. The information processing device 1 includes an AI (Artificial Intelligence) process manager as a control unit 10, a plurality of analysis engines 11 as AI engines, and an interface 12 for inputting and outputting data.

[0044] The analysis engine 11 executes various types of recognition processing, extraction processing, and judgment processing, and functions as a recognizer or extractor whose processing accuracy has been improved by, for example, AI machine learning (such as deep learning).

[0045] The analysis engine 11 is configured to include, for example, a deep neural network, dictionary data (DIC database), and the like.

[0046] The interface 12 passes necessary parameters input from the outside so that the analysis engine 11 can execute various analysis processes, for example, to the control unit 10. Also, the interface 12 receives analysis results using each analysis engine 11 from the control unit 10 and passes them to an external communication unit or the like. This enables the information processing device 1 to perform analysis processing based on the video data and parameters received from the broadcasting system 102 and transmit the analysis results to the broadcasting system 102 .

[0047] The control unit 10 appropriately executes the above-mentioned scene detection processing, scene extraction processing, and detail description processing based on the parameters received from the interface 12. To this end, the control unit 10 selects the optimal analysis engine 11 according to the processing content.

[0048] In order for the control unit 10 to execute such processes, the information processing device 1 specifically has a configuration as shown in FIG.

[0049] The information processing device 1 is configured by an information processing device having a calculation function, such as a general-purpose personal computer, a terminal device, a tablet terminal, or a smartphone.

[0050] A CPU 50 of the information processing device executes various processes according to a program stored in a ROM 51 or a program loaded from a storage unit 58 to a RAM 52. The RAM 52 also stores data and the like necessary for the CPU 50 to execute various processes, as appropriate. The CPU 50, the ROM 51, and the RAM 52 are interconnected via a bus 62. To this bus 62, an input / output interface 54 is also connected.

[0051] An input unit 55 including an operator or an operating device is connected to the input / output interface 54 . For example, the input unit 55 may be various types of operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. Alternatively, voice input may be possible. An operation by an operator is detected by the input unit 55, and a signal corresponding to the input operation is interpreted by the CPU 50.

[0052] A display unit 56 made up of an LCD or an organic EL panel or the like is connected to the input / output interface 54 either integrally or separately. The display unit 56 is a display unit that performs various displays, and is configured, for example, by a display device provided in the housing of the information processing device, or a separate display device connected to the information processing device. The display unit 56 executes display of various UI (User Interface) screens, images of movie content, etc. on the display screen based on instructions from the CPU 50. Also, on the UI screen, various operation menus, icons, messages, etc. are displayed based on instructions from the CPU 50.

[0053] The input / output interface 54 is connected to a storage unit 58 configured with a hard disk, solid-state memory, or the like, and a communication unit 59 configured with a modem, or the like.

[0054] The communication unit 59 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like.

[0055] A drive 60 is also connected to the input / output interface 54 as required, and a removable storage medium 61 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted thereon. The drive 60 allows data files such as image files and various computer programs to be read from the removable storage medium 61. The read data files are stored in the storage unit 58, and images and sounds contained in the data files are output on the display unit 56. In addition, the computer programs and the like read from the removable storage medium 61 are installed in the storage unit 58 as necessary.

[0056] In this information processing device, for example, software for the processing of the present disclosure can be installed via network communication by the communication unit 59 or via the removable storage medium 61. Alternatively, the software may be stored in advance in the ROM 51, the storage unit 58, etc.

[0057] For example, such software constructs a configuration for implementing various functions in the CPU 50 of each information processing device.

[0058] <4. Analysis engines and parameters used in each phase> <4-1.Analysis engine> As described above, the information processing device 1 selects and uses one or more analysis engines 11 from the multiple analysis engines 11 to detect a specific scene in scene detection. Similarly, the information processing device 1 selects one or more analysis engines 11 to determine an extraction range of a scene in scene extraction. Furthermore, the information processing device 1 selects one or more analysis engines 11 to extract detailed information about a scene in detail description.

[0059] The selected analysis engine 11 may be implemented in the information processing device 1, or may be provided in an information processing device other than the information processing device 1, etc.

[0060] In order to detect a specific scene in the scene detection phase, parameters may be set in the analysis engine 11. The parameters may be received from the broadcasting system 102 together with the media file, or the information processing device 1 may set the parameters in the analysis engine 11 based on information received from the broadcasting system 102.

[0061] For example, it is conceivable that the worker designates "touch down" as a scene to be extracted. In this case, the information processing device 1 gives the received information designating "touch down" to the analysis engine 11 as a parameter.

[0062] Alternatively, suppose that an operator wishes to extract exciting scenes regardless of scene type, in which case the operator selects an option indicating that an exciting scene is to be extracted from among a plurality of options as a detection target. The information processing device 1 that has received this information may provide the analysis engine 11 with parameters for setting "touchdown," "field goal," and "QB sack" as scenes to be detected. That is, the parameters designated by the operator may be provided to the analysis engine 11 as they are, or the parameters selected by the information processing device 1 based on the information designated by the operator may be provided to the analysis engine 11.

[0063] An example of classification of the analysis engine 11 according to purpose will be described.

[0064] [Audio-titling engine] The analysis engine 11 performs processing for analyzing the voice data and extracting the text data. The parameters given to the voice subtitling engine include, for example, a parameter for specifying the language.

[0065] [Object Recognition Engine] The analysis engine 11 recognizes objects such as people, animals, and objects captured in video data. The parameters given to the object recognition engine include, for example, a parameter that specifies the type of object.

[0066] [Character Recognition Engine] The analysis engine 11 detects characters that are reflected or superimposed on video image data. The parameters given to the character recognition engine include, for example, a parameter that specifies the type of language (English, Japanese, etc.).

[0067] [Facial Recognition Engine] The analysis engine 11 recognizes the face area of ​​a person appearing in video data. Parameters given to the face recognition engine include, for example, parameters specifying a specific person or a specific gender.

[0068] [Sports data analysis engine] The sports data analysis engine is, for example, an analysis engine 11 that analyzes STATS (statistics) information provided outside the information processing device 1. The STATS information may include, for example, text information that indicates the progress of a game and numerical information that indicates the performance of a player for a specific period of time. The parameters given to the sports data analysis engine include, for example, information for identifying a player and information for identifying a scene.

[0069] [Highlight Generation Engine] The highlight generation engine is an analysis engine 11 that extracts scenes that are highlights of a match. For example, the highlight video may be generated in cooperation with an excitement detection engine, which will be described later. The highlight generation engine may extract time information for generating a highlight video without generating a highlight video. The parameters given to the highlight generation engine include, for example, a parameter that specifies the duration of the highlight video, a parameter that identifies a player in order to generate a highlight video for that player, and the like.

[0070] [Time-saving generation engine] The time-shortened version generation engine is the analysis engine 11 that generates a time-shortened version of video data. For example, it performs a process of identifying scenes where the game is interrupted in order to eliminate scenes where the game is interrupted. The parameters given to the time-saving version generation engine include, for example, a parameter that specifies the duration of the time-saving version video data and a parameter that specifies unnecessary scenes.

[0071] [Emotion recognition engine] The emotion recognition engine is an analysis engine 11 that estimates emotions by analyzing the shapes of each part of the face of a person captured in video data. The emotion recognition engine may analyze emotions by having the function of a face recognition engine, or may analyze the emotions of a subject by cooperating with the face recognition engine. The parameters given to the emotion recognition engine include a parameter for identifying a player and a parameter for identifying a team.

[0072] [Excitement detection engine] The excitement detection engine is an analysis engine 11 that analyzes whether or not a scene is exciting. The parameters given to the excitement detection engine include a volume parameter as a threshold value for determining whether or not a scene is exciting.

[0073] [Camera angle recognition engine] The camera angle recognition engine is an analysis engine 11 that identifies camera angles and changes in camera angles. Parameters given to the camera angle recognition engine include, for example, a parameter specifying a wide shot and a parameter specifying an angle of view. The camera angle recognition engine is also an engine that detects how large the subject is in the image, i.e., whether a certain player is in close-up or the entire game is being shown. For example, if a person's skeletal information is obtained by image recognition, and the obtained skeletal information is of a certain size and only the upper body, the analysis result is that it is a bust-up image.

[0074] [Pan-tilt recognition engine] The pan-tilt recognition engine is an analysis engine 11 that identifies panning and tilting of the camera. Parameters given to the pan-tilt recognition engine include, for example, a parameter that specifies either panning or tilting, and a parameter that specifies a change in angle.

[0075] In this manner, the information processing device 1 is capable of using various analysis engines 11. These various analysis engines 11 may be provided for each type of sport. For example, as object recognition engines, an analysis engine 11 for American football specialized in recognizing American football players, balls, goal posts, etc., and an analysis engine 11 for soccer specialized in recognizing soccer players, balls, goals, etc. may be provided.

[0076] <4-2. Scene Detection> FIG. 5 shows an example of the analysis engine 11 and parameters used to perform scene detection. The analysis engines 11 used for scene detection can be classified according to the processing content into an external data analysis engine, an object detection engine, a voice analysis engine, a camera work analysis engine, a character recognition engine, etc. Note that an analysis engine 11 other than these may be selected for scene detection.

[0077] The external data analysis engine analyzes information acquired from outside the information processing device 1 and performs processing that contributes to identifying the time when a scene occurred, and is, for example, the sports data analysis engine described above. The parameters provided to the external data analysis engine include, for example, STATS information for each sport.

[0078] The object detection engine performs image analysis to analyze what is shown in the image, and is, for example, the object recognition engine described above. Specifically, the object detection engine identifies balls, people, and sports equipment shown in the image. It may also identify superimposed images such as subtitles, character images, and 3D images superimposed on the image. An excitement detection engine that identifies spectators and analyzes their facial expressions and movements to determine whether the venue is excited or not, and an emotion recognition engine that uses image analysis to identify players' faces and analyze their facial expressions, can also be called object detection engines because they perform recognition processing for spectator seats and facial areas.

[0079] The parameters given to the object detection engine are, for example, a dictionary for American football, a dictionary for soccer, a dictionary for tennis, etc. In other words, by giving different dictionaries for different sports as parameters, the object detection engine functions as an object detection engine for American football and an object detection engine for soccer.

[0080] A voice analysis engine is a device that performs processing to analyze voice data, and is, for example, the voice-to-subtitling engine described above. Note that an excitement detection engine that detects excitement in a venue by analyzing changes in the volume of the voice can also be called a voice analysis engine.

[0081] The parameters given to the voice analysis engine are, for example, a vocabulary list for American football, a vocabulary list for soccer, a vocabulary list for tennis, etc. In other words, by giving different vocabulary lists for different sports as parameters, the voice analysis engine functions as a voice analysis engine for American football or a voice analysis engine for soccer.

[0082] A camerawork analysis engine is a device that performs processing to identify the angle of view and analyze changes, such as the camera angle recognition engine and pan-tilt recognition engine mentioned above. Note that an excitement detection engine that analyzes camerawork to detect excitement in the venue can also be said to use a camerawork analysis engine. The parameters given to the camerawork analysis engine are, for example, a dictionary for American football and a dictionary for soccer.

[0083] The character recognition engine performs character recognition processing on subtitles and score displays superimposed on an image, and for example, detects a touchdown scene by performing character recognition processing on an image on which the decorated characters "touchdown" are superimposed. Alternatively, it is possible to identify the movement of the scores superimposed as subtitles by character recognition and estimate the scene that occurred from the amount of change in the scores. For example, the character recognition engine can also function as an excitement detection engine that estimates the excitement level by performing scene detection based on character recognition processing. The parameters given to the character recognition engine are, for example, a dictionary for American football and American football scoring rules.

[0084] These parameters may be assigned according to not only the type of sport but also the type of scene. For example, a parameter for detecting a touchdown scene in American football may be different from a parameter for detecting a field goal.

[0085] <4-3. Scene Extraction> FIG. 6 shows an example of the analysis engine 11 and parameters used to perform scene extraction. The analysis engines 11 used for scene extraction can be classified according to the processing content into a camera switching analysis engine, an exciting section analysis engine, a fixed-second cutout engine, a camerawork analysis engine, an object detection engine, etc. Note that, as in an example described later, an analysis engine 11 other than these may be selected in scene extraction.

[0086] The camera switching analysis engine analyzes whether or not the imaging device is being switched by the switcher, the switching timing, etc. It may function to detect the end of an exciting section by detecting the amount of switching, or it may detect the switching timing and use it to determine the scene extraction range by the highlight generation engine. The parameters given to the camera switching analysis engine are, for example, threshold values ​​for making switching decisions.

[0087] The excitement section analysis engine performs processing to determine the extraction range of an exciting scene, and functions as the excitement detection engine described above. The parameters given to the excitement section analysis engine are, for example, threshold values ​​for determining whether or not an excitement has occurred.

[0088] The fixed seconds extraction engine performs processing to extract a range before and after a specified time, and is used by the above-mentioned highlight generation engine and time-saving version generation engine. The parameters given to the fixed-second cutout engine include, for example, the scene occurrence time and the cutout number of seconds.

[0089] The camerawork analysis engine performs processing to identify the angle of view and analyze changes, and corresponds to the camera angle recognition engine and pan / tilt recognition engine described above. When generating a highlight video or a shortened version video using the results of the camerawork analysis, the camerawork analysis engine is used in the highlight generation engine and the shortened version generation engine. The parameters given to the camerawork analysis engine are, for example, a dictionary for American football and a dictionary for soccer.

[0090] The object detection engine has been explained in the scene detection section, so a detailed explanation will be omitted.

[0091] <4-4.Detail Description> FIG. 7 shows an example of the analysis engine 11 and parameters used to execute the detail description. The analysis engines 11 used in the detail description can be classified according to the processing content into an external data analysis engine, a uniform number recognition engine, a voice analysis engine, a character recognition engine, etc. Note that, as in an example described later, other analysis engines 11 may be selected in the detail description.

[0092] The external data analysis engine is similar to that described for scene detection.

[0093] The uniform number recognition engine performs image analysis to recognize the uniform numbers (uniform numbers) of players in an image. This process is performed, for example, to identify key players in important plays. The uniform number recognition engine is used in highlight generation engines, time-saving version generation engines, etc. The parameters given to the uniform number recognition engine are, for example, a dictionary for American football or a dictionary for soccer.

[0094] The voice analysis engine performs processing to analyze voice data, and is used, for example, to identify key players in important plays, and is therefore used by highlight generation engines, time-saving version generation engines, and the like. The parameters given to the voice analysis engine may be, for example, a vocabulary list for American football or a vocabulary list for soccer, etc. Alternatively, a parameter for identifying a language may be given.

[0095] A character recognition engine detects character information, including superimposed subtitles, from an image, and is used, for example, to extract detailed information (accompanying information) about the content of a play that was made. For example, in an image of an American football game, yardage notations are superimposed at regular intervals from the center line, and a character recognition engine is used to calculate how far a play moved the ball forward from these yardage notations, that is, the number of yards gained in a long run. The parameters given to the character recognition engine are, for example, a dictionary for American football and American football scoring rules.

[0096] <5. Parameter specification by operator> The parameter is specified by the operator using, for example, a UI (User Interface). For example, when specifying a parameter, first, a generic screen 200 as shown in FIG. 8 is provided to the operator. Various items for setting how to analyze video data are arranged on the generic screen 200. Among them, an advanced button 300, a first setting button 301, and a second setting button 302 are arranged as operators for determining the scenes to be extracted in scene detection and scene extraction.

[0097] The first setting button 301 is an operator for displaying the detection setting screen 201 for setting parameters (scene detection information) for detecting a desired scene by scene detection.

[0098] The second setting button 302 is an operator for displaying an extraction setting screen (described later) for setting parameters (scene-related information) for determining the scene cut-out range in a desired manner in scene extraction.

[0099] The advanced button 300 is an operator for displaying a screen for setting parameters (scene detection information and scene related information) used in both scene detection and scene extraction, for example.

[0100] FIG. 9 shows an example of the detection setting screen 201. A detection setting screen 201 has operators for setting parameters for detecting a desired scene. Specifically, a check box ("Point Scene" in the figure) is provided to indicate whether or not to specify a scene type. In addition, when the check box is set to ON, a STATS analysis check box 303 and an edit button 304, which become selectable, as well as a graphic check box 305 and an edit button 306 are provided.

[0101] In addition, the detection setting screen 201 is provided with a checkbox for setting whether or not to detect a particular scene based on a scene zoomed in by the camera ("Close Up" in the figure), a checkbox for setting whether or not to detect a scene based on the facial expressions of people such as players ("Emotion" in the figure), a checkbox for setting whether or not to detect a scene based on cheers ("Cheering" in the figure), and a checkbox for setting whether or not to detect a scene based on camera movement such as the camera angle and camera panning and tilting operations ("Camera Motion" in the figure).

[0102] When graphic check box 305 is set to ON and edit button 306 is pressed, image analysis setting screen 202 shown in Fig. 10 is displayed. Image analysis setting screen 202 has score check box 307 for specifying whether or not to obtain score information from subtitle information superimposed on an image, time check box 308 for specifying whether or not to obtain time information from the subtitle information, and superimposed character image check box 309 for specifying whether or not to analyze a decorated superimposed character image such as "TOUCHDOWN".

[0103] Furthermore, on the image analysis setting screen 202, it may be possible to specify more detailed parameters. More specifically, FIG. 11 shows another example of the image analysis setting screen 202. As shown in FIG. In the image analysis setting screen 202, it is possible to set more detailed parameters in the scene detection process based on the score. As shown in the figure, in addition to the score checkbox, it is also possible to set what kind of tag name is to be given according to the added score when a movement of the score is detected.

[0104] In addition, in order to specify the analysis of the superimposed text image in more detail, the image analysis setting screen 202 is provided with operators for specifying the superimposed text image, an input field for specifying the tag name to be assigned to the corresponding scene, and a field for specifying the threshold value to be used in the analysis.

[0105] In addition, in the detection setting screen and image analysis setting screen, operators for saving the designated state (setting state) and operators for canceling are provided so that the designated parameters can be easily reused.

[0106] When the second setting button 302 shown in FIG. 8 is pressed, the extraction setting screen 203 shown in FIG. 12 is displayed.

[0107] The extraction setting screen 203 has operators for setting parameters for determining the extraction range of the detected scene. Specifically, an analysis type selection field 310 for specifying an analysis type is provided.

[0108] The items displayed at the bottom of the extraction setting screen 203 differ depending on the option selected in the analysis type selection field 310 . 12 is a display mode when "Video Cut Point" is selected in the analysis type selection field 310. In the analysis type selection field 310, other options that can be selected include "Audio Event," "Fixed Length," and "None."

[0109] When "Video Cut Point" is selected, the extraction setting screen 203 has a pre-cut number input field 311 where the number of cut changes before the scene occurrence time can be specified, and a post-cut number input field 312 where the number of cut changes after the scene occurrence time can be specified (see FIG. 12).

[0110] When "Audio Event" is selected, an audio data type selection field 313 that allows the type of audio data to be specified is arranged on the extraction setting screen 203 (see FIG. 13).

[0111] When "Fixed Length" is selected, a pre-seconds input field 314 for specifying an in-point relative to the scene occurrence time and a post-seconds input field 315 for specifying an out-point are provided on the extraction setting screen 203 (see FIG. 14).

[0112] When "None" is selected, no other operators such as selection fields or input fields are arranged on the extraction setting screen 203 (see FIG. 15). In this case, scene extraction processing may be performed based on past trends, or a process may be performed to automatically determine the scene extraction range based on the learning results of the analysis engine 11 selected in scene extraction.

[0113] As specifically described with reference to the respective drawings, the parameters to be given to the analysis engine 11 can be designated by an operator via an interface such as a UI.

[0114] <6. Automatic parameter adjustment> The parameters given to the analysis engine 11 may be adjusted automatically without being specified by an operator. Some examples are described below.

[0115] <6-1. First Example> A first example of automatic parameter adjustment is shown in FIG. The first example of automatic parameter adjustment is an example in which parameters are automatically assigned when analysis processing is performed using analysis engine 11 on video data obtained by shooting an American football game. In scene detection, first, STATS information is analyzed using an external data analysis engine. This allows the time of scene occurrence to be identified. In this example, the scenes to be extracted are, for example, "QB sack," "touchdown," "field goal," and "long run."

[0116] Next, in scene extraction, a different analysis engine 11 is automatically selected for each type of scene detected in scene detection. In addition, parameters to be assigned to the analysis engine 11 at that time are also automatically selected. For example, when the scene type is "QB sack", the camera switching analysis engine is selected for both the detection of the in point and the out point that determine the scene cut-out range. Also, the camera switching analysis engine automatically selects parameters for appropriately identifying the cut-out range of the "QB sack" scene. Since camera switching rarely occurs during QB sack, the time between the time when the camera was switched and the time when the next switching was performed can be identified as the scene cut-out range.

[0117] Also, when the scene type is "touchdown", an object detection engine is selected for detecting the in-point. Specifically, the in-point is determined by detecting a huddle scene, which is a scene in which players gather between plays, using the object detection engine. Also, parameters for appropriately determining the in-point are automatically selected and given to the object detection engine. Then, the camerawork analysis engine is selected for detecting the out point. In American football, when a touchdown is successful, it is common to aim for an additional point with a kick. In that case, the camera follows the ball that is kicked, so a tilt movement of the camera occurs. Therefore, the time point at which the tilt down movement of the camera detected by the camerawork analysis engine occurs is set as the out point. Furthermore, parameters for appropriately determining the out point are automatically selected and provided to the camerawork analysis engine.

[0118] Furthermore, when the scene type is "field goal", the camerawork analysis engine is selected for both in-point and out-point detection. In a field goal, the camera follows the kicked ball, causing the camera to tilt up and down. When detecting an in-point, a tilt-up motion is detected, and when detecting an out-point, a tilt-down motion is detected. Also, parameters for appropriately determining the in-point and out-point in a "field goal" scene are automatically selected and applied to the camerawork analysis engine.

[0119] Furthermore, when the scene type is a "long run," the excitement section analysis engine is selected for both in-point and out-point detection. Since a long run is a scene in which the audience gets excited, the start of the excitement is detected as the in-point, and the end of the excitement is detected as the out-point. Furthermore, parameters for appropriately determining the in-point and out-point in a "long run" scene are automatically selected and applied to the excitement section analysis engine.

[0120] In this way, the selected analysis engine 11 may be different or the same depending on the scene type. When the same analysis engine 11 is selected, the efficiency of the process is improved.

[0121] Detail Description uses an external data analysis engine to extract detailed information, which provides additional information about the extracted scene.

[0122] FIG. 17 shows the flow of processing executed by the control unit 10 in the first example. In step S101, the control unit 10 acquires the STATS information. Next, in step S102, the control unit 10 performs scene detection processing on the acquired STATS information, that is, detects a detection target scene from the acquired STATS information.

[0123] Next, in step S103, the control unit 10 performs branching processing according to the detection result of the scene detection. Specifically, it is determined whether or not the scene to be detected has been detected in the scene detection.

[0124] If the acquired STATS information indicates the detection of the scene to be detected, the control unit 10 selects the analysis engine 11 and parameters to be used in scene extraction according to the scene type in step S104. That is, the control unit 10 selects the analysis engine 11 suitable for detecting in-points and out-points according to the scene type and assigns parameters.

[0125] Next, in step S105, the control unit 10 performs an in-point detection process using the selected analysis engine 11 and parameters to extract time information. In addition, in step S106, the control unit 10 performs an out-point detection process using the selected analysis engine 11 and parameters to extract time information.

[0126] In step S107, the control unit 10 outputs time information of the IN point and OUT point as metadata. In addition, in step S107, information such as the scene type and the scene occurrence time of the scene acquired in the scene detection executed in the processing of step S102 is also output as metadata.

[0127] On the other hand, if it is determined in step S103 that the scene to be detected could not be detected in the scene detection, the control unit 10 determines in step S108 whether or not the scene to be detected includes detailed information. That is, it determines whether or not detailed information that supplements the already detected scene can be acquired from the acquired STATS information. The detailed information is, for example, information on the uniform numbers and names of players who played an active role in the scene, and information indicating the content of the play (such as the number of yards gained), as described above.

[0128] If it is determined that the detailed information can be acquired, the control unit 10 proceeds to step S107 and performs processing to output the acquired detailed information, scene type, and the like as metadata.

[0129] Moreover, if it is determined in step S108 that the detailed information cannot be acquired, the control unit 10 ends the series of processes shown in FIG.

[0130] The control unit 10 appropriately performs the process shown in Fig. 17 every time STATS information is acquired. That is, the control unit 10 executes the process shown in Fig. 17 every time STATS information is acquired as information indicating one play. By repeating the process shown in Fig. 17, it is possible to specify the time when a scene to be detected occurred, determine the extraction range, and extract detailed information from video data of one game. In this embodiment, for example, in a case where STATS information is updated in parallel with the progress of a match, the information is acquired in step S101 each time the STATS information is updated, and the subsequent processes are executed, and as the match progresses, the occurrence time of the scene to be detected, the IN point and the OUT point are detected, and detailed information is extracted. For example, information required for editing a highlight video in parallel with the progress of the match is provided from the information processing device 1 as metadata. Then, in order to enable editing of the highlight video, the metadata is provided, for example, linked to timestamp information of the input video data in order to clarify which part of the video data the metadata relates to.

[0131] <6-2. Second Example> A second example of automatic parameter adjustment is shown in Fig. 18. Note that the process executed by the control unit 10 in the second example is similar to that shown in Fig. 17, so a description of the process flow will be omitted.

[0132] The second example is an example in which the sport type being photographed is "soccer." In scene detection, as in the case of American football, an external data analysis engine is used to analyze STATS information and identify the time when the scene occurred. In this example, the scenes to be extracted are, for example, "penalty kick" and "goal".

[0133] In scene extraction, different analysis engines 11 and parameters are automatically selected for each type of scene detected in scene detection. When the scene type is "PK", the camera switching analysis engine is selected for both detection of in-points and out-points. Also, parameters for appropriately determining in-points and out-points in a "PK" scene are automatically selected and applied to the camera switching analysis engine.

[0134] Furthermore, when the scene type is "goal", the character recognition engine and the fixed-second extraction engine are used, and the parameters to be assigned to each of the analysis engines 11 are automatically determined. Specifically, before detecting the in-point and out-point, the character recognition engine first performs character recognition to obtain score information from the subtitles showing the score superimposed on the captured image. In other words, the character recognition engine identifies the timing at which the score changes.

[0135] In addition, if the time of occurrence of a goal scene detected using an external data analysis engine in the scene detection phase differs from the time of occurrence of a goal scene detected using a character recognition engine in the scene extraction phase, the STATS information obtained from outside and stored in the memory unit 58 of the information processing device 1 may be modified.

[0136] After the occurrence time of the goal scene is identified by the character recognition engine, a fixed seconds extraction engine is used to determine a range of a fixed number of seconds before and after the occurrence time of the scene.

[0137] Detail Description uses an external data analysis engine to extract detailed information, which provides additional information about the extracted scene.

[0138] <6-3.Third Example> A third example of automatic parameter adjustment is shown in Fig. 19. Note that the process executed by the control unit 10 in the third example is similar to that shown in Fig. 17, and therefore a description thereof will be omitted.

[0139] The third example is an example in which the sport type to be photographed is "tennis." In scene detection, like American football and soccer, an external data analysis engine is used to analyze STATS information and identify the time when a scene occurred. In this example, the scenes to be extracted are, for example, "break points" and "match points."

[0140] In scene extraction, different analysis engines 11 and parameters are automatically selected for each type of scene detected in scene detection. When the scene type is "break point", a character recognition engine for extracting character information such as score 40 or A (Advantage) from the subtitles superimposed on the captured image is selected as the analysis engine 11 for detecting the in-point, and parameters suitable for the analysis of the character recognition engine are automatically selected and assigned. Also, a camera switching analysis engine for detecting the timing at which the scene changes is selected for detection of the out-point, and parameters suitable for the analysis of the camera switching analysis engine are automatically selected and assigned.

[0141] When the scene type is "match point", a character recognition engine for extracting score information from subtitles superimposed on a captured image is selected as the analysis engine 11 for detecting an in-point, and parameters suitable for the analysis of the character recognition engine are automatically selected and assigned. Also, an excitement section analysis engine for detecting the end of the excitement at the match venue is selected for detecting an out-point, and parameters suitable for the analysis of the excitement section analysis engine are automatically selected and assigned.

[0142] Detail Description uses an external data analysis engine to extract detailed information, which provides additional information about the extracted scene.

[0143] <6-4. Example 4> In each of the first to third examples, a specific scene is detected by using STATS information as external data in scene detection. In the fourth example, a specific scene is detected by analyzing the received video data instead of external data (see Figure 20). In this example, the analysis is performed on video data captured during an American football game.

[0144] In scene detection, an object detection engine, a voice analysis engine, and a character recognition engine are first selected as the analysis engine 11 to be applied to the uploaded video data. The scenes to be extracted are, as in the first example, scenes such as "QB sack," "touchdown," "field goal," and "long run."

[0145] Specifically, the system detects touchdown scenes by analyzing objects captured in the images and detecting when the ball lands in the end zone. However, since there is a possibility of false positives, the system improves the accuracy of determining that the detected scene is indeed a touchdown scene by using a voice analysis engine to analyze the excitement in the stadium and the words of the commentator, and a character recognition engine to analyze decorative characters superimposed on the image. That is, the various analysis engines 11 are made to work together to detect the scene to be detected.

[0146] In scene extraction, depending on the scene type, a camera switching analysis engine, an object detection engine, a camera work analysis engine, an exciting section analysis engine, etc. are appropriately selected for in-point detection, and optimal parameters are automatically selected and applied depending on the selected analysis engine 11 and scene type.

[0147] In addition, in out-point detection, a camera switching analysis engine, a camera work analysis engine, an exciting section analysis engine, etc. are appropriately selected, and optimal parameters are automatically selected and applied depending on the selected analysis engine 11 and the scene type. The selection of these analysis engines 11 in scene extraction is the same as in the first example, so a detailed description will be omitted.

[0148] In the detailed description, unlike the above-mentioned examples, a different analysis engine 11 is selected for each scene to be detected. For example, when the scene type is "QB sack" or "field goal", it is not necessary to select the analysis engine 11. In this way, the decision not to use the analysis engine 11 is also one of the options.

[0149] In addition, when the scene type is "touchdown," a camerawork analysis engine 11 is selected to detect scenes with bust-ups or close-ups of players, and a uniform number recognition engine is selected to identify the player who made the successful touchdown or the player who made an important pass, and appropriate parameters are automatically assigned to each of them.

[0150] When the scene type is "long run," a character recognition engine is selected that detects the yardage notation superimposed on the field image of the game in order to detect how long the play is, i.e., to detect the number of yards gained, and parameters are automatically assigned.

[0151] When STATS information can be acquired as external data, the local STATS information acquired and stored in the storage unit 58 may be compared with the detailed information extracted in the detail description. In this case, if the detailed information differs, the local STATS information may be corrected using the detailed information extracted in the detail description.

[0152] In this way, the analysis engine 11 selected and the parameters assigned may differ or may be the same depending on the scene type, and multiple analysis engines 11 may be selected or it may be possible to choose not to use any analysis engine 11.

[0153] The flow of processing executed by the control unit 10 in the fourth example is shown in Fig. 21. Note that the same processes as those shown in Fig. 17 are denoted by the same reference numerals and the description thereof will be omitted as appropriate.

[0154] In step S201, the control unit 10 selects the analysis engine 11 and parameters to be used in scene detection, and performs analysis processing. Specifically, the control unit 10 selects an object detection engine, a voice analysis engine, and a character recognition engine, and performs analysis processing according to the assigned parameters. If the scene to be detected cannot be detected in this analysis process, the series of processes shown in FIG. 21 may be ended.

[0155] If one or more target scenes are detected, the control unit 10 executes the processes of steps S104, S105, S106, S202, S203, and S107 that follow for each detected scene.

[0156] Specifically, in step S104, the control unit 10 selects the analysis engine 11 and parameters to be used in scene extraction according to the scene type and the like. Furthermore, the control unit 10 extracts the time of the IN point according to the scene type in step S105, and similarly extracts the time of the OUT point in step S106.

[0157] Next, in step S202, the control unit 10 selects the analysis engine 11 to be used in the detail description for the scene extraction range and assigns parameters to the analysis engine 11 for each scene type.

[0158] Next, in step S203, the control unit 10 extracts detailed information using an analysis engine.

[0159] Finally, in step S107, the control unit 10 outputs the information extracted in each phase as metadata.

[0160] 21 is an example of a case where complete video data that has been shot from the start to the end of a match is uploaded. That is, after detecting the occurrence times of all scenes to be detected from each scene recorded in the video data in step S201, the processes in and after step S104 are executed for the detected scenes, so that the process does not loop.

[0161] In order to detect the target scene and output metadata in parallel with the progress of the match, the process flow shown in Fig. 21 may be modified as shown in Fig. 17 to have a program configuration including a loop for repeatedly executing the process of each step. Specifically, the analysis process of step S201 is executed for the latest video data uploaded at any time, and if it is determined that the target scene is included, the process of step S104 and subsequent steps is executed, and if the target scene is not included, the process waits until the next uploaded video data and executes the process of step S201 again.

[0162] <7. Functional blocks for real-time analysis processing> <7-1. Highlight video generation> FIG. 22 shows an example of the configuration of functional blocks built in the control unit 10 for generating a highlight video while receiving game footage in real time. The control unit 10 is configured with functions such as an external data acquisition unit 400, a scene detection unit 401, a first scene occurrence time identification unit 402, a program (PGM) signal acquisition unit 403, a second scene occurrence time identification unit 404, a scene identification element detection unit 405, a scene change detection unit 406, a cut-out range determination unit 407, a first camera image acquisition unit 408, a second camera image acquisition unit 409, a cut-out unit 410, a combination unit 411, a transcoding unit 412, and a transmission unit 413.

[0163] The external data acquisition unit 400 performs a process of acquiring the above-mentioned STATS information as external data. The scene detection unit 401 receives STATS information from the external data acquisition unit 400 and detects a scene to be detected. The first scene occurrence time identification unit 402 identifies the scene occurrence time based on the STATS information. That is, the external data acquisition unit 400, the scene detection unit 401, and the first scene occurrence time identification unit 402 are functions for scene detection using the analysis engine 11 and parameters.

[0164] The program signal acquisition unit 403 acquires a PGM output signal, which is a video signal and audio signal on which effects, subtitles, etc. are superimposed. The PGM output signal is, for example, a signal for video to be distributed to each home.

[0165] The second scene occurrence time determination unit 404 performs OCR (Optical Character Recognition) processing on the image of the PGM output signal to determine the occurrence time of the scene to be detected. This processing can be performed efficiently by narrowing down the section to be processed based on the scene occurrence time determined from the STATS information as external data. This processing also verifies whether the scene occurrence time based on the STATS information is correct.

[0166] The scene identification element detection unit 405 detects decorative characters and the like superimposed on the image by performing OCR processing on the image based on the PGR output signal. This processing is a processing for confirming that the detected scene is definitely the scene to be detected.

[0167] The second scene occurrence time identification unit 404 and the scene identification element detection unit 405 are used in the scene detection phase.

[0168] The scene change detection unit 406 detects scene changes by analyzing camera work. The cut-out range determination unit 407 specifies an IN point and an OUT point for determining the cut-out range of a scene. The scene change detection unit 406 and the extraction range determination unit 407 are used in scene extraction.

[0169] The first camera image acquisition unit 408 and the second camera image acquisition unit 409 acquire images from cameras capturing the game. The first camera that is the acquisition target of the first camera image acquisition unit 408 and the second camera that is the acquisition target of the second camera image acquisition unit 409 are different imaging devices. In addition, when three or more cameras are set up at the match venue, the camera image acquisition units may be provided in the same number as the cameras.

[0170] The cut-out unit 410 performs cut-out processing of the video according to the scene cut-out range from the video of the first camera and the video of the second camera in accordance with the cut-out range.

[0171] A combining unit 411 joins together the cut-out videos to create one piece of video data (highlight video data).

[0172] The transmission unit transmits the generated highlight video data to an external information processing device.

[0173] In the example shown in Fig. 22, an external data analysis engine, a character recognition engine, etc. are selected as the analysis engine 11 used in scene detection. Also, a camera work analysis engine, etc. are selected as the analysis engine 11 used in scene extraction. Detail description does not have to be performed, and may be performed in the same manner as the above-mentioned example.

[0174] It should be noted that by using the functional configuration shown in FIG. 22, not only the highlight video but also the shortened version video and the like can be generated.

[0175] <7-2. Real-time distribution> FIG. 23 shows an example of the configuration of functional blocks built in the control unit 10 when video data to be distributed is generated by capturing game footage and superimposing CG (Computer Graphics), effect images, subtitles, etc. in approximately real time.

[0176] The control unit 10 is configured with functions such as an external data acquisition unit 500, a scene detection unit 501, a superimposed image selection unit 502, a first camera image acquisition unit 503, a second camera image acquisition unit 504, a first image analysis unit 505, a second image analysis unit 506, and an image superimposition unit 507.

[0177] The external data acquisition unit 500 performs a process of acquiring the above-mentioned STATS information as external data. The scene detection unit 501 detects a scene to be detected by receiving STATS information from the external data acquisition unit 500. Specifically, the analysis engine 11 is used to identify the occurrence time of the scene to be detected.

[0178] The superimposed image selection unit 502 selects a superimposed image according to the detected scene. For example, if the detected scene is a "touchdown" scene, a CG image or a 3D image decorated with the character string "TOUCHDOWN" is selected.

[0179] The first camera video acquisition unit 503 acquires moving image data (captured image data) from the first camera capturing the match. Similarly, the second camera image acquisition unit 504 acquires video data from the second camera. As in the previous example, three or more camera image acquisition units may be provided, or one camera image acquisition unit may be capable of acquiring video image data from a plurality of cameras.

[0180] The first video analysis unit 505 performs an analysis process on the video data of the first camera and determines whether the target scene detected from the STATS information can be detected from the video data. The second video analysis unit 506 performs an analysis process on the video data of the second camera and determines whether the target scene can be detected from the video data.

[0181] The processing of first video analysis unit 505 and second video analysis unit 506 is also processing for selecting appropriate video data for conveying the detection target scene from the video data of either the first camera or the second camera. In other words, if it is determined that the video of the first camera is more appropriate for conveying to the viewer that a "touchdown" scene has occurred, the video data of the first camera is selected as the video data to be distributed (broadcast).

[0182] It is not necessary to select video data from one camera for one detection target scene. For example, for a 20-second scene, video data from a first camera may be selected for the first 5 seconds, video data from a second camera may be selected for the next 10 seconds, and video data from the first camera may be selected again for the last 5 seconds. This allows the game to be broadcast using the most suitable video data from multiple cameras.

[0183] It should be noted that the state shown in FIG. 23 shows a state in which the video data from the first camera is selected. In addition, if it is determined that the video data from the first camera and the video data from the second camera are appropriate for conveying the scene to be detected, the video data from the second camera may be delivered after the video data from the first camera, or the display area may be divided so that images from multiple cameras are displayed on a single screen.

[0184] The image superimposition unit 507 performs a process of superimposing the superimposition image selected by the superimposition image selection unit 502 on the video data selected by the video analysis unit. In this process, for example, a process of adjusting the size of the superimposition image and a process of determining the position of superimposition are performed. Moreover, the time point on the time axis from which the superimposed image is to be superimposed may be based on the results of scene extraction processing (e.g., time information of the in-point and out-point). However, in this case, superimposing the superimposed image from the in-point may dampen the excitement. For example, if the superimposed image for touchdown is superimposed from the in-point, the image of "TOUCHDOWN" will be displayed before the time of occurrence of the touchdown, and the result of the play will be known in advance. In such a case, it is desirable to determine the section to be superimposed based on the results of scene detection processing (e.g., the time of occurrence of the scene) and the results of scene extraction processing (e.g., time information of the out-point).

[0185] The moving image data on which the superimposed image is superimposed is output as distribution video data. During times other than when the scene to be detected is occurring, there may be no appropriate superimposed image, and the video from the first camera or the second camera may simply be output as the distribution video data.

[0186] <8.Other> The scene occurrence time specified by scene detection in the above example and the time information determined as the in point and out point of the scene extraction range may be the elapsed time from the start of the match or may be real time.

[0187] Furthermore, the various functions provided by the information processing device 1 described above may be used when a user edits a video using application software. For example, when editing a video, if an operation using an analysis process using an AI engine (the above-mentioned analysis engine 11) is performed, video data is uploaded to an analysis system (the above-mentioned information processing device 1) constructed on the cloud based on the processing of the application software. The analysis system transmits the video data and metadata of the analysis result to a terminal (such as a mobile terminal) used by the user. This allows the user to use various analysis functions and editing functions on the cloud, improving work efficiency.

[0188] In this case, the analysis system constructed on the cloud may be capable of using not only various analysis engines 11 managed within the analysis system, but also analysis engines 11 external to the analysis system (for example, analysis engines 11 whose functions are provided by information processing devices of other companies). This allows the functions of a wide variety of analysis engines 11 to be provided, making it possible to appropriately respond to a wide range of user requests and provide analysis results that are highly satisfying to users. In addition, since the analysis system does not need to prepare all of the analysis engines 11, it is possible to construct a compact analysis system, shortening the time required for construction and keeping development costs low.

[0189] <9. Summary> The above-described information processing device 1 includes a control unit 10 that performs a first control process (a process executed in scene detection) and a second control process (a process executed in scene extraction). The first control process is a process of determining an analysis engine 11 for scene detection from multiple analysis engines 11 based on scene detection information (parameters assigned for scene detection processing) for scene detection in an input video (video data from a camera installed at the game venue). In addition, the second process is a process of determining, from among multiple analysis engines, an analysis engine 11 for obtaining second result information related to the scene (e.g., time information of the in-point and out-point) based on scene-related information (parameters assigned for scene extraction) about the scene obtained as first result information (e.g., information on the time of scene occurrence) by the analysis engine 11 determined in the first control process. The input video is assumed to be, for example, content produced based on images captured by one or more cameras for a certain sport. Also assumed is a system having multiple analysis engines 11 for analyzing such videos. In such a system, an information processing device having a function as a control unit 10 determines, in a first control process, one or more analysis engines 11 for detecting a specific scene in the video as first result information, and, in a second control process, determines one or more analysis engines 11 for identifying second result information related to the scene. According to the information processing device 1 of the embodiment, the analysis engine 11 that performs analysis as scene detection and scene extraction is appropriately selected according to the user's intention, the type of sport such as sports, the situation of the action shooting site, and the like. This improves the analysis accuracy, and improves the performance of the system that provides the user with appropriate metadata, image data, and the like. Note that the subject of shooting is a sport here, but is not limited to this. For example, it may be a concert filmed, a speech filmed for recording, a dinner party filmed, other daily life, or a security camera image. In this sense, the type of sport can be said to be a video type.

[0190] In the information processing device 1, the second result information may include time information related to the scene obtained as the first result information. That is, in the second control process, the analysis engine 11 for obtaining the second result information including at least time information related to the scene is determined. By obtaining time information related to a scene detected by scene detection, such as a time code specifying a scene section, as second result information, scene extraction, i.e., information extraction that serves the purpose of scene extraction, can be realized.

[0191] In the information processing device 1, the second result information includes scene start information regarding the start of the scene obtained as the first result information and scene end information regarding the end of the scene obtained as the first result information, and in the second control process (processing executed in scene extraction), an analysis engine 11 for identifying the scene start information and an analysis engine 11 for identifying the scene end information may be determined. That is, in the second control process, an analysis engine 11 is determined for obtaining second result information including scene start information and scene end information as the second result information, and the analysis engine 11 corresponding to each of the scene start information and scene end information is determined. For example, as explained with reference to Figs. 6 and 16, in order to obtain an IN point as scene start information and an OUT point as scene end information in scene extraction, the analysis engine 11 is determined for each. Since appropriate conditions for proper detection of the in point and the out point are assumed for each detection, detecting the in point / out point using different analysis engines 11 is useful for improving accuracy. Of course, in some cases it may be appropriate to detect the in point / out point using the same analysis engine 11, but in any case, by specifying the analysis engine 11 for each of the scene start information and scene end information, it is possible to facilitate the detection process and improve the detection accuracy.

[0192] In the information processing device 1, the scene-related information includes scene type information, and in the second control process (processing executed in scene extraction), the analysis engine 11 for obtaining the second result information may be determined based on the scene type information. That is, in the second control process, the analysis engine 11 for obtaining the second result information is determined, and the corresponding analysis engine 11 is determined depending on the type of the target scene. For example, as described with reference to FIG. 16, the control unit 10 determines the analysis engine 11 for scene extraction depending on the type of scene, for example, whether it is a "touchdown" scene or a "field goal" scene. Regarding the analysis for scene extraction, appropriate conditions for obtaining the second result information are assumed according to the type of scene. Therefore, by performing scene extraction using the analysis engine 11 according to the type of sport, it is appropriate to improve accuracy.

[0193] In the information processing device 1, the second result information includes scene start information regarding the start of the scene obtained as the first result information and scene end information regarding the end of the scene obtained as the first result information, and in the second control process (processing executed in scene extraction), an analysis engine 11 for identifying the scene start information and an analysis engine 11 for identifying the scene end information may be determined depending on the type of scene obtained as the first result information. That is, in the second control process, an analysis engine 11 is determined for obtaining second result information including scene start information and scene end information as the second result information, and the analysis engine 11 for obtaining the scene start information and the analysis engine 11 for obtaining the scene end information are determined depending on the type of scene. For example, as explained with reference to FIG. 16, in order to obtain an IN point as scene start information and an OUT point as scene end information, the analysis engine 11 is determined according to the type of scene. Since the appropriate conditions for detecting in-points and out-points vary depending on the scene type, it is possible to improve accuracy by determining the analysis engine 11 that detects in-points and out-points according to the scene type.

[0194] In the information processing device 1, the second result information includes time information related to the scene obtained as the first result information, and the control unit 10 may perform a third control process (processing executed in the detail description) of determining, from among multiple analysis engines 11, an analysis engine 11 for obtaining third result information by analyzing a section in the video identified by the time information. When a section within a moving image is identified as the second result information with respect to a certain scene, an analysis engine 11 that performs a detailed analysis of that section is determined. The control unit 10 also determines the analysis engine 11 for analyzing the detail description. In particular, by analyzing the detail description of a section extracted by scene extraction, such as the section from the in point to the out point, detailed information can be extracted without unnecessarily increasing the processing load.

[0195] In the information processing device 1, in the third control process (the process executed in the detail description), the analysis engine 11 for obtaining the third result information may be determined based on the scene type information. That is, in the third control process, the analysis engine 11 for obtaining the third result information is determined, and the corresponding analysis engine 11 is determined depending on the type of the target scene. By determining an appropriate analysis engine 11 for detailed description depending on the type of sport or scene, it is possible to improve the accuracy and simplify the analysis process.

[0196] In the information processing device 1, scene-related information is managed in correspondence with scene detection information (parameters assigned for scene detection processing), and the scene-related information (parameters assigned for scene extraction processing) may be set in accordance with the setting of the scene detection information. Management is performed so that scene-related information is identified in response to scene detection information for scene detection being identified. The control unit 10 specifies scene detection parameters (scene detection information) based on, for example, the type of sport or the type of scene, and also specifies scene extraction parameters (scene-related information) in response to this. In this way, appropriate parameters for scene detection and scene extraction are set according to the type of sport and the type of scene.

[0197] In the information processing device 1, the control unit 10 may set scene detection information in response to an input of a scene type. For example, scene detection information is set according to a scene type inputted according to a user operation or automatic determination. The control unit 10 sets scene detection parameters (scene detection information) by inputting a scene type according to a user operation, automatic determination, etc. This makes it possible to determine an appropriate analysis engine 11 according to the scene to be detected.

[0198] In the information processing device 1, the control unit 10 may set scene detection information (parameters given for scene detection processing) in response to the input of the sport type. For example, the scene detection information is set according to the type of sport (for example, the type of sport such as a sport) that is input according to a user operation or automatic determination. The control unit 10 sets scene detection parameters (scene detection information) by inputting the type of sport according to a user operation, automatic judgment, etc. This makes it possible to determine an appropriate analysis engine 11 according to the scene to be detected.

[0199] In the information processing device 1, the control unit 10 may generate metadata based on the first result information (e.g., information on the time when a scene occurs) and the second result information (e.g., time information on the in-point and out-point), and perform a process of associating the generated metadata with the input video. For example, information on the detected scene and its time information are generated as metadata, and are treated as information related to the input video. The control unit 10 generates metadata based on the scene detection analysis result (first result information) and the scene extraction analysis result (second result information), associates it with the input video, and outputs it as the analysis result (see FIG. 16, etc.). This makes it possible to transmit the metadata associated with the input video as the analysis target to the user side, and to provide the user with an appropriate analysis service.

[0200] In the information processing device 1, the control unit 10 may generate metadata based on first result information (e.g., scene occurrence time information), second result information (e.g., in-point and out-point time information), and third result information (e.g., information about players who performed well in the scene), and perform a process of associating the generated metadata with the input video. For example, information about the detected scene, its time information, and further detailed information are generated as metadata, and are treated as information related to the input video. The control unit 10 generates metadata based on the analysis results of scene detection (first result information), scene extraction (second result information), and detail description, associates it with the input video, and outputs it as the analysis result (see figure ). This makes it possible to transmit metadata containing more detailed information associated with the input video as the analysis target to the user side, thereby enhancing the content of the analysis service provided to the user.

[0201] In the information processing device 1, the control unit 10 may compare time information (e.g., information on the time when the scene occurs) obtained as the first result information for a scene that is to be detected in the input video with time information provided as external data (e.g., time information in STATS information), and if they differ, may perform a process of overwriting the time information provided as the external data with the time information obtained as the first result information. For example, external data provided as STATS information regarding scene time codes is rewritten according to the analysis results. This allows the STATS information to be revised based on the analysis results, and provides data for editing based on information consistent with the analysis results.

[0202] In the information processing device 1, the control unit 10 may compare, for a scene that has been detected in the input video, the associated information (detailed extraction information) obtained as the third result information by the analysis engine 11 determined in the third control process with the associated information provided as external data, and if they differ, perform a process of overwriting the associated information provided as external data with the associated information obtained as the third result information. For example, external data provided for scene-related information, such as STATS information, is rewritten according to the analysis results. This allows the accompanying information provided in the STATS information to be modified in accordance with the analysis results, and provides data for editing based on information consistent with the analysis results.

[0203] In the information processing device 1, the control unit 10 may select or generate image information (superimposed image) corresponding to the scene obtained as the first result information, and perform a process of synthesizing the image information with the input moving image. For example, it generates an image that matches the scene (for example, an image with the character string "TOUCHDOWN" decorated), or selects an appropriate image from among the prepared images. Such an image is then composited into the input video. This makes it possible to provide users with images that have been synthesized with appropriate CG images for different sections of a scene, thereby expanding the content of cloud services. Of course, instead of providing an image, a CG image and information on the section within the video to be composited may be provided, allowing the user to easily perform the composite process.

[0204] In the information processing device 1, the second result information includes time information related to the scene obtained as the first result information, and the control unit 10 may perform a process of superimposing image information on the input video based on the time information obtained in the first result information or the second result information. For example, when synthesizing images according to a scene, the section to be synthesized is set according to time information obtained from the analysis result. The control unit 10 can superimpose the composite image in an appropriate section within the moving image based on the time information (time information of the in-point and out-point) obtained by scene extraction. In this case, for example, when compositing a "touchdown" image in a touchdown scene, if the compositing is done not from the in point to the out point, but from the time code at the time the scene occurs to the out point (or even up to a point just before the out point), a video can be generated in which the "touchdown" image is displayed from the moment of touchdown. In other words, by effectively using the time information determined by scene detection and scene extraction, it becomes possible to automatically composite images in appropriate sections.

[0205] The information processing method executed by the information processing device 1 includes a first control process of determining an analysis engine 11 for scene detection from among a plurality of analysis engines 11 based on scene detection information for scene detection in an input video, and a second control process of determining, from the plurality of analysis engines 11, an analysis engine 11 for obtaining second result information related to the scene based on scene-related information about the scene obtained as first result information by the analysis engine 11 determined in the first control process.

[0206] In addition, the program executed by the information processing device 1 includes a first control process for determining an analysis engine 11 for scene detection from among the multiple analysis engines 11 based on scene detection information for scene detection in an input video, and a second control process for determining, from the multiple analysis engines 11, an analysis engine 11 for obtaining second result information related to the scene based on scene-related information about the scene obtained as first result information by the analysis engine 11 determined in the first control process.

[0207] Such an information processing method and program make it possible to configure the various functional configurations described above, and to achieve the above-mentioned actions and effects.

[0208] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained. Furthermore, the above-mentioned examples can be combined in any manner as long as the combination is not impossible.

[0209] <10. This Technology> (1) a first control process for determining an analysis engine for scene detection from among a plurality of analysis engines based on scene detection information for scene detection for an input video; a second control process for determining, based on scene-related information about a scene obtained as first result information by the analysis engine determined in the first control process, an analysis engine for obtaining second result information related to the scene from among a plurality of analysis engines; An information processing device comprising a control unit that performs the above. (2) The second result information includes time information related to the scene obtained as the first result information. The information processing device according to (1) above. (3) the second result information includes scene start information relating to a start of a scene obtained as the first result information, and scene end information relating to an end of a scene obtained as the first result information, In the second control process, an analysis engine for identifying the scene start information; an analysis engine for identifying the scene end information; Determine The information processing device according to (1) or (2) above. (4) the scene-related information includes scene type information, In the second control process, an analysis engine for obtaining second result information is determined based on the scene type information. An information processing device according to any one of (1) to (3) above. (5) the second result information includes scene start information relating to a start of a scene obtained as the first result information, and scene end information relating to an end of a scene obtained as the first result information, In the second control process, An analysis engine for identifying the scene start information and an analysis engine for identifying the scene end information are determined according to a type of the scene obtained as the first result information. An information processing device according to any one of (1) to (4) above. (6) the second result information includes time information related to the scene obtained as the first result information; The control unit performs a third control process of determining, from a plurality of analysis engines, an analysis engine for obtaining third result information obtained by analyzing a section in the video identified by the time information. An information processing device according to any one of (1) to (5) above. (7) In the third control process, an analysis engine for obtaining third result information is determined based on the scene type information. The information processing device according to (6) above. (8) the scene-related information is managed in association with the scene detection information; The scene-related information is set in response to the setting of the scene detection information. An information processing device according to any one of (1) to (7) above. (9) The control unit sets the scene detection information in response to an input of a scene type. An information processing device according to any one of (1) to (8) above. (10) The control unit sets the scene detection information in response to an input of a sport type. An information processing device according to any one of (1) to (9) above. (11) The control unit is generating metadata based on the first result information and the second result information; The generated metadata is associated with the input video. An information processing device according to any one of (1) to (10) above. (12) The control unit is The first result information; The second result information; and The third result information; and Generate metadata based on The generated metadata is associated with the input video. The information processing device according to (6) or (7) above. (13) The control unit is Regarding the scene to be detected in the input video, Time information obtained as the first result information; Time information provided as external data; and if they are different, a process of overwriting the time information provided as the external data with the time information obtained as the first result information is performed. The information processing device according to claim 1. An information processing device according to any one of (1) to (12) above. (14) The control unit is For the scenes to be detected in the input video, associated information obtained as third result information by the analysis engine determined in the third control process; Accompanying information provided as external data; If they are different, the associated information provided as the external data is overwritten with the associated information obtained as the third result information. The information processing device according to (6) above. (15) The control unit is Selecting or generating image information corresponding to the scene obtained as the first result information; The image information is synthesized with the input moving image. An information processing device according to any one of (1) to (14) above. (16) the second result information includes time information related to the scene obtained as the first result information; The control unit performs a process of superimposing the image information on the input moving image based on time information obtained from the first result information or the second result information. The information processing device according to (15) above. (17) a first control process for determining an analysis engine for scene detection from among a plurality of analysis engines based on scene detection information for scene detection for an input video; a second control process for determining, based on scene-related information about a scene obtained as first result information by the analysis engine determined in the first control process, an analysis engine for obtaining second result information related to the scene from among a plurality of analysis engines; The information processing method is performed by an information processing device. (18) a first control process for determining an analysis engine for scene detection from among a plurality of analysis engines based on scene detection information for scene detection for an input video; a second control process for determining, based on scene-related information about a scene obtained as first result information by the analysis engine determined in the first control process, an analysis engine for obtaining second result information related to the scene from among a plurality of analysis engines; A program for causing an information processing device to execute the above.

[0210] The correspondence between the terms described in the present technology and the terms described in the embodiments will be described below.

[0211] In response to a request from an operator or in response to an automatically set purpose, a process of selecting the analysis engine 11 is performed as a "first control process" in the scene detection phase. Moreover, various parameters are assigned to the analysis engine 11 selected in the scene detection phase as “scene detection information.” In other words, the scene detection information is information for detecting a specific scene. For example, information for identifying a type of sport (such as American football or soccer), information for identifying a scene to be extracted (such as a touchdown or a field goal), and various dictionary information are part of the scene detection information. In the scene detection phase, information about the detected scene, such as the time when the scene occurred, is detected (extracted) as "first result information".

[0212] That is, in scene detection, the scene detection information and video data are input to perform a first control process, and after determining the analysis engine 11, an analysis process is executed and the first result information is output.

[0213] In the scene detection phase, a process for selecting an analysis engine 11 for identifying the range of the detected scene is performed as a "second control process". Furthermore, various parameters are assigned as "scene-related information" to the analysis engine 11 selected in the scene extraction phase. For example, the excitement determination threshold and the switching determination threshold are part of the scene-related information used to identify the start (in-point) and end (out-point) of a particular scene. In the scene extraction phase, time information about in points and out points, etc. is extracted as the "second result information."

[0214] That is, in scene extraction, the video data, the first result information, and the scene-related information are input, and the second control process is performed to determine the analysis engine 11, after which the analysis process is executed and the second result information is output.

[0215] In the detail description phase, a process for selecting an analysis engine 11 for obtaining detailed information about the identified scene is performed as a "third control process." In addition, various parameters are assigned as "detailed extraction information" to the analysis engine 11 selected in the detail description phase. For example, dictionary information for each sport and scoring rule information for each sport are parameters for extracting detailed information, and therefore can be considered part of the detailed extraction information. In the detail description phase, detailed information about the detected scene, such as the uniform numbers and names of the players who performed well, is extracted as the "third result information."

[0216] That is, in the detail description, the video data, the first result information, the second result information, and the detailed extraction information are used as input to perform the third control process, and after determining the analysis engine 11, the analysis process is executed and the third result information is output.

[0217] The scene detection information given to the process related to scene detection and the scene related information given to the process related to scene extraction may be managed in association with each other. For example, the parameters for scene detection and the parameters used to determine the extraction range may be associated with information on the sport type or information on the scene type, so that the two parameters are associated with each other.

[0218] The parameters (scene detection information, scene related information, detailed extraction information) given to the analysis engine 11 in each phase include parameters designated by the operator and parameters that are set automatically. Furthermore, parameters are not necessarily required in each phase, and parameters may be unnecessary depending on the analysis engine 11 selected. [Explanation of symbols]

[0219] 1. Information processing device 10 Control section 11 Analysis Engine

Claims

1. a first control process for determining an analysis engine for scene detection from among a plurality of analysis engines based on scene detection information for scene detection for an input video; a second control process for determining, based on scene-related information about a scene obtained as first result information by the analysis engine determined in the first control process, an analysis engine for obtaining second result information related to the scene from among a plurality of analysis engines; An information processing device comprising a control unit that performs the above.

2. The second result information includes time information related to the scene obtained as the first result information. The information processing device according to claim 1 .

3. the second result information includes scene start information relating to a start of a scene obtained as the first result information, and scene end information relating to an end of a scene obtained as the first result information, In the second control process, an analysis engine for identifying the scene start information; an analysis engine for identifying the scene end information; Determine The information processing device according to claim 1 .

4. the scene-related information includes scene type information, In the second control process, an analysis engine for obtaining second result information is determined based on the scene type information. The information processing device according to claim 1 .

5. the second result information includes scene start information relating to a start of a scene obtained as the first result information, and scene end information relating to an end of a scene obtained as the first result information, In the second control process, determining an analysis engine for identifying the scene start information and an analysis engine for identifying the scene end information according to a type of the scene obtained as the first result information; The information processing device according to claim 1 .

6. the second result information includes time information related to the scene obtained as the first result information, The control unit performs a third control process of determining, from a plurality of analysis engines, an analysis engine for obtaining third result information obtained by analyzing a section in the moving image that is identified by the time information. The information processing device according to claim 1 .

7. In the third control process, an analysis engine for obtaining third result information is determined based on scene type information. The information processing device according to claim 6.

8. the scene-related information is managed in association with the scene detection information; The scene-related information is set in response to the setting of the scene detection information. The information processing device according to claim 1 .

9. The control unit sets the scene detection information in response to an input of a scene type. The information processing device according to claim 1 .

10. The control unit sets the scene detection information in response to an input of a sport type. The information processing device according to claim 1 .

11. The control unit is generating metadata based on the first result information and the second result information; The generated metadata is associated with the input video. The information processing device according to claim 1 .

12. The control unit is The first result information; The second result information; and The third result information; and Generate metadata based on The generated metadata is associated with the input video. The information processing device according to claim 6.

13. The control unit is Regarding the scene to be detected in the input video, Time information obtained as the first result information; Time information provided as external data; If they are different, the time information provided as the external data is overwritten with the time information obtained as the first result information. The information processing device according to claim 1 .

14. The control unit is For the scenes to be detected in the input video, associated information obtained as third result information by the analysis engine determined in the third control process; Accompanying information provided as external data; If they are different, the associated information provided as the external data is overwritten with the associated information obtained as the third result information. The information processing device according to claim 6.

15. The control unit is Selecting or generating image information corresponding to the scene obtained as the first result information; The image information is synthesized with the input moving image. The information processing device according to claim 1 .

16. the second result information includes time information related to the scene obtained as the first result information, The control unit performs a process of superimposing the image information on the input moving image based on time information obtained from the first result information or the second result information. The information processing device according to claim 15.

17. a first control process for determining an analysis engine for scene detection from among a plurality of analysis engines based on scene detection information for scene detection for an input video; a second control process for determining, based on scene-related information about a scene obtained as first result information by the analysis engine determined in the first control process, an analysis engine for obtaining second result information related to the scene from among a plurality of analysis engines; The information processing method is performed by an information processing device.

18. a first control process for determining an analysis engine for scene detection from among a plurality of analysis engines based on scene detection information for scene detection for an input video; a second control process for determining, based on scene-related information about a scene obtained as first result information by the analysis engine determined in the first control process, an analysis engine for obtaining second result information related to the scene from among a plurality of analysis engines; A program for causing an information processing device to execute the above.

Images