Information processing device and information processing system

Abstract

To generate an appropriate highlight moving image matching details of content by taking a priority of either or both of a video highlight scene and sound highlight scene according to the details of the content.SOLUTION: According to the present invention, an information processing device includes first determination means for determining a video highlight scene and a sound highlight scene on the basis of feature amounts of video data and sound data included in a moving image file, acquisition means for acquiring information about a time zone of the highlight scene determined by the first determination means, reception means for receiving priority information from an electronic apparatus, second determination means for determining a time zone of a highlight moving image on the basis of the priority information received from the reception means and the information about the time zone of the highlight scene, and notification means for notifying the electronic apparatus of the time zone of the highlight moving image determined by the second determination means.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing system. [Background technology]

[0002] Conventionally, there has been a highlight scene extraction device that can more accurately extract highlight scenes from video by combining scene composition features from video data with audio features from audio information separated from the video when determining highlight scenes (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-187687 A Summary of the Invention [Problem to be solved by the invention]

[0004] Conventionally, highlight scenes are extracted by using audio features from audio information in addition to scene composition features of frame images of video data. In this case, frame images without scene composition features are not judged as highlight scenes even if the audio features meet a certain condition. For example, in a video image of a concert such as an orchestra, even if the music is exciting and the audio features meet a certain condition, there are cases where the scene composition features do not meet the certain condition because the screen is dark or the faces of many performers are small. Such cases are not selected as highlight scenes.

[0005] Even if both the audio features and the scene composition features satisfy predetermined conditions and are extracted as a highlight scene, depending on the scene, the parts that satisfy the respective predetermined conditions of the audio features and the scene composition features may be out of order. In such a case, it is conceivable that the video will become unnatural, such as the highlight scene starting in the middle of an audio conversation or ending with the audio conversation still in progress.

[0006] Therefore, an object of the present invention is to generate a highlight video video that is appropriate for the content by prioritizing either or both of the video highlight scenes and the audio highlight scenes depending on the content. [Means for solving the problem]

[0007] In order to achieve the above object, the information processing device of the present invention is an information processing device capable of communicating with an electronic device, and is characterized in having a first determination means for determining video highlight scenes based on features of video data included in a video file and for determining audio highlight scenes based on features of audio data included in the video file, an acquisition means for acquiring information regarding time periods of the video highlight scenes and the audio highlight scenes determined by the first determination means, a receiving means for receiving from the electronic device priority information regarding whether to prioritize either or both of the video highlight scenes and the audio highlight scenes, a second determination means for determining a time period of a highlight video based on the priority information received by the receiving means and information regarding the time periods of the video highlight scenes and the audio highlight scenes, and a notification means for notifying the electronic device of the time period of the highlight video determined by the second determination means. Effect of the Invention

[0008] According to the present invention, by prioritizing either or both of the video highlight scenes and the audio highlight scenes depending on the content, it is possible to generate a highlight moving image appropriate to the content. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating the overall configuration of a system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram illustrating a configuration of an electronic device 102 or an information processing device 103 according to the present embodiment. [Diagram 3] FIG. 1 is a block diagram showing a configuration of an imaging device 101 according to the present embodiment. [Figure 4] 13 is a flowchart illustrating a process in which the information processing device 103 generates a highlight moving image including video highlight scenes and audio highlight scenes. [Diagram 5] 13 is a flowchart illustrating a process in which the information processing device 103 generates a highlight moving image including video highlight scenes and audio highlight scenes. [Figure 6] 13 is a flowchart illustrating a process in which the information processing device 103 generates a highlight moving image including video highlight scenes and audio highlight scenes. [Figure 7] 13 is a flowchart illustrating a process in which the information processing device 103 generates a highlight moving image including video highlight scenes and audio highlight scenes. [Figure 8] 13 is a display example of information related to the time period of each highlight scene notified to the electronic device 102. [Figure 9] 13 is a display example of information related to the time period of each highlight scene notified to the electronic device 102. [Figure 10] 13 is a display example of information related to the time period of each highlight scene notified to the electronic device 102. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0011] This embodiment is intended for a case where a video file generated by an imaging device is uploaded to a cloud and data processing is performed by an information processing device on the cloud that is equipped with the system of the present invention. Also, a user refers to anyone who uses the system of the present invention.

[0012] 1 is a diagram illustrating the overall configuration of a system according to this embodiment. The system according to this embodiment includes an imaging device 101, an electronic device 102, and an information processing device 103.

[0013] The imaging device 101 is used to generate a moving image file to be processed by an information processing device 103 described later. The moving image file used in this embodiment is not limited to a specific format, codec, or the like.

[0014] The electronic device 102 is a terminal device of a user who uses the system of the present invention, and is also used as a medium for acquiring information notified from an information processing device 103 described later.

[0015] The information processing device 103 performs data processing on the video file generated and uploaded by the imaging device 101 to notify information regarding the time periods of highlight scenes, which will be described later, and notifies the electronic device 102 of the information regarding the time periods of highlight scenes, which will be described later.

[0016] A moving image file generated by the imaging device 101 is uploaded from a user's local environment to the information processing device 103 via a network 104. The network 104 may be wireless or wired, depending on its form.

[0017] In addition, in the system of the present invention, the means for transmitting the moving image file to the information processing device is not limited to a specific form, and other means than communication via a network may be used. In this embodiment, the moving image file may be uploaded directly from the imaging device 101 to the information processing device 103, or may be uploaded from the imaging device 101 to the information processing device 103 via the electronic device 102.

[0018] The information processing device described here may take the form of a server, may be a device having the same functions as the respective servers, or may be a server that aggregates the functions of the respective servers.

[0019] In this embodiment, the electronic device 102 is described as an example of a data output destination from the information processing device 103, but this is not limited to this depending on the format of the output data. When a server or the like of another system is connected as the output destination of the information processing device 103, it is not necessary to output data in text file format or displayed on a UI, and data may be output in a format such as a database.

[0020] In this embodiment, a video file is uploaded to a cloud from the imaging device 101 or the electronic device 102, and processing is performed on the cloud in which the system of the present invention is installed. The present embodiment is not limited to this, and a form in which the system of the present invention is installed in the electronic device 102 is also conceivable.

[0021] FIG. 2 is a block diagram illustrating the configuration of the electronic device 102 or the information processing device 103 according to this embodiment.

[0022] 2, the electronic device 102 and the information processing device 103 have a common configuration.

[0023] The electronic device 102 and the information processing device 103 each include a keyboard 204 , a pointing device (PD) 205 , a central processing unit (CPU) 206 , and a read only memory (ROM) 207 .

[0024] Furthermore, the electronic device 102 and the information processing device 103 each include a RAM (Random Access Memory) 208 , a HDD (Hard Disk Drive) 209 , a recording media drive 210 , a network I / F 211 , and a bus 212 .

[0025] The display 201 displays, for example, icons, messages, menus, and other user interface information for managing the electronic device 102 .

[0026] An image to be displayed on the display 201 is drawn in the VRAM 202. The image data generated in the VRAM 202 is transferred to the display 201 in accordance with a predetermined rule, and the image is displayed on the display 201.

[0027] The BMU 203 controls, for example, data transfer between memories (eg, between the VRAM 202 and other memories) and data transfer between memories and each I / O device (eg, the network I / F 211).

[0028] The keyboard 204 has various keys for inputting characters, etc. The PD 205 is used, for example, to point to icons, menus, and other content displayed on the display 201 or to drag and drop objects.

[0029] The CPU 206 controls each device based on an OS stored in the ROM 207, the HDD 209, or the recording media drive 210, and control programs such as programs described below.

[0030] The ROM 207 stores various control programs and data. The RAM 208 has a work area for the CPU 206, an area for saving data during error processing, an area for loading control programs, and the like.

[0031] The HDD 209 stores various control programs executed in the electronic device 102 and data such as temporarily stored data.

[0032] The network I / F 211 communicates with other electronic devices 102, printers, etc. via a network.

[0033] The bus 212 includes an address bus, a data bus, and a control bus.

[0034] The control program can be provided to the CPU 206 from the ROM 207, the HDD 209, or the recording media drive 210, or can be provided from another server or the like via the network I / F 211 over the network.

[0035] 3 is a block diagram showing the configuration of an image capturing device 101 according to this embodiment. The image capturing device 101 is, for example, a terminal such as a digital camera 300 having an image capturing function.

[0036] The photographing lens 301 includes a zoom lens and a focus lens. The shutter 302 has an aperture function. The imaging unit 303 is composed of a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor) element that converts an optical image into an electrical signal.

[0037] The A / D converter 304 converts the analog image data input from the imaging unit 303 into digital image data. The barrier 305 covers the photographing lens 301 and the like to prevent the imaging system, including the photographing lens 301, the shutter 302, and the imaging unit 303, from becoming dirty or being damaged.

[0038] The image processing unit 306 performs image processing such as color conversion processing such as predetermined pixel interpolation and resizing processing such as reduction on the digital image data from the A / D converter 304 or the digital image data from the memory control unit 307, and outputs the result as digital image data. The digital image data is compressed and encoded according to a compression encoding standard such as JPEG or MPEG, and stored in one of various image file formats to generate an image file. The generated moving image file contains video data and audio data.

[0039] In addition, the image processing unit 306 performs a predetermined calculation process using the captured image data, and the system control unit 308 performs exposure control and distance measurement control based on the calculation results. The digital image data output from the A / D converter 304 is temporarily recorded in a memory 309 via the image processing unit 306 and a memory control unit 307, or via the memory control unit 307 only.

[0040] The system control unit 308 controls the entire image capturing apparatus 101. The system control unit 308 executes a program stored in a non-volatile memory 312 to realize processing of the image capturing apparatus 101.

[0041] Memory 309 temporarily records digital image data from A / D converter 304 and image data to be displayed on display unit 310. Memory 309 has a storage capacity sufficient to store a predetermined number of still images, a predetermined length of moving images, and audio. Memory 309 also serves as a memory for displaying images (video memory).

[0042] D / A converter 311 converts image data for image display stored in memory 309 into an analog signal and supplies it to display unit 310. In this way, the image data for display temporarily stored in memory 309 is displayed by display unit 310 via D / A converter 311. Display unit 310 performs display according to the analog signal from D / A converter 311 on a display device such as an LCD (Liquid Crystal Display).

[0043] The non-volatile memory 312 is an electrically erasable and recordable memory, such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), etc. The non-volatile memory 312 stores constants for the operation of the system control unit 308, control programs such as a program for performing cooperation, which will be described later, and the like.

[0044] The system memory 313 is, for example, a RAM, and develops constants and variables for the operation of the system control unit 308, programs read from the non-volatile memory 312, and the like.

[0045] A mode changeover switch 314, a first shutter switch 315, a second shutter switch 316, and an operation unit 317 are operation means for inputting various operation instructions to the system control unit 308. The mode changeover switch 314 switches the operation mode of the system control unit 308 to one of a still image recording mode, a moving image recording mode, a playback mode, etc.

[0046] A first shutter switch 315 is turned on when a shutter button provided on the image capture device 101 is pressed halfway (instruction to prepare for shooting) during operation, and generates a first shutter switch signal SW1.

[0047] The second shutter switch 316 is turned on when the shutter button is fully pressed (photographing instruction) and generates a second shutter switch signal SW2. The system control unit 308 starts a series of photographing processing operations from reading a signal from the imaging unit 303 to writing image data to the recording medium 325 in response to the second shutter switch signal SW2.

[0048] Each operating member of the operating unit 317 acts as various function buttons with appropriate functions assigned for each scene, for example, by selecting and operating various function icons displayed on the display unit 310. Examples of the function buttons include a confirmation button, an end button, a back button, an image forward button, a jump button, a filter button, and an attribute change button. For example, when the menu button is pressed, a menu screen in which various settings can be made is displayed on the display unit 310. The user can intuitively make various settings using the menu screen displayed on the display unit 310, the four-way button, and the SET button.

[0049] A controller wheel 318 is a rotatable operating member included in the operating unit 317, and is used together with the direction buttons to specify a selection item, etc. A system control unit 308 controls each component of the imaging device 101 based on a pulse signal.

[0050] Furthermore, the system control unit 308 can determine the angle at which the controller wheel 318 has been rotated, the number of rotations, etc., based on the pulse signal. Note that the controller wheel 318 may be any operating member that can detect a rotation operation.

[0051] The controller ring 319 is a rotary operation member included in the operation unit 317, and can be operated to rotate around the optical axis around the lens barrel. For example, by operating the controller ring 319, an electrical pulse signal corresponding to the amount of rotation (amount of operation) is generated. The system control unit 308 controls each component of the imaging device 101 based on the pulse signal. In addition, when a function switching button of the controller ring 319 included in the operation unit 317 is pressed, a menu screen that allows the function assigned to the controller ring 319 to be changed is displayed on the display unit 310. The controller ring 319 and the controller wheel 318 are normally used to select a mode item or change a value.

[0052] The power switch 320 switches the power on and off.

[0053] The power supply control unit 321 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to be energized, etc. The power supply control unit 321 detects whether a battery is installed, the type of battery, and the remaining battery level. The power supply control unit 321 also controls the DC-DC converter based on the detection results and instructions from the system control unit 308, and supplies the required voltage for the required period to each component, including the recording medium 325.

[0054] The power supply unit 322 is composed of a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like.

[0055] The communication unit 323 transmits the images recorded on the recording medium 325 to an external device.

[0056] The I / F 324 is an interface with a recording medium 325, and the recording medium 325 is composed of a semiconductor memory, a magnetic disk, or the like.

[0057] The NET I / F 326 is a network I / F that is controlled by the system control unit 308 and performs communication via the network.

[0058] 3 is an example of a general imaging device, and the shutter 302, the mode change switch 314, the display unit 310, and the like are not necessarily required for the imaging device 101.

[0059] 4 to 7 are flowcharts for explaining the process of generating a highlight moving image including video highlight scenes and audio highlight scenes by the information processing device 103. Fig. 4 particularly explains the flow up to the point where the information processing device 103 receives a user request from the electronic device 102 as to whether the video or audio highlight scene is to be prioritized.

[0060] In S401, the information processing device 103 acquires a moving image file transmitted from the user's electronic device 102. As described above, the means for transmitting the moving image file to the information processing device is not limited to a specific form.

[0061] In S402, the information processing device 103 extracts and compares scene features from video data included in the video file to be processed, and determines video highlight scenes based on the features of the video data. For example, by extracting and comparing the features of each frame image of the video data, scenes with more intense action can be detected in a sports video, and by extracting and comparing the features of a person's face in a frame image, scenes in which a person is shown up close can be detected. In this way, video highlight scenes are determined. Here, the method of extracting and comparing the features and extracting the characteristic scenes is not limited to a specific method.

[0062] In S403, the information processing device 103 performs voice recognition on the audio data included in the video file, extracts and compares audio features, and determines audio highlight scenes based on the features of the audio data. In this embodiment, by extracting and comparing audio features, scenes with distinctive audio, such as a scene where a musical instrument is being played, a scene where the audience is cheering, or a scene where the subject person is talking, are detected and determined as audio highlight scenes.

[0063] In S404, the information processing device 103 acquires information about the time periods of the video highlight scenes and audio highlight scenes determined in S402 and S403. The time period of each highlight scene refers to the time period from the start time to the end time of the highlight scene in the moving image.

[0064] In S405, the information processing device 103 notifies the electronic device 102 of information related to the time period of each highlight scene acquired in S404. Since the user can check the information related to the time period of each highlight scene notified to the electronic device 102, the user can select a priority highlight scene by looking at the degree of overlap of the highlight scenes. The method of notification from the information processing device 103 to the electronic device 102 is not limited to a specific method. The notification may be made by displaying it on the screen of the electronic device 102 like a GUI, or may be made by displaying it in a format such as a text file in which linked time information is written.

[0065] Fig. 8 is an example of information relating to the time period of each highlight scene notified to the electronic device 102 in S405. Fig. 8 illustrates an example in which the information processing device 103 notifies the electronic device 102 of each piece of information, and the electronic device 102 displays a GUI in accordance with the notified information.

[0066] Reference numeral 801 denotes a time axis, and audio highlight scenes 1 to 4 and video highlight scenes 1 to 3 are the highlight scenes determined in S402 and S403. To show the start time and end time of each highlight scene, the time from the start time to the end time is indicated by an arrow, and it is shown where each highlight scene is located on the time axis.

[0067] Reference numeral 802 denotes a seek bar that is normally provided in music / video playback software and the like, and exists on the time axis 801. By operating this seek bar, the user can check a video at a specific time by playing and displaying it, and it is also possible to check videos before and after each highlight scene, and videos during the time period of each highlight scene.

[0068] In S406, the information processing device 103 receives information regarding whether the user is to prioritize a video highlight scene or an audio highlight scene from the electronic device 102. Here, the information received is information regarding whether the user is to prioritize a video highlight scene or an audio highlight scene, or whether the user is to prioritize both a video highlight scene and an audio highlight scene.

[0069] An example of a user prioritizing a video highlight scene is when the user wants to record the moment when an airplane takes off from an airport as a highlight video, where the user prioritizes a video highlight scene extracted from the characteristics of the airplane's movement when taking off over an audio highlight scene extracted from audio characteristics such as the engine sound or cheers before takeoff.

[0070] Another example of a user prioritizing audio highlight scenes is when the user wants to record a scene of a singer singing at a live performance as a highlight video, in which the subject of the video does not move and the scene is not extracted as a video highlight scene, so audio highlight scenes are prioritized.

[0071] [Highlight video generation process when video highlight scenes and audio highlight scenes do not overlap] The process performed when the time zone of a video highlight scene and the time zone of an audio highlight scene do not overlap based on the time zones of the highlight scenes acquired in S404 will be described with reference to the flowchart of FIG.

[0072] In S501, the information processing device 103 determines whether the time period of the video highlight scene and the time period of the audio highlight scene overlap. If it is determined that the time period of the video highlight scene and the time period of the audio highlight scene overlap, the process proceeds to S601, and if not, the process proceeds to S502.

[0073] In S502, the information processing device 103 determines whether or not to prioritize both the video highlight scene and the audio highlight scene. If it is determined that both the video highlight scene and the audio highlight scene are prioritized, the process proceeds to S503, and if not, the process proceeds to S504.

[0074] In S503, the information processing device 103 determines the video of the time period in which the video highlight scenes and audio highlight scenes are included as the time period of the highlight video, and notifies the electronic device 102 of information relating to the time period.

[0075] In S504, the information processing device 103 determines whether or not to prioritize the video highlight scene. If it is determined that the video highlight scene is prioritized, the process proceeds to S505, and if not, the process proceeds to S506.

[0076] In S505, the information processing device 103 determines the time period of the video highlight scene as the time period of the highlight video, and notifies the electronic device 102 of information relating to the time period.

[0077] In S506, the information processing device 103 determines that the audio highlight scene is to be prioritized. In S507, the time period of the audio highlight scene is determined as the time period of the highlight video, and the electronic device 102 is notified of information relating to the time period.

[0078] In S508, the information processing device 103 receives from the electronic device 102 an instruction as to whether or not to generate a highlight video for the determined time period, and if the received instruction is to “not generate a highlight video” (No in S508), the information processing device 103 returns to the processing of S406.

[0079] If the received instruction is to "generate a highlight video" (Yes in S508), the portions corresponding to the time period of the highlight video are extracted from each of the video data and audio data. A highlight video file including the extracted video data and audio data is generated (S509). Then, the generated highlight video file is recorded in the HDD 209, and this process ends. The generated highlight video file may be transmitted to the electronic device 102.

[0080] [Highlight video generation process when video highlight scenes and audio highlight scenes overlap and both video highlight scenes and audio highlight scenes are prioritized] The process performed when it is determined in S501 that the time period of the video highlight scene and the time period of the audio highlight scene overlap will be described with reference to the flowchart of FIG.

[0081] In particular, FIG. 6 is a flowchart in the case where the time zone of a video highlight scene and the time zone of an audio highlight scene overlap, and both the video highlight scene and the audio highlight scene are given priority.

[0082] In S601, the information processing device 103 determines whether or not to prioritize both the video highlight scene and the audio highlight scene. If it is determined that both the video highlight scene and the audio highlight scene are to be prioritized, the process proceeds to S602, and if not, the process proceeds to S701.

[0083] In S602, the information processing device 103 determines whether the time period of the video highlight scene includes the time period of the audio highlight scene. If it is determined that the time period of the video highlight scene includes the time period of the audio highlight scene, the process proceeds to S603, and if not, the process proceeds to S605.

[0084] In S603, the information processing device 103 determines the period from the start time to the end time of the video highlight scene as the time period of the highlight video. Then, in S604, the information processing device 103 notifies the electronic device 102 of information relating to the time period.

[0085] 9 shows an example of displaying information related to the time period of a highlight video generated when both video highlight scenes and audio highlight scenes are prioritized on the screen of electronic device 102 by notifying the user. In FIG. 9, the time period of video highlight scene 1 includes the time period of audio highlight scene 1, and in this case, the period from the start time to the end time of video highlight scene 1 is determined as the time period of highlight video 1. Note that the notification process in S604 may be omitted.

[0086] In S605, the information processing device 103 determines whether or not the time period of the video highlight scene includes only the start time of the audio highlight scene. If it is determined that the time period of the video highlight scene includes only the start time of the audio highlight scene, the process proceeds to S606, and if not, the process proceeds to S608.

[0087] In S606, the information processing device 103 determines the period from the start time of the video highlight scene to the end time of the audio highlight scene as the time period of the highlight video. Then, in S607, the information processing device 103 notifies the electronic device 102 of information related to the time period. This notification allows the user to check in advance whether a highlight video for a desired time period can be generated. Then, the user can adjust the time period of the highlight video or change it so that either the video highlight scene or the audio highlight scene is given priority.

[0088] In FIG. 9, the time period of video highlight scene 2 includes only the start time of audio highlight scene 3. In this case, the time period from the start time of video highlight scene 2 to the end time of audio highlight scene 3 is determined as the time period of highlight video 3.

[0089] In S608, the information processing device 103 determines that the time period of the video highlight scene includes only the end time of the audio highlight scene.

[0090] In S609, the information processing device 103 determines the period from the start time of the audio highlight scene to the end time of the video highlight scene as the time period of the highlight video. Then, in S610, the information processing device 103 notifies the electronic device 102 of information related to the time period. This notification allows the user to check in advance whether a highlight video for a desired time period can be generated. The user can also adjust the time period of the highlight video, or change the priority to be given to either the video highlight scene or the audio highlight scene.

[0091] In FIG. 9, the time period of video highlight scene 3 includes only the end time of audio highlight scene 4. In this case, the time period from the start time of audio highlight scene 4 to the end time of video highlight scene 3 is determined as the time period of highlight video 4.

[0092] In addition, in FIG. 9, audio highlight scene 2 does not overlap with any video highlight scenes, and since both video highlight scenes and audio highlight scenes are given priority, the time period of audio highlight scene 2 is determined as the time period of highlight video 2.

[0093] In S611, the information processing device 103 receives from the electronic device 102 an instruction as to whether or not to generate a highlight video for the determined time period, and if the received instruction is to “not generate a highlight video” (No in S611), the information processing device 103 returns to the processing of S406.

[0094] If the received instruction is to "generate a highlight video" (Yes in S611), portions corresponding to the time period of the highlight video are extracted from each of the video data and audio data. A highlight video file including the extracted video data and audio data is generated (S612). The generated highlight video file is recorded in the HDD 209, and this process ends. The generated highlight video file may be transmitted to the electronic device 102.

[0095] [Highlight video generation process when video highlight scenes and audio highlight scenes overlap and either the video highlight scenes or the audio highlight scenes are prioritized] The process when it is determined in S601 that neither the video highlight scene nor the audio highlight scene is prioritized (either one of them is prioritized) will be described with reference to the flowchart in FIG.

[0096] In S701, the information processing device 103 determines whether or not to prioritize the video highlight scene. If it is determined that the video highlight scene is prioritized, the process proceeds to S702, and if not, the process proceeds to S711.

[0097] In S702, the information processing device 103 determines whether the time period of the video highlight scene includes the time period of the audio highlight scene. If it is determined that the time period of the video highlight scene includes the time period of the audio highlight scene, the process proceeds to S703, and if not, the process proceeds to S705.

[0098] In S703, the information processing device 103 determines the video from the start time to the end time of the video highlight scene as the time period of the highlight video. Then, in S704, the information processing device 103 notifies the electronic device 102 of information related to the time period.

[0099] 10 is an example of information related to the time periods of highlight videos generated when video highlight scenes are prioritized, displayed on the screen of electronic device 102. In Fig. 10, the time period of video highlight scene 1 includes the time period of audio highlight scene 1, and in this case, the time period from the start time to the end time of video highlight scene 1 is determined as the time period of highlight video 1.

[0100] In S705, the information processing device 103 determines whether or not the time period of the video highlight scene includes only the start time of the audio highlight scene. If it is determined that the time period of the video highlight scene includes only the start time of the audio highlight scene, the process proceeds to S706, and if not, the process proceeds to S708.

[0101] In S706, the information processing device 103 determines the time period from the start time of the video highlight scene to the end time of the conversation detected in the audio highlight scene at the end time of the video highlight scene as the time period of the highlight video. Then, in S707, the information processing device 103 notifies the electronic device 102 of information related to the time period.

[0102] For example, if the playback of a highlight video ends in the middle of a conversation between people in the video, it is unnatural and inappropriate. Therefore, the time until the end of the conversation detected at the end time of the video highlight scene is determined as the time period of the highlight video. For example, the time of the position of the period at which the conversation detected at the end time of the video highlight scene ends is set as the end time of the highlight video. Note that the phrases and punctuation positions of the conversation used here are extracted from the conversation by detecting the conversation in the voice data using the voice recognition performed in S403 and converting it into text data (conversation).

[0103] This notification allows the user to check in advance whether a highlight video can be generated for the desired time period, and the user can then adjust the time period of the highlight video or change the priority of video highlight scenes over audio highlight scenes.

[0104] 10, the time period of video highlight scene 2 and the time period of audio highlight scene 3 overlap, and audio highlight scene 3 does not end even though video highlight scene 2 ends. In this case, the time period until the end of the conversation detected at the end time of video highlight scene 2 is determined as the time period of highlight video 2.

[0105] In S708, the information processing device 103 determines that the time period of the video highlight scene includes only the end time of the audio highlight scene.

[0106] In S709, the information processing device 103 determines the period from the start of the conversation detected in the audio highlight scene at the start time of the video highlight scene to the end time of the video highlight scene as the period of the highlight video. Then, in S710, the information processing device 103 notifies the electronic device 102 of information related to the period of the highlight video.

[0107] For example, if playback of a highlight video starts in the middle of a conversation between people in the video, it is unnatural and inappropriate. Therefore, the time from the start of the conversation detected at the start time of the video highlight scene to the end of the video highlight scene is determined as the time period of the highlight video. For example, the time of the position of a period in the conversation immediately before the conversation detected at the start time of the video highlight scene is determined as the start time of the highlight video. Note that the phrases and positions of punctuation in the conversation used here are assumed to be extracted from the conversation detected in the voice data by the voice recognition performed in S403 and converted into text data (conversation).

[0108] This notification allows the user to check in advance whether a highlight video for the desired time period can be generated, and the user can then adjust the time period himself or herself, or change the priority of video highlight scenes over audio highlight scenes.

[0109] 10, the time period of video highlight scene 3 and the time period of audio highlight scene 4 overlap, and audio highlight scene 4 starts before video highlight scene 3. In this case, the time period of highlight video 3 is determined to be from the time when the conversation detected in audio highlight scene 4 at the start time of video highlight scene 3 starts to the end time of video highlight scene 3.

[0110] In S711, the information processing device 103 determines that an audio highlight scene is to be prioritized.

[0111] In S712, the information processing device 103 determines the period from the start time to the end time of the audio highlight scene as the time period of the highlight video. Then, in S713, the information processing device 103 notifies the electronic device 102 of information related to the time period. This notification allows the user to check in advance whether a highlight video for a desired time period can be generated. The user can also adjust the time period himself or herself, or change the period so that video highlight scenes have priority over audio highlight scenes.

[0112] While audio highlight scenes can be clearly set as breaks in conversation or musical interludes, it is difficult to clearly set break positions in video highlight scenes because the subject in the video moves continuously. Therefore, in this embodiment, when audio highlight scenes are prioritized, the start and end positions of the video highlight scenes are not set to match the audio highlight scenes, but a time period that takes only the audio highlight scenes into consideration is determined as the time period of the highlight video.

[0113] However, since there is a possibility that the highlight video desired by the user may not be generated, the user is notified in advance of information regarding the time period that takes into consideration only audio highlight scenes, and the user is also able to adjust the time period himself / herself.

[0114] Furthermore, when audio highlight scenes are given priority, the process of FIG. 6 may be performed in the same manner as when both video highlight scenes and audio highlight scenes are given priority.

[0115] In S714, the information processing device 103 receives from the electronic device 102 an instruction as to whether or not to generate a highlight video for the determined time period, and if the received instruction is to “not generate a highlight video” (No in S714), the information processing device 103 returns to the processing of S406.

[0116] If the received instruction is to "generate a highlight video" (Yes in S714), portions corresponding to the time period of the highlight video are extracted from each of the video data and audio data. A highlight video file including the extracted video data and audio data is generated (S715). The generated highlight video file is recorded in the HDD 209, and this process ends. The generated highlight video file may be transmitted to the electronic device 102.

[0117] In this embodiment, the method of notifying information relating to the time period is to display a schematic diagram on the electronic device 102, but this is not the only possible example, and other formats such as displaying in words are also possible. As an example of another form, it is possible to display a video in which the highlight videos recorded in this embodiment are joined together on the screen of the electronic device 102.

[0118] Also, the created highlight movie may be output to the electronic device 102 as a movie file.

[0119] Furthermore, editing can be applied that allows the transition between the video highlight scene and the audio highlight scene to be easily recognized by considering the degree of overlap between the video highlight scene and the audio highlight scene based on information on the time zone of the video highlight scene and the time zone of the audio highlight scene. Also, it is possible to notify the user of the editing. For example, when the video highlight scene starts in the middle of an audio highlight scene, the user is notified of a method of applying an audio transition that fades out the audio.

[0120] When the feature quantities of the joined video highlight scenes are significantly different, a video transition may be applied to the join.

[0121] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0122] The disclosure of this embodiment includes the following configurations and methods.

[0123] (Configuration 1) An information processing device capable of communicating with an electronic device, A first determination means for determining a video highlight scene based on a feature amount of video data included in the video file, and determining an audio highlight scene based on a feature amount of audio data included in the video file; an acquisition means for acquiring information on the time periods of the video highlight scenes and the audio highlight scenes determined by the first determination means; a receiving means for receiving priority information indicating whether to prioritize one or both of the video highlight scenes and the audio highlight scenes from the electronic device; a second determination means for determining a time period of the highlight video based on the priority information received by the receiving means and information on time periods of the video highlight scenes and the audio highlight scenes; 20. An information processing device comprising: a notification unit that notifies the electronic device of the time period of the highlight video determined by the second determination unit.

[0124] (Configuration 2) When the time period of the video highlight scene and the time period of the audio highlight scene do not overlap, and priority information giving priority to both the video highlight scene and the audio highlight scene is received, 2. The information processing device according to configuration 1, wherein the second determination means determines time periods of the video highlight scenes and the audio highlight scenes as time periods of the highlight video.

[0125] (Configuration 3) When the time period of the video highlight scene and the time period of the audio highlight scene do not overlap, and priority information that prioritizes either the video highlight scene or the audio highlight scene is received, 2. The information processing device according to configuration 1, wherein the second determination means determines a time period of the video highlight scene or the audio highlight scene, whichever has a higher priority, as a time period of the highlight video.

[0126] (Configuration 4) The time period of the video highlight scene and the time period of the audio highlight scene overlap, When priority information that prioritizes both the video highlight scene and the audio highlight scene is received, and the time period of the video highlight scene includes the time period of the audio highlight scene, 2. The information processing device according to configuration 1, wherein the second determination means determines a time period of the video highlight scene as a time period of the highlight video.

[0127] (Configuration 5) The time period of the video highlight scene and the time period of the audio highlight scene overlap, When priority information that prioritizes both the video highlight scene and the audio highlight scene is received, and the time period of the video highlight scene includes only the start time of the audio highlight scene, 2. The information processing device according to configuration 1, wherein the second determination means determines a time period from a start time of the video highlight scene to an end time of the audio highlight scene as a time period of the highlight video.

[0128] (Configuration 6) The time period of the video highlight scene and the time period of the audio highlight scene overlap, When priority information that prioritizes both the video highlight scene and the audio highlight scene is received, and the time period of the video highlight scene includes only the end time of the audio highlight scene, 2. The information processing device according to configuration 1, wherein the second determination means determines a time period from a start time of the audio highlight scene to an end time of the video highlight scene as a time period of the highlight video.

[0129] (Configuration 7) When a time period of the video highlight scene and a time period of the audio highlight scene overlap, priority information giving priority to the video highlight scene is received, and the time period of the video highlight scene includes the time period of the audio highlight scene, 2. The information processing device according to configuration 1, wherein the second determination means determines a time period of the video highlight scene as a time period of the highlight video.

[0130] (Configuration 8) A detection means for detecting a conversation from the voice data is further provided, When a time period of the video highlight scene and a time period of the audio highlight scene overlap, priority information giving priority to the video highlight scene is received, and the time period of the video highlight scene includes only a start time of the audio highlight scene, The video generating means determines, as a time period of the highlight video, a time period from a start time of the video highlight scene to a time period when the conversation detected in the audio highlight scene ends at the end time of the video highlight scene. 2. The information processing device according to configuration 1.

[0131] (Configuration 9) A detection means for detecting a conversation from the voice data is further provided, The time period of the video highlight scene and the time period of the audio highlight scene overlap, When priority information giving priority to the video highlight scene is received, and the time period of the video highlight scene includes only the end time of the audio highlight scene, The second determination means determines, as a time period of the highlight video, a time period from a time when a conversation detected at a start time of the video highlight scene among the audio highlight scenes starts to an end time of the video highlight scene. 2. The information processing device according to configuration 1.

[0132] (Configuration 10) When a time period of the video highlight scene and a time period of the audio highlight scene overlap, and priority information giving priority to the audio highlight scene is received, 2. The information processing device according to configuration 1, wherein the second determination means determines a time period of the audio highlight scene as a time period of the highlight video.

[0133] (Configuration 11) The time period of the video highlight scene and the time period of the audio highlight scene overlap, when priority information giving priority to the video highlight scene is received and a time period of the video highlight scene includes a time period of the audio highlight scene, the notification means does not notify the user, The time period of the video highlight scene and the time period of the audio highlight scene overlap, When priority information giving priority to the video highlight scene is received and the time period of the video highlight scene does not include the time period of the audio highlight scene, a notification is given by the notification means. 10. The information processing device according to any one of configurations 7 to 9.

[0134] (Configuration 12) a second receiving means for receiving an instruction as to whether or not to generate a highlight video from the electronic device; The present invention further includes a generating unit configured to, when an instruction to generate the highlight video is received, extract the video data and the audio data for a time period of the highlight video, and generate a highlight video file. 12. The information processing device according to any one of configurations 1 to 11.

[0135] (Configuration 13) 13. The information processing device according to any one of configurations 1 to 12, further comprising a second notification means for notifying the electronic device of information relating to the time periods of the video highlight scenes and the audio highlight scenes acquired by the acquisition means.

[0136] (Configuration 14) A method for controlling an information processing device capable of communicating with an electronic device, comprising: A first determination step of determining a video highlight scene based on a feature amount of video data included in the video file, and determining an audio highlight scene based on a feature amount of audio data included in the video file; an acquisition step of acquiring information regarding the time periods of the video highlight scenes and the audio highlight scenes determined by the first determination step; a receiving step of receiving priority information indicating whether to prioritize one or both of the video highlight scenes and the audio highlight scenes from the electronic device; a second determination step of determining a time period of the highlight video based on the priority information received by the receiving means and information regarding the time periods of the video highlight scenes and the audio highlight scenes; The information processing device further comprises a notification step of notifying the electronic device of the time period of the highlight video determined by the second determination means.

[0137] (Configuration 15) 15. A program for causing a computer to execute the method for controlling an information processing device according to configuration 14.

[0138] (Configuration 16) A computer-readable recording medium having recorded thereon a program for causing a computer to execute the method for controlling an information processing device according to configuration 14.

[0139] (Configuration 17) An information processing system having an electronic device and an information processing device capable of communicating with the electronic device, The information processing device includes: A first determination means for determining a video highlight scene based on a feature amount of video data included in the video file, and determining an audio highlight scene based on a feature amount of audio data included in the video file; an acquisition means for acquiring information on the time periods of the video highlight scenes and the audio highlight scenes determined by the first determination means; a receiving means for receiving priority information indicating whether to prioritize one or both of the video highlight scenes and the audio highlight scenes from the electronic device; a second determination means for determining a time period of the highlight video based on the priority information received by the receiving means and information on time periods of the video highlight scenes and the audio highlight scenes; a notification means for notifying the electronic device of the time period of the highlight video determined by the second determination means; An information processing system comprising:

Claims

1. An information processing device capable of communicating with an electronic device, A first determination means for determining a video highlight scene based on a feature amount of video data included in the video file, and for determining an audio highlight scene based on a feature amount of audio data included in the video file; an acquisition means for acquiring information on the time periods of the video highlight scenes and the audio highlight scenes determined by the first determination means; a receiving means for receiving priority information indicating whether to prioritize one or both of the video highlight scenes and the audio highlight scenes from the electronic device; a second determination means for determining a time period of the highlight video based on the priority information received by the receiving means and information relating to a time period of each of the video highlight scenes and the audio highlight scenes; an information processing device further comprising a notification unit that notifies the electronic device of the time period of the highlight video determined by the second determination unit.

2. When the time period of the video highlight scene and the time period of the audio highlight scene do not overlap, and priority information giving priority to both the video highlight scene and the audio highlight scene is received, 2 . The information processing apparatus according to claim 1 , wherein the second determination means determines a time period during which the video highlight scenes and the audio highlight scenes are included as a time period for the highlight moving image.

3. When the time period of the video highlight scene and the time period of the audio highlight scene do not overlap, and priority information giving priority to either the video highlight scene or the audio highlight scene is received, 2 . The information processing apparatus according to claim 1 , wherein the second determination means determines a time period of the video highlight scene or the audio highlight scene, whichever has a higher priority, as the time period of the highlight moving image.

4. When a time period of the video highlight scene and a time period of the audio highlight scene overlap, priority information that prioritizes both the video highlight scene and the audio highlight scene is received, and the time period of the video highlight scene includes the time period of the audio highlight scene, 2 . The information processing apparatus according to claim 1 , wherein the second determination means determines a time period of the video highlight scene as a time period of the highlight moving image.

5. When a time period of the video highlight scene and a time period of the audio highlight scene overlap, priority information that prioritizes both the video highlight scene and the audio highlight scene is received, and the time period of the video highlight scene includes only a start time of the audio highlight scene, 2 . The information processing apparatus according to claim 1 , wherein the second determination means determines a time period from a start time of the video highlight scene to an end time of the audio highlight scene as the time period of the highlight moving image.

6. When a time period of the video highlight scene and a time period of the audio highlight scene overlap, priority information that prioritizes both the video highlight scene and the audio highlight scene is received, and the time period of the video highlight scene includes only the end time of the audio highlight scene, 2 . The information processing apparatus according to claim 1 , wherein the second determination means determines a time period from a start time of the audio highlight scene to an end time of the video highlight scene as the time period of the highlight moving image.

7. When a time period of the video highlight scene and a time period of the audio highlight scene overlap, priority information giving priority to the video highlight scene is received, and the time period of the video highlight scene includes the time period of the audio highlight scene, 2 . The information processing apparatus according to claim 1 , wherein the second determination means determines a time period of the video highlight scene as a time period of the highlight moving image.

8. A detection means for detecting a conversation from the voice data is further provided, When a time period of the video highlight scene and a time period of the audio highlight scene overlap, priority information giving priority to the video highlight scene is received, and the time period of the video highlight scene includes only a start time of the audio highlight scene, The information processing device according to claim 1, characterized in that the video generation means determines, as a time period of the highlight video, a time period from a start time of the video highlight scene to a time period at which a conversation detected in the audio highlight scene ends at an end time of the video highlight scene.

9. A detection means for detecting a conversation from the voice data is further provided, When a time period of the video highlight scene and a time period of the audio highlight scene overlap, priority information giving priority to the video highlight scene is received, and the time period of the video highlight scene includes only the end time of the audio highlight scene, 2. The information processing device according to claim 1, wherein the second determination means determines, as the time period of the highlight video, a time period from a time when a conversation detected at the start time of the video highlight scene among the audio highlight scenes starts to an end time of the video highlight scene.

10. When a time period of the video highlight scene and a time period of the audio highlight scene overlap, and priority information giving priority to the audio highlight scene is received, 2 . The information processing apparatus according to claim 1 , wherein the second determination means determines a time period of the audio highlight scene as a time period of the highlight moving image.

11. when a time period of the video highlight scene and a time period of the audio highlight scene overlap, priority information giving priority to the video highlight scene is received, and the time period of the video highlight scene includes the time period of the audio highlight scene, notifying the user by the notification means; An information processing device as described in any one of claims 7 to 9, characterized in that when the time period of the video highlight scene and the time period of the audio highlight scene overlap, priority information that prioritizes the video highlight scene is received, and the time period of the video highlight scene does not include the time period of the audio highlight scene, a notification is made by the notification means.

12. a second receiving means for receiving, from the electronic device, an instruction as to whether or not to generate a highlight video; 2. The information processing device according to claim 1, further comprising a generation unit that, when an instruction to generate the highlight movie is received, extracts the video data and the audio data for a time period of the highlight movie, and generates a highlight movie file.

13. 2. The information processing apparatus according to claim 1, further comprising a second notification means for notifying the electronic device of information relating to the time periods of the video highlight scenes and the audio highlight scenes acquired by the acquisition means.

14. A method for controlling an information processing device capable of communicating with an electronic device, comprising: a first determination step of determining a video highlight scene based on a feature amount of video data included in the video file, and determining an audio highlight scene based on a feature amount of audio data included in the video file; an acquisition step of acquiring information regarding time periods of the video highlight scenes and the audio highlight scenes determined by the first determination step; a receiving step of receiving priority information indicating whether to prioritize one or both of the video highlight scenes and the audio highlight scenes from the electronic device; a second determination step of determining a time period of the highlight video based on the priority information received by the receiving means and information regarding time periods of the video highlight scenes and the audio highlight scenes; The information processing device further comprises a notification step of notifying the electronic device of the time period of the highlight movie determined by the second determination means.

15. A program for causing a computer to execute the method for controlling an information processing device according to claim 14.

16. 15. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the method for controlling an information processing device according to claim 14.

17. An information processing system having an electronic device and an information processing device capable of communicating with the electronic device, The information processing device includes: a first determination means for determining a video highlight scene based on a feature amount of video data included in the video file, and for determining an audio highlight scene based on a feature amount of audio data included in the video file; an acquisition means for acquiring information on the time periods of the video highlight scenes and the audio highlight scenes determined by the first determination means; a receiving means for receiving priority information indicating whether to prioritize one or both of the video highlight scenes and the audio highlight scenes from the electronic device; a second determination means for determining a time period of the highlight video based on the priority information received by the receiving means and information relating to a time period of each of the video highlight scenes and the audio highlight scenes; a notification means for notifying the electronic device of the time period of the highlight video determined by the second determination means; An information processing system comprising:

Images