Electronic devices, video management methods, and video editing equipment

Abstract

This invention provides an electronic device, a video management method and program, and a video editing device that facilitate the editing of videos in scenarios containing multiple cuts, linked to audio source data. [Solution] In the imaging system, the information support terminal 200 that controls the digital camera comprises a display unit 240 that displays information, an operation unit 230 that inputs user operations, and a control unit 210 that controls the display unit according to the operations input in the input unit. The control unit displays a setting screen related to sound source data associated with a scenario on the display unit, and acquires sound setting information that associates cuts in multiple cuts with the sound indicated by the sound source data based on user operations in the input unit, and generates management data that manages the positional relationship in which video information indicating each video is arranged in chronological order in the sound of the sound source data on an editing screen for editing multiple videos associated with multiple cuts, based on the acquired sound setting information.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a video management method for managing videos in a scenario including a plurality of cuts, and a video editing device.

Background Art

[0002] Patent Document 1 discloses a synchronous playback device that displays a video of a dance being danced in time with music, etc., in synchronization with the reproduced music sound. The synchronous playback device generates playback synchronization information for making the synchronization relationship of the respective playback timings of the video shot in synchronization with such music sound and the reproduced music sound based on the music content corresponding to the music sound the same as the synchronization relationship between the music sound and the video at the time of shooting. The playback synchronization information is generated to include relative time information with respect to the reference time for each of the start command and end command of the acquisition operation of the video data, and the reception times from the input means of the start command and end command of the music sound playback or synchronous playback, with the start time of the playback of the reproduced music sound as the reference time.

[0003] Further, Patent Document 1 discloses identifying music content corresponding to the music sound data by analyzing the sound collection data during the period of acquiring video data from the shooting means and identifying the music sound and other than the music sound. In this case, the synchronous playback device generates playback synchronization information based on the identified music content instead of the relative time information described above.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] This disclosure provides an electronic device and a video management method, as well as a video editing device, that can facilitate the editing of videos in a scenario including multiple cuts in association with sound source data. [Means for solving the problem]

[0006] In this disclosure, the electronic device manages video in a scenario that includes multiple cuts. The electronic device comprises a display unit for displaying information, an input unit for receiving user operations, and a control unit for controlling the display unit in accordance with the operations entered in the input unit. The control unit displays a settings screen for sound source data associated with the scenario on the display unit, and obtains audio setting information that associates cuts in multiple cuts with the audio indicated by the sound source data based on user operations in the input unit. Based on the acquired audio setting information, the control unit generates management data that manages the positional relationship in which video information representing each video is arranged in chronological order in the audio of the sound source data on an editing screen for editing multiple videos associated with multiple cuts.

[0007] In this disclosure, the video management method is a method for managing videos in a scenario that includes multiple cuts. In this method, the control unit of an electronic device displays a setting screen for sound source data associated with the scenario on the display unit, and acquires sound setting information that associates cuts in multiple cuts with the audio indicated by the sound source data through user operation on the input unit. Based on the acquired sound setting information, the control unit generates management data that manages the positional relationship in which video information representing each video is arranged in chronological order in the audio of the sound source data on an editing screen for editing multiple videos associated with multiple cuts.

[0008] In this disclosure, the video editing device comprises a display unit that displays an editing screen for editing multiple videos, a data input unit for inputting data, an operation unit that accepts user operations, and a control unit that controls the display unit based on the data input from the data input unit. The control unit acquires from the data input unit video data indicating videos shot in association with each cut in a scenario including multiple cuts, sound source data associated with the scenario, and management data that associates the cuts in the multiple cuts with the audio indicated by the sound source data. The management data manages the positional relationship in which the video information indicating each video associated with the multiple cuts is arranged in chronological order in the audio of the sound source data on the editing screen. Based on the management data, the control unit displays on the display unit a timeline in which the video information of the cuts associated with the audio in the multiple cuts is arranged in chronological order in the audio source data, and accepts user operations on the editing screen to edit each video indicated by the video information via the operation unit. [Effects of the Invention]

[0009] The electronic devices, video management methods, and video editing equipment described herein make it easier to edit videos in scenarios that include multiple cuts in association with audio data. [Brief explanation of the drawing]

[0010] [Figure 1] A diagram illustrating the configuration of the imaging system in Embodiment 1 of this disclosure. [Figure 2] Diagram showing the configuration of a digital camera in an imaging system. [Figure 3] Diagram showing the configuration of the information support terminal in the imaging system. [Figure 4] This diagram shows an example of the display of the function selection screen on an information support terminal. [Figure 5] A flowchart illustrating the operation of the scenario creation function in an imaging system. [Figure 6] This diagram shows an example of the display of the scenario input screen on an information support terminal. [Figure 7]Figure showing a display example in setting the music range by the scenario creation function [Figure 8] Figure illustrating the data structure of cut division data in the information support terminal [Figure 9] Figure showing a display example of the cut selection screen in the information support terminal [Figure 10] Flowchart illustrating the operation of the cut shooting function in the imaging system [Figure 11] Figure showing a display example of the rating screen in the information support terminal [Figure 12] Flowchart illustrating the processing of the recording mode in the imaging system [Figure 13] Figure showing a display example in the recording mode of the information support terminal [Figure 14] Figure for explaining the synchronous playback function in the recording mode of the information support terminal [Figure 15] Figure illustrating the data structure of video metadata in the information support terminal [Figure 16] Figure showing a display example of the video editing screen on the video editing PC of the imaging system [Figure 17] Flowchart illustrating the operation of the export function in the imaging system [Figure 18] Figure illustrating the video timeline for lip sync on the video editing screen [Figure 19] Figure illustrating the video timeline for B-roll on the video editing screen [Figure 20] Flowchart illustrating the timeline setting process in the imaging system [Figure 21] Figure illustrating the data structure of management data by the timeline setting process [Figure 22] Flowchart illustrating the operation of the video editing PC in the imaging system [Figure 23] Figure for explaining the operation of the video editing PC

Mode for Carrying Out the Invention

[0011] The embodiments will be described in detail below, with reference to the drawings as appropriate. However, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. The accompanying drawings and the following description are provided so that those skilled in the art can fully understand this disclosure, and are not intended to limit the subject matter described in the claims.

[0012] (Embodiment 1) Embodiment 1 of this disclosure describes a system that uses an electronic device separate from the imaging device that performs video recording.

[0013] 1. Structure The imaging system in Embodiment 1 of this disclosure will be described with reference to Figure 1.

[0014] As shown in Figure 1, for example, this system 10 comprises a digital camera 100, an information support terminal 200, and a video editing personal computer (PC) 300. In this system 10, the digital camera 100 and the information support terminal 200 are connected in a way that enables data communication, for example, by wired communication or wireless communication.

[0015] This system 10 can be applied, for example, to the production of desired video works such as music videos or dance videos by a user shooting and editing multiple videos with a digital camera 100. This system 10 provides useful information support for a series of workflows, such as a user creating a scenario outlining the concept of a video work associated with such a song, repeatedly shooting videos according to multiple cuts set in the scenario, and editing the multiple shot videos.

[0016] In this system 10, the information support terminal 200 can, for example, manage the scenario of a video production or control the digital camera 100 to manage video shooting for each cut. For example, the live view image from the digital camera 100 can be viewed on the information support terminal 200. The video data of the shooting results from the digital camera 100 is edited on the video editing PC 300. This system 10 uses the data managed on the information support terminal 200 to facilitate video editing on the video editing PC 300 and the like.

[0017] In this system 10, the video editing PC 300 may or may not communicate with one or both of the digital camera 100 and the information support terminal 200. For example, data from the digital camera 100 and / or the information support terminal 200 may be input to the video editing PC 300 via a portable recording medium such as a memory card. This system 10 does not necessarily have to include the video editing PC 300.

[0018] 1.1. Digital Camera Configuration The configuration of the digital camera 100 in this embodiment will be explained with reference to Figure 2.

[0019] Figure 2 is a diagram illustrating the configuration of the digital camera 100 in this system 10. The digital camera 100 is an example of an imaging device in this embodiment. The digital camera 100 in this embodiment includes an image sensor 115, an image processing engine 120, a display monitor 130, and a control unit 135. Furthermore, the digital camera 100 includes a buffer memory 125, a card slot 140, a flash memory 145, an operation unit 150, a communication module 155, a microphone 160, and a speaker 170. The digital camera 100 also includes, for example, an optical system 110 and a lens drive unit 112.

[0020] The optical system 110 includes a focus lens, a zoom lens, an optical image stabilization (OIS) lens, an aperture, a shutter, etc. The focus lens is a lens for changing the focus state of the subject image formed on the image sensor 115. The zoom lens is a lens for changing the magnification of the subject image formed by the optical system. The focus lens, etc., are each composed of one or more lenses.

[0021] The lens drive unit 112 drives the focus lens and other components in the optical system 110. The lens drive unit 112 includes a motor and moves the focus lens along the optical axis of the optical system 110 based on the control of the control unit 135. The configuration for driving the focus lens in the lens drive unit 112 can be implemented using a DC motor, stepping motor, servo motor, or ultrasonic motor, etc.

[0022] The image sensor 115 captures an image of a subject formed through the optical system 110 and generates imaging data. The imaging data constitutes image data representing the image captured by the image sensor 115. The image sensor 115 generates image data of a new frame at a predetermined frame rate (e.g., 30 frames / second). The timing of image data generation and the operation of the electronic shutter in the image sensor 115 are controlled by the control unit 135. The image sensor 115 can use various image sensors, such as a CMOS image sensor, a CCD image sensor, or an NMOS image sensor.

[0023] The image sensor 115 performs operations such as capturing still images and capturing through images. Through images are mainly moving images and are displayed on the display monitor 130 for the user to determine the composition for capturing still images. Through images and still images are examples of captured images in this embodiment. The image sensor 115 is an example of the imaging unit in this embodiment.

[0024] The image processing engine 120 performs various processes on the imaging data output from the image sensor 115 to generate image data, and also performs various processes on the image data to generate an image for display on the display monitor 130. Examples of various processes include, but are not limited to, white balance correction, gamma correction, YC conversion, electronic zoom, compression, and decompression. The image processing engine 120 may be composed of hardwired electronic circuits, or it may be composed of a microcomputer or processor using a program.

[0025] The display monitor 130 is an example of a display unit that displays various information. For example, the display monitor 130 displays an image (through image) represented by image data captured by the image sensor 115 and processed by the image processing engine 120. The display monitor 130 also displays a menu screen or the like for the user to make various settings for the digital camera 100. The display monitor 130 can be made of, for example, a liquid crystal display device or an organic EL device.

[0026] The operation unit 150 is a general term for hard keys such as operation buttons and operation levers provided on the exterior of the digital camera 100, and accepts operations from the user. The operation unit 150 includes, for example, a shutter release button, a mode dial, and a touch panel. When the operation unit 150 accepts an operation from the user, it transmits an operation signal corresponding to the user operation to the control unit 135.

[0027] The control unit 135 provides overall control over the operation of the digital camera 100. The control unit 135 includes a CPU, and the CPU executes programs (software) to realize predetermined functions. Instead of a CPU, the control unit 135 may include a processor consisting of dedicated electronic circuits designed to realize predetermined functions. In other words, the control unit 135 can be realized with various processors such as a CPU, MPU, GPU, DSP, FPGA, and ASIC. The control unit 135 may consist of one or more processors. Alternatively, the control unit 135 may be configured on a single semiconductor chip together with the image processing engine 120, etc.

[0028] The buffer memory 125 is a recording medium that functions as work memory for the image processing engine 120 and the control unit 135. The buffer memory 125 is implemented using DRAM (Dynamic Random Access Memory) or the like. The flash memory 145 is a non-volatile recording medium. Although not shown in the diagram, the control unit 135 may also have various internal memories, such as built-in ROM. The ROM stores various programs that the control unit 135 executes. The control unit 135 may also have built-in RAM that functions as a work area for the CPU.

[0029] The card slot 140 is a means for inserting a removable memory card 142. The card slot 140 can electrically and mechanically connect to the memory card 142. The memory card 142 is an external memory equipped with recording elements such as flash memory. The memory card 142 can store data such as image data generated by the image processing engine 120.

[0030] The communication module 155 is a module (circuit) that connects to an external device according to a predetermined communication standard in wired or wireless communication. The predetermined communication standard includes, for example, USB, HDMI®, IEEE 802.11, Wi-Fi, Bluetooth, etc. The digital camera 100 can communicate with other devices via the communication module 155.

[0031] The microphone 160 includes, for example, one or more microphone elements built into the digital camera 100. The microphone 160 outputs an audio signal indicating the picked-up sound to the control unit 135. In the digital camera 100, an external microphone may be used. The digital camera 100 may be provided with a connection part such as a terminal for connecting to an external microphone, in place of or in addition to the built-in microphone 160.

[0032] The speaker 170 includes, for example, one or more speaker elements built into the digital camera 100, and outputs sound to the outside of the digital camera 100 under control from the control unit 135. In the digital camera 100, an external speaker or earphone may be used. The digital camera 100 may also be provided with a connection part for connecting to an external speaker or the like, in place of or in addition to the built-in speaker 170.

[0033] 1.2. Configuration of the Information Support Terminal The configuration of the information support terminal 200 in this embodiment will be explained with reference to Figure 3.

[0034] Figure 3 is a diagram illustrating the configuration of an information support terminal 200. The information support terminal 200 is an example of an electronic device, such as a smartphone, tablet, or PC. The information support terminal 200 illustrated in Figure 3 comprises a control unit 210, a storage unit 220, an operation unit 230, a display unit 240, a communication unit 250, a microphone 260, and a speaker 270.

[0035] The control unit 210 includes, for example, a CPU or MPU that works in cooperation with software to realize predetermined functions. The control unit 210 controls, for example, the overall operation of the information support terminal 200. The control unit 210 reads data and programs stored in the storage unit 220, performs various calculations, and realizes various functions.

[0036] The control unit 210 executes a program that includes a set of instructions for realizing each of the above functions. This program may be provided via a communication network such as the Internet, or it may be stored on a portable recording medium. The control unit 210 may also be a dedicated electronic circuit or a hardware circuit such as a reconfigurable electronic circuit designed to realize each of the above functions. The control unit 210 may be composed of various semiconductor integrated circuits such as a CPU, MPU, GPU, GPGPU, TPU, microcontroller, DSP, FPGA, and ASIC.

[0037] The storage unit 220 is a storage medium that stores programs and data necessary to realize the functions of the information support terminal 200. As shown in Figure 3, the storage unit 220 includes a storage unit 221 and a temporary storage unit 222.

[0038] The storage unit 221 stores parameters, data, and control programs for realizing predetermined functions. The storage unit 221 is composed of, for example, an HDD or an SSD. For example, the storage unit 221 stores the above-mentioned programs, various image data, and sound source data indicating the audio of music played in a video work.

[0039] The temporary storage unit 222 is composed of RAM such as DRAM or SRAM, and temporarily stores (i.e., holds) data. For example, the temporary storage unit 222 holds image data that is being edited. The temporary storage unit 222 may also function as a work area for the control unit 210, or it may be composed of a storage area in the internal memory of the control unit 210.

[0040] The operation unit 230 is a general term for the operating components that the user operates. The operation unit 230 is, for example, a touch panel superimposed on the display unit 240 for inputting various touch operations, and is an example of an input unit of the information support terminal 200. The input unit may also be a connection software unit that communicates with various external input devices and receives operation signals. The operation unit 230 may be a physical button or switch provided on the information support terminal 200, or a keyboard, mouse, or touchpad may be used. The operation unit 230 may also be various GUIs such as virtual buttons, icons, cursors, software keyboards, and objects displayed on the display unit 240.

[0041] The display unit 240 is composed of, for example, a liquid crystal display or an organic EL display. The display unit 240 may display various types of information, such as various GUIs for operating the operation unit 230 and information input from the operation unit 230.

[0042] The communication unit 250 is a module (circuit) that connects to an external device according to a predetermined communication standard in wired or wireless communication. The predetermined communication standard includes, for example, USB, HDMI, IEEE802.11, Wi-Fi, Bluetooth, etc. The communication unit 250 may also connect the information support terminal 200 to a communication network such as the Internet. The communication unit 250 is an example of an acquisition unit that receives various information from an external device or a communication network. The communication unit 250 may also constitute a data input unit that inputs various data to the information support terminal 200.

[0043] The microphone 260 includes, for example, one or more microphone elements built into the information support terminal 200. The microphone 260 outputs an audio signal indicating the picked-up sound to the control unit 210. The information support terminal 200 may be provided with connection parts such as terminals for connecting to an external microphone, in place of or in addition to the built-in microphone 260.

[0044] The speaker 270 includes, for example, one or more speaker elements built into the digital camera 100, and outputs sound to the outside of the information support terminal 200 under control from the control unit 210. The information support terminal 200 may also be equipped with a connection part for connecting to an external speaker or earphones, etc., in place of or in addition to the built-in speaker 270.

[0045] The configuration of the information support terminal 200 described above is just one example, and the configuration of the information support terminal 200 is not limited to this. For example, the display unit 240 of the information support terminal 200 may use various display devices such as a projector and a head-mounted display. Also, for example, when using an external display device, the display unit 240 of the information support terminal 200 may be an output interface circuit for video signals, such as one compliant with the HDMI standard.

[0046] 1.3. Video Editing PC Configuration The video editing PC 300 in this system 10 is configured as a PC as an example of an information processing device. The video editing PC 300 is an example of a video editing device in this embodiment. The video editing PC 300 of this system 10 has a configuration similar to, for example, the information support terminal 200 shown in Figure 3. The video editing PC 300 does not need to have a built-in microphone. In addition, the video editing PC 300 may be equipped with a connection part for connecting to an external speaker, etc., and an output interface circuit for video signals, etc., instead of or in addition to the built-in speaker and display unit.

[0047] 2.Operation The operation of this system 10, which is configured as described above, will be explained below.

[0048] In this system 10, the information support terminal 200 has various functions for sequentially providing information support to the user in the video production workflow. Figure 4 shows an example of the display of the screen for selecting these various functions of the information support terminal 200.

[0049] The display unit 240 of the information support terminal 200 displays a scenario creation button 11, a shooting button 12, and an export button 13 on the function selection screen illustrated in Figure 4. Hereinafter, the length direction of the display unit 240 screen will be referred to as the X direction, and the width direction as the Y direction.

[0050] The scenario creation button 11 is a virtual button that accepts user input to execute a function (i.e., a scenario creation function) in this system 10 that provides information support for the process of the user creating a scenario before shooting video. The information support terminal 200 of this system 10 manages various information for each cut that manages video shooting in the scenario thus created. A cut constitutes, for example, a section in multiple video shoots for a scenario.

[0051] The shooting button 12 is a virtual button for executing a function (i.e., a cut shooting function) that supports video recording of each cut in a scenario created by, for example, the scenario creation function. The number of video shots for one cut is not limited to one take, but may be multiple takes. In this embodiment, the information support terminal 200 controls video recording by the digital camera 100 in the cut shooting function and manages the shooting results for each cut.

[0052] The export button 13 is a virtual button for executing a function (i.e., the export function) that applies pre-processing for external output to the video shooting management results obtained using the cut shooting function before outputting them. The pre-processing performed by the export function provides information support that makes it easier to perform the process of editing multiple shooting results into videos according to a scenario on a video editing PC 300, for example.

[0053] The information support terminal 200 of this system 10 can provide consistent information support from scenario creation to pre-processing for video editing by, for example, the user sequentially using the functions of the scenario creation button 11, the shooting button 12, and the export button 13.

[0054] For example, video works that play back video in sync with audio data such as music, such as dance videos or music videos, are produced by editing together multiple video clips that are then shot as source material. In such video editing, the process of synchronizing multiple video clips to the time range indicated in the audio data is complicated, and conventional automation technologies have problems with low accuracy and excessive editing load. Therefore, this system 10 utilizes information obtained from user operations in the scenario creation function to manage the time range in the audio data for each cut in the scenario, in accordance with the user's intentions.

[0055] This system 10 can output management data, which manages the time range of the sound source for each cut as described above, to the video editing PC 300 using the export function of the information support terminal 200. In this way, the video editing PC 300 can easily achieve a state where the video of the shooting result for each cut is synchronized with the sound source data according to the user's intentions when the output management data is loaded into editing software, etc. With this system 10, in video editing where video material is matched to sound source data, for example, the arrangement of video material according to the user's intentions is prepared from the initial stages of the editing work, making it easier for the user to perform the desired editing and reducing the editing burden.

[0056] 2.1. About the Scenario Creation Function The scenario creation function of the information support terminal 200 of this system 10 will be explained using Figures 5 to 8.

[0057] Figure 5 is a flowchart illustrating the operation of the scenario creation function in the imaging system 10. Each process shown in this flowchart is executed, for example, by the control unit 210 of the information support terminal 200. The processes in this flowchart are started, for example, in response to user operation of the scenario creation button 11 on the function selection screen in Figure 4.

[0058] When the control unit 210 receives user input from the operation unit 230, such as tapping the scenario creation button 11 on the function selection screen in Figure 4, it displays the scenario input screen shown in Figure 6 on the display unit 240 (S10). Figure 6 shows an example of the display of the scenario input screen on the information support terminal 200.

[0059] The scenario input screen is a screen used by the user to input a scenario into the information support terminal 200 in the scenario creation function of this system 10. The scenario input screen includes, for example, a storyboard input field 20 for each cut, a cut editing button 14, a back button 15, a music setting button 16, and a scene setting button 17, as shown in Figure 6. The control unit 210 of the information support terminal 200 receives various user operations related to the scenario input screen displayed on the display unit 240 via the operation unit 230 (S11~S15).

[0060] In the information support terminal 200, the storyboard input field 20 accepts user input of information indicating the storyboard, such as an outline of the scenario concept, for each cut that makes up the scenario. The storyboard input field 20 for each cut includes, for example, a composition field 21, a script field 22, a shooting time field 23, a shooting location field 24, and a memo field 25, as shown in Figure 6. The storyboard input field 20 further includes a lip-sync icon 27 and a B-roll icon 28 related to the synchronization of the video and music audio for each cut.

[0061] The composition field 21 accepts input of image information indicating the composition of the video shot for that cut. The input of image information may be by drawing with user operation or by specifying image data. The script field 22 accepts text input such as the script divided into parts for that cut in the scenario.

[0062] The shooting time field 23 accepts a numerical input indicating the approximate duration of time for shooting the video of the relevant shot. The shooting location field 24 accepts input of information indicating the location where the video of the relevant shot will be shot. The shooting location can be entered as text, or data search or other methods may be used as appropriate. The memo field 25 accepts input of various user-desired information regarding the video shooting of the relevant shot, such as shooting equipment and / or zoom of the subject, for example, as text input.

[0063] The lip-sync icon 27 and the B-roll icon 28 each indicate the type of cut, and accept user input to select one of them for each cut. The lip-sync icon 27 indicates a lip-sync cut, for example, in a video of performance or dance set to music, where the audio of the music is synchronized with the video. The B-roll icon 28 indicates a B-roll cut, for example, in a video of narration or scenery, where the audio of the music is not synchronized with the video. For cuts in which the lip-sync icon 27 is selected, the information support terminal 200 further displays a music editing field 26 in the storyboard input field 20. The music editing field 26 accepts user input indicating the range of music to be associated with the video for synchronization in the lip-sync cut.

[0064] In the example shown in Figure 6, the display unit 240 displays storyboard input fields 20 for two cuts. The control unit 210 acquires storyboard information for each cut according to user input to various fields 21-25 in the storyboard input fields 20 for each cut in the scenario. The control unit 210 also acquires storyboard information through user input to the music editing field 26, in response to user operations selecting icons 27 and 28. On the scenario input screen of the information support terminal 200, the storyboard input fields 20 for each cut displayed on the display unit 240 can be changed, for example, by swiping in the X direction to scroll, where the storyboard input fields 20 for each cut are arranged.

[0065] The cut editing button 14 toggles on / off the state in which various user operations such as adding, deleting, and rearranging cuts to be included in the scenario can be input. For example, when the cut editing button 14 is on, a touch operation allows the user to place the desired number of storyboard input fields 20 in chronological order in the scenario.

[0066] The back button 15 accepts a user operation on the information support terminal 200 that returns one screen to the previous screen. For example, the control unit 210 transitions the display unit 240 to the function selection screen (Figure 4) in response to a user operation of the back button 15 on the scenario input screen (Figure 6).

[0067] The music setting button 16 is a virtual button used to set the music to be played in the video work according to the scenario. The control unit 210 of the information support terminal 200 executes various functions related to music setting in response to the user operation of the music setting button 16 (S11).

[0068] For example, the control unit 210 accepts a user operation to select a song via the operation unit 230, and displays the song title and other information as options on the display unit 240 based on data stored in the storage unit 221, etc. (S11). After the song is selected, the control unit 210 accepts user input of information indicating the range to be included in the video work within the period during which the audio can be played back, based on the sound source data of the song, using a display similar to the setting of the song range for lip-sync cuts described later (see Figure 7) (S11).

[0069] The Scene Setting Button 17 is a virtual button used to set the shooting scene for one or more shots in a scenario. The Scene Setting Button 17 accepts information indicating the shooting scene, such as the structure of the music or the shooting location, as text input.

[0070] In the operation of the scenario creation function illustrated in Figure 5, the control unit 210, for example, after the execution of the function to set music for the scenario (S11), sets each cut according to the user input into the storyboard input field 20 for each cut on the scenario input screen (Figure 6) (S12~S15).

[0071] For example, the control unit 210 receives user input via the operation unit 230 to select the type of cut using a lip-sync icon 27 or a B-roll icon 28 in the storyboard input field 20 for each cut (S12).

[0072] When a user input is received to select the lip-sync icon 27 (S12(i)), the control unit 210 sets the range of music associated with the cut in which the lip-sync icon 27 is selected (S13). The control unit 210 displays the music editing field 26 in the storyboard input field 20 of the cut, for example, and displays a screen for setting this music range on the display unit 240 in response to a user operation such as tapping the music editing field 26. Figure 7 shows an example of the display when setting the music range using the scenario creation function.

[0073] For example, the screen shown in Figure 7 includes a song icon 2 indicating a file containing the audio data of a song, a song title 91 indicating the title of the song or the file name of the corresponding audio data, a playback field 92, a play button 95, and a back button 15. The song title 91 displays, for example, the title of the song selected in step S11. The play button 95 accepts user input to start or stop audio output from the speaker 270, etc.

[0074] The playback field 92 accepts user input to specify a range in the audio of the music. The playback field 92 includes, for example, a playback head 93, an audio timeline 94, markers 96a and 96b, a duration indicator 97, and a marker time 98, as shown in Figure 7.

[0075] The playback head 93 is positioned, for example, on the audio timeline 94 and indicates a position corresponding to the time being played in the audio data of the song indicated by the song title 91. For example, the audio timeline 94 is updated sequentially in accordance with the playback of the song, displaying the played-up period in the -Y direction from the position indicated by the playback head 93, and the period before playback in the +Y direction from that position. The audio timeline 94 may also display predetermined periods before and after the position of the playback head 93 in the audio data. For example, the predetermined period displayed on the audio timeline 94 may be changed by a user operation such as pinching in or pinching out on the audio timeline 94.

[0076] Furthermore, the playback head 93 may be fixed to the center in the Y direction on the audio timeline 94, and the playback head's timing may be changed by, for example, a user operation that scrolls the audio timeline 94 in the ±Y direction.

[0077] Markers 96a and 96b indicate the start and end points of a specified range in the audio of the song, respectively, and move in the Y direction on the audio timeline 94 in response to drag operations on each marker 96. The duration display 97 displays the duration in the audio of the song. For example, the duration display 97 shown in Figure 7 compares the duration from the start point to the end point specified by the marker 96 with the duration of the song used throughout the video work in the scenario, as set in step S11.

[0078] Marker time 98 indicates the start or end time specified by marker 96a or 96b during operation, for example, based on the start time of the entire period in the period display 97. The control unit 210 acquires information indicating the time range specified by each of these markers 96, and based on this information, sets the range of the song to be associated with the cut selected as lip sync from the time range of the song (S13). For lip sync cuts, the entire period set in the scenario may be associated as the default range before the range of the song is specified by marker 96.

[0079] The control unit 210 sets the lip-sync cut in association with the range of the music set in step S13 (S14). The control unit 210 sets the cut based on storyboard information obtained according to user input to various fields 21 to 25 in the storyboard input field 20, in addition to the type of cut and the range of the music. In the scenario input screen illustrated in Figure 6, the first of the two displayed cuts is set as the lip-sync cut. In the music editing field 26, the control unit 210 displays a numerical value indicating the range of the music set for the cut, along with a waveform corresponding to the sound source data of the music.

[0080] On the other hand, when a user input is received in the storyboard input field 20 for each cut to select the B-roll icon 28 (S12(ii)), the control unit 210 sets the B-roll cut according to the user input in the various fields 21 to 25 in addition to the cut type (S15). In the example in Figure 6, the second cut is set as the B-roll cut.

[0081] The control unit 210 determines whether the cut settings have been completed, for example, based on whether or not the user has pressed the back button 15 on the scenario input screen (S16).

[0082] When the cut settings are complete (YES in S16), the control unit 210 generates cut breakdown data, including storyboard information for each cut, as output of the scenario creation function and stores it in the storage unit 220 (S17). Figure 8 shows an example of such cut breakdown data at the end of the scenario creation function.

[0083] The cut breakdown data D1 manages the following for each "cut number," as shown in Figure 8, for example: "scene," "script," "composition," "shooting time," "shooting location," "notes," "shooting completion flag," "video metadata list," "cut type," and "music range." The cut breakdown data D1 is an example of the management information in this embodiment.

[0084] The control unit 210 of the information support terminal 200 assigns cut numbers, which indicate cut identification information in the cut division data D1, in ascending order to the storyboard input fields 20 for each cut arranged on the scenario input screen, for example. If the order of the cuts is changed, the control unit 210 reassigns the cut numbers according to the changed order. Furthermore, for one or more cuts, the control unit 210 records the information entered via the scene setting button 17 on the scenario input screen in the "scene" of the corresponding cut in the cut division data D1.

[0085] For each cut, the control unit 210 records the information entered in the script field 22, composition field 21, shooting time field 23, shooting location field 24, and memo field 25 of the storyboard input field 20 into the "script," "composition," "shooting time," "shooting location," and "memo" fields of the cut division data D1, respectively.

[0086] In the cut breakdown data D1, the "shooting complete flag" manages whether the shot in question has been shot or not by setting it to ON / OFF. At the end of the scenario creation function, the shooting complete flag is set to OFF for all shots as the initial setting.

[0087] The "Video Metadata List" is a list for storing metadata of the video footage associated with the relevant shot. At the end of the scenario creation process, the Video Metadata List is initially set to an empty value.

[0088] The "Cut Type" manages the type of cut set for that cut. The cut type is selected from lip sync and B-roll according to the user operation in step S12, for example, and the selected type is set for that cut in steps S14 and S15.

[0089] The "music range" manages the time range of the music set for a cut with the cut type being lip-sync in steps S13 and S14. In this example, for cuts with the cut type being B-roll, the music range is set to an empty value. The cut division data D1 may include information indicating the range of music in the video work input in step S11, or this information may be managed separately from the cut division data D1 in the storage unit 220.

[0090] After generating the cut division data D1 (S17), the control unit 210 terminates the processing of this flowchart.

[0091] As described above, the scenario creation function of the information support terminal 200 of this system 10 generates cut division data D1 from user input on the scenario input screen (S11~S17). This enables information support for the process of creating a scenario for a desired video work cut by cut. For example, lip-sync or B-roll cuts can be set depending on whether or not to synchronize the audio and video of the music set in the scenario (S11~S15). Furthermore, lip-sync cuts can be set by associating them with a range of music desired by the user (S13, S14). For example, a common range may be associated with multiple lip-sync cuts.

[0092] The scenario creation function of the information support terminal 200 is not limited to the above. For example, the information support terminal 200 may receive a user instruction to output the storyboard information of the scenario entered on the scenario input screen as data using a data format that can be shared on another device (e.g., PDF format), and perform said data output.

[0093] 2.2. About the Cut Shot Function The operation of the cut shooting function in the information support terminal 200 of this system 10 will be explained with reference to Figure 9.

[0094] Figure 9 shows an example of the cut selection screen displayed on the information support terminal 200. The cut selection screen is a screen used in the cut shooting function of this system 10 to select a cut desired by the user from among the cuts provided, for example, by the scenario creation function. The cut selection screen is an example of a selection screen in the information support terminal 200 of this embodiment.

[0095] The cut selection screen, as shown in Figure 9 for example, includes a cut list 30, a storyboard display area 31, a filter button 32, a recording mode button 34, a playback mode button 35, and a back button 15. The cut list 30 is a list that enumerates various cuts as options available to the user. The storyboard display area 31 is a display area that shows storyboard information for the selected cut. Details of the cut selection screen will be described later.

[0096] In the cut shooting function of this system 10, the information support terminal 200 provides information support that makes it easier for the user to comprehensively proceed with video shooting for each cut while checking various cuts, for example, using the cut selection screen shown in Figure 9. Users may shoot videos in an order different from the order of cuts in the scenario, or shoot multiple takes of a single video cut.

[0097] Therefore, the information support terminal 200 of this system 10 receives user evaluations of the video for each take of the selected cut at the time of shooting, manages whether or not the shooting of the cut is complete, and visualizes the progress of video shooting for each cut to the user in the cut list 30. The operation of this system 10 will be explained in detail below.

[0098] 2.2.1. Overall operation of the cut shooting function The overall operation of the cut shooting function in this system 10 will be explained using Figures 9 to 11.

[0099] Figure 10 is a flowchart illustrating the operation of the cut shooting function in this system 10. Each process shown in the flowchart of Figure 10 is executed, for example, by the control unit 210 of the information support terminal 200. The process in this flowchart starts, for example, when the shooting button 12 on the function selection screen (Figure 4) is operated, after the cut division data D1 created by the scenario creation function has been stored in the storage unit 220 and a communication connection with the digital camera 100 has been established in the communication unit 250.

[0100] First, the control unit 210 of the information support terminal 200 generates a cut list 30 for display on the cut selection screen (Figure 9) based on the cut division data D1 (S21). The cut list generation process (S21) is repeatedly executed in this system 10, for example, during the execution of the cut shooting function, in accordance with the progress of video shooting and various user operations, and the cut list 30 is updated sequentially.

[0101] Next, the control unit 210 displays a cut selection screen on the display unit 240 based on the generated cut list 30 and cut division data D1, for example, as shown in Figure 9 (S22).

[0102] The cut list 30 on the cut selection screen includes multiple cut icons 3, as shown in Figure 9, for example. Each cut icon 3 represents an individual cut as a selection. The selected cut icon 3 is initially set to cut number "1", for example.

[0103] For example, the control unit 210 controls the display unit 240 to highlight the cut icon 3 that indicates the selected cut (S22). The highlighting of the selected cut icon 3 is, for example, done by making it larger than other cut icons 3 and by adding a border of a highlight color. The control unit 210 also refers to the cut division data D1 and displays storyboard information for the cut indicated by the selected cut icon 3 in the storyboard display area 31 (S22).

[0104] In the example in Figure 9, cut number "1" is in an incomplete state, while cuts "2" and "3" are in a completed state. In the cut list 30 of this embodiment, the cut icon 3 has display attributes that distinguish between a completed state and an incomplete state. These display attributes are set, for example, so that the display mode for the completed state is highlighted over the display mode for the incomplete state.

[0105] The control unit 210 accepts various user operations via the operation unit 230, such as a touch panel, when the display unit 240 is displaying the cut selection screen, as shown in Figure 9 (S23). The user operations targeted in step S23 include (I) cut selection operation, (II) transition to recording mode operation, (III) transition to playback mode operation, (IV) filtering operation, and (V) termination operation.

[0106] A cut selection operation (S23(I)) is a user operation that changes the currently selected cut, for example, by tapping a cut icon 3 other than the currently selected cut icon 3 in the cut list 30 displayed on the cut selection screen. Cut selection operations are not limited to this; for example, a swipe operation in the storyboard display area 31 may be input as a cut selection operation that changes from the currently selected cut to an adjacent cut.

[0107] When a cut selection operation is input (S23(I)), the control unit 210 changes the selected cut icon 3 according to the input cut selection operation (S24) and repeats the processing from step S2 onwards. As a result, the selected cut icon 3 on the cut selection screen is changed, and the storyboard display area 31 is displayed for the newly selected cut (S22).

[0108] The operation to switch to recording mode (S23(II)) is a user operation to switch to recording mode, which is an operation mode for recording video for the selected cut, and is, for example, a tap operation of the recording mode button 34. In addition to or instead of this, the transition operation may be a swipe operation in one of the ±X directions on the cut selection screen. The recording mode button 34 may also be omitted.

[0109] When an operation to switch to recording mode is input (S23(II)), the control unit 210 performs various processes for recording a single take of video in association with the selected cut as the recording mode (S25). An example of the display in step S25 is shown in Figure 11.

[0110] Figure 11 shows an example of the rating screen display on the information support terminal 200. The rating screen is a screen that prompts the user to rate the video of the captured take. The rating screen is an example of the evaluation screen on the information support terminal 200 in this embodiment.

[0111] The rating screen, as shown in Figure 11 for example, includes an information display area 40 for the recorded video, and OK, KEEP, and NG buttons 41, 42, and 43 as rating options. The information display area 40 displays information about the recorded take, including, for example, a thumbnail image of the video for that take, the cut number associated with that take, and the number of takes.

[0112] The OK button 41 indicates an evaluation "OK" where the user has decided to use the take in the corresponding cut. The KEEP button 42 indicates an evaluation "KEEP" where the user is unsure whether to use the take or not. The NG button 43 indicates an evaluation "NG (No Good)" where the user has determined that it is clear that the take will not be used. In this embodiment, the evaluation "NG" is an example of a first evaluation, and the evaluations "OK" and "KEEP" are examples of second evaluations, respectively.

[0113] In this embodiment, the recording mode processing (S25) involves, for example, displaying the rating screen shown in Figure 11 each time a video take is recorded, and obtaining evaluation information indicating the user's evaluation of that take. Based on the rating results of this recording mode processing (S25), the control unit 210 performs the cut list generation process (S21) again, as shown in Figure 10, to update the cut list 30. Details of the process in step S25 will be described later.

[0114] The operation to switch to playback mode (S23(III)) is a user operation to switch to playback mode, which is an operation mode that displays the video recorded for the selected cut, for example, by operating the playback mode button 35. In addition to or instead of this, the operation to switch to playback mode may be a swipe operation in the direction opposite to the operation to switch to recording mode among the ±X directions of the cut selection screen. Also, the playback mode button 35 may be omitted.

[0115] When an operation to switch to playback mode is input (S23(III)), the control unit 210 executes a process to play back the video of various takes related to the cut selected as the playback mode (S26). In the playback mode processing (S26) in this embodiment, re-rating can be performed to change the evaluation of the video of each take. Based on the re-rating results of this playback mode processing (S26), the control unit 210 performs the cut list generation process (S21) again to update the cut list 30.

[0116] The filtering operation (S23(IV)) is a user operation to narrow down the cuts to be displayed in the cut list 30, for example, by operating the filter button 32. When a filtering operation is input (S23(IV)), the control unit 210 obtains the conditions for filtering the cuts to be displayed according to the user's selection (S27).

[0117] In this embodiment, the information support terminal 200 uses the scene from the storyboard information of each cut as the filtering condition for the cut list 30. The control unit 210 performs the cut list generation process (S21) again based on the scene acquired as the filtering condition. In this way, the cut list 30 is updated to limit the cut icons 3 to those corresponding to the scene in the filtering condition.

[0118] The termination operation (S23(V)) is a user operation to terminate the cut shooting function, for example, by pressing the back button 15 on the cut selection screen (Figure 9). When the termination operation is input (S23(V)), the control unit 210 transitions the display unit 240 from, for example, the cut selection screen to the function selection screen (Figure 4), and terminates the process shown in this flowchart.

[0119] Through the above process, the user of this system 10 can check various cuts on the cut selection screen (Figure 9) of the cut shooting function of the information support terminal 200 (S24), shoot video of the desired cut (S25), or play it back (S26). In this way, the user can easily manage the shooting of multiple cuts in a scenario.

[0120] In the cut selection screen of this embodiment, each cut icon 3 is identified and displayed as either completed or not, thus preventing users from forgetting to shoot certain cuts. The identification of whether each cut is completed or not is performed in accordance with the user's evaluation of the video for each take, making it easier to ensure video quality that matches the user's intentions. Such rating is performed for each take (S25), and it is also possible to re-rate in playback mode (S26). This makes it easier to achieve quality control of video shooting that matches the user's intentions.

[0121] Furthermore, the cut selection screen of this embodiment (Figure 9) is not limited to updating the cut list 30 according to the cut rating / re-rating as described above (S25, S26, S21), but can also be updated according to the filtering of the display target (S27, S21). This allows the user to efficiently utilize the desired cuts on the cut selection screen, for example, in a video shooting environment, making it easier to use the cut shooting function of this system 10.

[0122] In the cut shooting function of this embodiment, the communication connection with the digital camera 100 may be managed, and for example, a button for managing the communication connection may be provided on the cut selection screen. The control unit 210 may disable the operation to switch to recording mode (S23(II)) when a communication connection with the digital camera 100 has not been established.

[0123] 2.2.2. About Recording Modes The details of the recording mode processing in step S25 of Figure 10 will be explained using Figures 12 to 15.

[0124] Figure 12 is a flowchart illustrating the recording mode processing (S25) in this system 10. The processing shown in the flowchart of Figure 13 is initiated, for example, when the operation to switch to recording mode is input on the cut selection screen (Figure 9) (S23 (II)).

[0125] First, the control unit 210 of the information support terminal 200 switches to recording mode and transitions the display unit 240 to a screen for waiting to record video (S41). An example of the display of the information support terminal 200 in step S41 is shown in Figure 13(A).

[0126] The shooting standby screen in step S41 includes, for example, a timer button 44, a live view image 45, a record button 46, and a synchronized playback button 48, as shown in Figure 13(A). The timer button 44 accepts user input to set a timer period for timing synchronization at the start of shooting. The record button 46 accepts user input to start recording video. The synchronized playback button 48 accepts user input to switch on or off the synchronized playback function, which plays the audio of a song in accordance with the timer period at the start of shooting.

[0127] Furthermore, the information support terminal 200 may manage the communication connection with an external speaker via the communication unit 250, and for example, information corresponding to the communication connection status may be displayed on the shooting standby screen.

[0128] In this system 10, when the information support terminal 200 switches to recording mode, for example, the control unit 210 requests the digital camera 100 to transmit the live view image 45 via the communication unit 250 (S41). The control unit 210, for example, in recording mode, sequentially receives image data of the live view image 45 from the digital camera 100 via the communication unit 250 and displays the live view image 45. Furthermore, the control unit 210 receives audio data from the digital camera 100 as needed, for example, the sound pickup result from the microphone 160.

[0129] The control unit 210 determines whether the cut type is lip-sync or not based on the cut division data D1 for the cut that is selected on the cut selection screen (Figure 9) and is the target of the recording mode (S42).

[0130] If the cut type is lip-sync (YES in S42), the control unit 210 determines whether the synchronized playback function for music during video recording is on, i.e., enabled, based on the result of a user operation, for example, on the synchronized playback button 48 (S43).

[0131] If the synchronized playback function is turned on (YES in S43), the timer period is set (S44) according to user operation, for example, the timer button 44. An example of the display on the information support terminal 200 in step S44 is shown in Figure 13(B). The timer period may be set for each take or uniformly for each cut.

[0132] Figure 13B illustrates the timer selection field 44a that appears in response to the operation of the timer button 44 on the shooting standby screen of Figure 13A. Figure 13B shows an example where the timer period is set to "5 seconds" in the timer selection field 44a. The timer selection field 44a includes options that indicate a numerical value for the timer period, such as "5 seconds". In step S44, when the control unit 210 receives a user operation on the timer selection field 44a at the operation unit 230, it sets the timer period according to the user operation and returns the display unit 240 from Figure 13(B) to Figure 13(A).

[0133] For example, after setting the timer period (S44), the control unit 210 performs various controls to start recording a video of one take associated with the selected cut and playing music, in response to the user operation of the recording button 46 (S45). For example, in step S45, the control unit 210 instructs the digital camera 100 to start recording video via the communication unit 250. Furthermore, the control unit 210 instructs the speaker 270 to output audio so that music is played based on the sound source data of the music stored in the storage unit 221, for example. An example of the display of the information support terminal 200 in step S45 is shown in Figures 13(C) and (D).

[0134] Figure 13(C) illustrates the shooting screen when the recording button 46 is operated when a timer period is set as in the example in Figure 13(B). Figure 13(D) illustrates the shooting screen after the timer period has elapsed, as shown in Figure 13(C). The control unit 210 counts down the elapsed time for the set timer period from the time the recording button 46 is operated (S45) and displays it, for example, superimposed on the live view image. In this example, video recording is performed including this timer period.

[0135] The shooting screen in step S45 includes, for example, a live view image 45, a time display field 47, and a recording stop button 46a, as shown in Figures 13(C) and (D). The live view image 45 on the shooting screen includes highlighting, such as a frame indicating that recording is in progress. For example, in the shooting screen of Figure 13(D), the recording stop button 46a accepts user input to stop recording video. The time display field 47 compares, for example, the shooting time of the selected cut in the cut division data D1 with the elapsed time from the start of video recording for that take.

[0136] In step S45, the control unit 210 controls the display unit 240 to switch the display from the shooting standby screen (Figure 13(A)) to the shooting screen (Figures 13(C), (D)). The control unit 210 also records video files showing the live view images 45 received sequentially from the digital camera 100 after the recording button 46 is operated in the storage unit 220 of the information support terminal 200 (S45). The recorded video files include, for example, audio data generated from audio signals picked up by the microphone 160 in synchronization with the video recording of the digital camera 100.

[0137] Furthermore, the control unit 210 determines the filename of the video file based, for example, on the cut division data D1 and the number of takes already shot for the selected cut. The control unit 210 may include the determined filename in the instructions given to the digital camera 100. The control unit 135 of the digital camera 100 starts shooting video according to instructions received, for example, from the information support terminal 200 via the communication module 155.

[0138] In this case, the control unit 135 repeats the imaging operation of the image sensor 115 and records the resulting video data onto a memory card 142, for example, via the card slot 140. The video data includes, for example, audio data from the sound captured by the microphone 160. The control unit 135 may also start sound capture synchronized with video recording on the digital camera 100 based on this shooting instruction.

[0139] In step S45, the information support terminal 200 displays a countdown as shown in Figure 13(C) and plays the music according to the set timer period. The control unit 210 plays the selected lip-sync cuts in sync with the video recording, based on the cut division data D1, within the range of the music set by the processing in steps S13 and S14 of the scenario creation function (Figure 5). Figure 14 is a diagram illustrating the synchronized playback function in the information support terminal 200.

[0140] Figure 14(A) shows an example where the start time ts of the set song range Tp is later than the start time of the song in the sound source data by the timer period. In the example in Figure 14(A), the control unit 210 starts playback of the song from a time ta that is the timer period before the song range Tp, based on the sound source data of the song, at the same time as the countdown display starts. For example, in step S45, the control unit 210 determines the playback start time ta for the previous playback of such a song based on the song range Tp and the timer period set in step S44.

[0141] Figure 14(B) shows an example where the start time ts of the music range Tp coincides with the beginning of the music. In the example in Figure 14(B), the control unit 210 determines the music playback start time ta to be the start time ts of the music range Tp, and during the timer period, it only performs countdown display and video recording. Figure 14(C) shows an example where the timer period is set to "15 seconds" and the start time ts of the music range Tp is set to 10 seconds after the beginning of the music. In the example in Figure 14(C), the control unit 210 determines the music playback start time ta to be 10 seconds before the start time ts of the music range Tp, and during the timer period, after starting the countdown display and video recording, it performs further playback of the music from the playback start time ta.

[0142] Thus, in step S45, the control unit 210 performs playback from the start time ta, which is the start time for playback of the music, according to the timer period and the start time ts of the music range Tp set for lip-sync cut. In this example, video recording and music playback are performed up to the recording stop time tb, regardless of whether the recording stop time tb, when video recording is stopped by operating the recording stop button 46a, is later than the end time te of the music range Tp from the playback start time ta.

[0143] If music synchronization playback is not enabled (NO in S43), the control unit 210 performs various controls in response to the user operation of the recording button 46, for example, in the same process as in step S45, to start recording the video associated with the selected cut (S46). In this case, the control unit 210 does not perform the countdown display as shown in Figure 13(C) or the music playback described in Figure 14, for example, and transitions the display unit 240 from the shooting screen as shown in Figure 13(A) to the shooting screen as shown in Figure 13(D).

[0144] Furthermore, even if the cut type of the selected cut is not lip-sync (NO in S42), the control unit 210 starts recording video associated with that cut in response to the user operation of the record button 46 (S46).

[0145] In this example, the control unit 210 does not determine whether to enable or disable the synchronized playback function (S43) or set the timer period (S44) for non-lip-sync, i.e., B-roll cuts (NO in S42). For example, if the information support terminal 200 switches to recording mode with B-roll cuts selected, it may hide the timer button 44 on the shooting standby screen in Figure 13(A) and not accept user input to set the timer period.

[0146] After the start of video recording, etc. (S45, S46), the control unit 210 performs various controls to stop the ongoing recording, etc. in response to the user operation of the recording stop button 46a (S47). For example, in step S47, the control unit 210 instructs the digital camera 100 via the communication unit 250 to stop video recording. The control unit 210 also stops the video recording of the live view image 45 in the information support terminal 200 (S47). The control unit 135 of the digital camera 100 terminates video recording in accordance with the instructions from the information support terminal 200. Furthermore, for example, in step S47 after the start of music playback (S45), the control unit 210 instructs the speaker 270 to stop audio output.

[0147] Furthermore, the control unit 210 prompts the user to rate the video of the take captured as described above, by displaying a rating screen on the display unit 240, for example, as shown in Figure 11 (S48).

[0148] The control unit 210 receives user input from various buttons 41-43 on a rating screen, such as in Figure 11, and obtains the user's evaluation as a result of rating the captured video take (S49). In this embodiment, each time a video take is captured, the user can arbitrarily select a desired evaluation from the three evaluations "OK", "KEEP", and "NG" for the captured video, without interfering with the evaluations of other video takes.

[0149] The control unit 210 determines, for example, whether the user's evaluation is "NG" or not based on the acquired evaluation (S50). For example, if the user's evaluation is "OK" or "KEEP", the determination in step S50 is "NO".

[0150] If the acquired user evaluation is not "NG" (NO in S50), the control unit 210 sets the shooting completion flag of the cut associated with the take (i.e., the selected cut) in the cut division data D1 to "ON" (S51). For example, if the number of takes for the video is "1", or if the evaluation of the video of an existing take is "NG" for a number of takes of "2" or more, the shooting completion flag is switched from "OFF" to "ON" by executing step S51.

[0151] On the other hand, if the acquired user evaluation is "NG" (YES in S50), the control unit 210 proceeds to step S52 without specifically updating the shooting completion flag setting. This ensures that, for example, if the shooting completion flag for a corresponding cut is OFF when a video with an evaluation of "NG" is shot, that OFF state is maintained. Also, if, for example, a previously shot video has a "KEEP" or "OK" rating and the shooting completion flag is ON, that ON state is maintained.

[0152] The control unit 210 generates metadata for the video of the captured takes as described above and records it, for example, in the cut data D1 in the storage unit 220 (S52). An example of this video metadata D2 is shown in Figure 15.

[0153] The video metadata D2 includes, for example, the "video file name," "evaluation information," "timer period," and "video period," as shown in Figure 15. The control unit 210 includes in the video metadata D2 the video file name determined to reflect the number of takes for the videos shot in steps S45 to S47, the user evaluation obtained in step S49, and the timer period set in step S44. The control unit 210 also includes the video shooting time from the start of video recording (S45, S46) to the end of video recording (S47) as the video period in the video metadata D2.

[0154] The control unit 210 stores the generated video metadata D2 in the video metadata list for the cut associated with the video in the cut division data D1 (Figure 8) (S52). The video metadata D2 is not limited to the above, and may include, for example, the video file name in addition to or instead of the number of takes of the video.

[0155] The control unit 210 terminates the recording mode processing (S25) by, for example, storing the video metadata D2 (S52), and proceeds to step S21 in Figure 10.

[0156] According to the above recording mode processing (S25), the system 10 can record a single take of video for the selected cut and prompt the user to rate it (S45-S49). Furthermore, if the cut is set to lip-sync (YES in S42), the system 10 can play back the music synchronized with the recording, according to the range of the music associated with the cut (S43, S45). In this case, depending on the timer setting (S44), before the start of video recording, in addition to displaying a countdown, the music is played back from a time earlier than the music range Tp of the cut (S45, see Figure 14), making it easier to shoot video according to the music.

[0157] Furthermore, the system 10 manages the shooting completion flag for the relevant shot based on the acquired evaluation information (S50, S51). In this way, the user's evaluation information for each take can be appropriately reflected in the management of whether or not the relevant shot is in a shooting completion state. According to the recording mode processing of this embodiment (S25), the information support terminal 200 can control the shooting recording of video by the digital camera 100, thereby realizing video shooting management.

[0158] In the video rating for each take (S48-S49), multiple takes associated with the same cut may have the same rating. For example, multiple takes of the same cut may have the rating "OK".

[0159] Furthermore, the rating screen displayed in step S48 may be displayed as a dialog box. For example, the control unit 210 may control the display unit 240 to superimpose the rating screen dialog box onto the display screen before and after step S47.

[0160] The recording mode shooting standby screen (Figure 13(A)) may further include a back button 15 for returning to the cut selection screen, for example. This back operation may also be a swipe operation in one of the ±X directions on the video management screen. The information support terminal 200 may also switch to playback mode by a swipe operation in the opposite direction.

[0161] The information support terminal 200 may also display the timer button 44 on the shooting standby screen (Figure 13(A)) even if the selected cut is set to B-roll (NO in S42), and may accept user input to set the timer period in the same way as in step S44. Furthermore, even if music synchronization playback is not turned on (NO in S43), the timer may be set in the same way as in step S44. In such cases, in step S46, a countdown display for the set timer period may be shown in conjunction with the start of video recording. The execution order of steps S43 and S44 is not limited to the above example; for example, after setting the timer, a decision similar to that in step S43 may be made regardless of whether music synchronization playback is enabled or disabled.

[0162] The information support terminal 200 does not need to acquire audio data from the digital camera 100 using the microphone 160 or the like in the video recording (S45) for lip-sync cuts. Furthermore, in the recording of lip-sync cuts, sound acquisition using the microphone 160 or the like is not required.

[0163] 2.3. About the Export Function The export function of the information support terminal 200 of this system 10 will be explained with reference to Figure 16.

[0164] Figure 16 shows an example of the video editing screen display on the video editing PC 300 of this system 10. The video editing PC 300 (Figure 1) reads, for example, the management data output from the export function of the information support terminal 200, the sound source data of the music, and the video data of the shooting result from the digital camera 100 into predetermined editing software and displays the video editing screen as shown in Figure 16. The editing software may be various non-linear editing (NLE) software, such as DaVinci Resolve, Adobe Premiere Pro, Final Cut Pro, or Vegas Pro.

[0165] The video editing screen illustrated in Figure 16 includes a material display area 61, a timeline editing area 62, a metadata display area 63, and a preview image 64. The video editing screen is a screen for the user to perform various video editing operations, and is an example of an editing screen with the video editing PC 300 as an example of an external device.

[0166] The material display area 61 displays a list of video data and music audio data (i.e., video materials) that have been loaded as video editing materials on the video editing PC 300. In this system 10, for example, the material display area 61 displays the folder that manages the video materials for each cut, i.e., the video folder 70 (details will be described later).

[0167] The timeline editing area 62 displays a video timeline 80 containing multiple video materials arranged on the time axis 81, and accepts user operations to edit a video work created by combining video materials on the video timeline 80. The video timeline 80 has tracks for each row in which the video materials are arranged along the time axis 81. In the video timeline 80, multiple tracks are arranged in a direction V that intersects the time axis 81, and different video materials can be placed at the same position on the time axis 81. Hereinafter, the +V side in this arrangement direction V may be referred to as the upper side, and the opposite -V side may be referred to as the lower side.

[0168] The timeline editing area 62 of this embodiment displays two video timelines 80 in different ways: one containing video material related to lip-sync cuts and music audio, and another containing only video material related to B-roll cuts. The timeline editing area 62 in this example includes a lip-sync tab 87 and a B-roll tab 88 for switching between and displaying these lip-sync video timelines 80 and B-roll video timelines 80, respectively. In the example in Figure 16, the lip-sync tab 87 is selected, and the lip-sync video timeline 80 is displayed.

[0169] The metadata display area 63 displays the metadata of the video material displayed in the material display area 61 or the timeline editing area 62. The preview image 64 displays an image of the video material at a timing corresponding to the position of the playhead 82 on the time axis 81 in the video timeline 80.

[0170] For example, on the video editing screen (Figure 16), the user can adjust the placement of video materials in the timeline editing area 62 while viewing the preview image 64, or place new video materials in the material display area 61 onto the video timeline 80. For example, on the video timeline 80 for lip-syncing, video materials for each cut can be placed in association with the audio of the music. Through these user operations, video editing operations are performed on the system 10.

[0171] In this embodiment, the information support terminal 200 generates management data in the export function that manages video materials according to the cut settings associated with the music range in the cut division data D1, from the viewpoint of making it easier for the user to perform editing operations on the video work as described above. For example, after executing a cut shooting function that enables synchronized playback of the music range set for a lip-sync cut, the information support terminal 200 generates management data that reflects the synchronization relationship between the cut and the music range. As a result, in this system 10, based on the management data, video materials for each cut read by the video editing PC 300 can be arranged on the video timeline 80 according to the synchronization relationship with the music range, making it easier to perform the video editing process according to the music. Furthermore, the processing efficiency of this process by the video editing PC 300 can also be improved.

[0172] A typical function of this type of editing software is, for example, to play the audio of a song along with the recording of a video, and if the recorded video data includes the sound recording results of the song, to place the video material on the video timeline 80 in synchronization with the audio of the song. In this function, as described in Patent Document 1, for example, synchronization of the video and the song audio in the video data is achieved by comparing the audio identified by analyzing the sound recording results in the video data with the song audio indicated by the sound source data of the song loaded into the editing software. However, there are concerns that synchronization by analyzing sound recording results may be difficult to synchronize accurately due to the influence of noise such as ambient sound during recording, and that the processing load due to the analysis may increase on the PC running the editing software.

[0173] In contrast, with this system 10, the export function outputs video material management data that reflects the settings of cuts associated with the music range, allowing for accurate synchronization based on the management data without, for example, having to analyze the sound recording results during the video editing process. Furthermore, it eliminates the need for user operations such as issuing commands to generate synchronization information during video recording. The details of the export function of this system 10 will be described below.

[0174] 2.3.1. Overall operation of the export function The overall operation of the export function in this system 10 will be explained using Figures 17 to 19.

[0175] Figure 17 is a flowchart illustrating the operation of the export function in this system 10. Each process shown in the flowchart of Figure 17 is executed, for example, by the control unit 210 of the information support terminal 200. The processing in this flowchart starts, for example, when the export button 13 on the function selection screen (Figure 4) is operated, with the cut division data D1, which includes video metadata D2 from the cut shooting function, stored in the storage unit 220.

[0176] First, the control unit 210 of the information support terminal 200 sets the number of tracks in the video timeline 80 for lip-sync and the video timeline 80 for B-roll, respectively, based on the cut division data D1 (S61). For example, the number of tracks in the video timeline 80 for lip-sync is set according to the number of cuts in the cut division data D1 where the cut type is lip-sync and the shooting completion flag is on.

[0177] Furthermore, the control unit 210 configures a video folder 70 that is displayed in the material display area 61 of the video editing screen (Figure 16), for example, based on the cut division data D1 (S62). The video folder 70 obtained through this folder configuration process (S62) includes, for example, folders for each cut, i.e., cut folders 71, and KEEP folders 72 and OK folders 73 provided within each cut folder 71, as shown in Figure 16. The video folder 70 illustrated in Figure 16 further includes a music folder 74 for storing the sound source data of the music.

[0178] The KEEP folder 72 stores video files that have a "KEEP" rating in the video footage for that cut. The OK folder 73 stores video files that have an "OK" rating in the video footage for that cut. In the folder configuration process of step S62, the control unit 210 automatically writes the directory structure for realizing this configuration of video folders 70 into the management data.

[0179] In the folder configuration process (S62), the control unit 210 further describes the management data, for example, the file name and detailed information of each video file. The detailed information of the video file includes, for example, the duration of the video material (i.e., video period), the recording date and time, and the timer period. In step S62, for example, the control unit 210 generates management data for the video folder 70.

[0180] The icon representing a video file in the video folder 70 is an example of video information in this embodiment. The material display area 61 may also display an icon representing the video timeline 80 along with the video folder 70 as described above. In addition, although the video icon in cut folder 71 with cut number "1" is displayed in the example in Figure 16, according to the video folder 70, video icons in other cut folders 71 can also be displayed as appropriate through user operation on the material display area 61.

[0181] Furthermore, the control unit 210 sets the video timeline 80 that will be displayed in the timeline editing area 62 (Figure 16) when the user starts editing a video, based on the results of setting the number of tracks (S61) and the folder configuration process (S62) (S63). The video timeline 80 obtained through this timeline setting process (S63) is illustrated in Figures 18 and 19.

[0182] Figure 18 illustrates a video timeline 80 for lip-syncing in a video editing screen. The video timeline 80 for lip-syncing includes, for example, the number of video tracks 83 set according to the number of cuts, one music audio track 89, and one final video track 90, as shown in Figure 18. Each video track 83 contains a video clip 85 that shows the video in the video file for each cut. The music audio track 89 contains an audio clip 86a that shows the audio of the music. The final video track 90 is set as a blank track in the timeline setting process (S63), for example, and accepts editing operations on the video editing PC 300 to place the video clips 85 to be included in the video work.

[0183] In the video timeline 80 for lip-syncing, as shown in Figure 18, for example, the video tracks 83 corresponding to each cut are arranged in a direction V that intersects with the time axis 81. This allows for easy viewing of the video clips 85 of each video track 83 in relation to the time of the audio clip 86a of the music audio track 89. In the export function of this example, management data is generated so that, for example, the audio from the video file of the lip-sync cut is not loaded into the video timeline 80 for lip-syncing.

[0184] Figure 19 illustrates a video timeline 80 for B-roll in a video editing screen. The video timeline 80 for B-roll includes, for example, one video track 83 and a number of audio tracks 84 corresponding to each audio signal, such as the sound captured by the built-in microphone 160 of the digital camera 100 and an external microphone. The video track 83 includes video clips 85 for each cut. The audio track 84 includes audio clips 86b that indicate the audio corresponding to the video in the video file. The video / audio clips 85 and 86b are examples of video information in this embodiment.

[0185] In the B-roll video timeline 80, as shown in Figure 19, for example, each video clip 85 for each cut is placed on a single video track 83 in the direction along the time axis 81, and an audio clip 86b corresponding to the audio signal is displayed on each audio track 84. This makes it possible to easily display, for example, one or more audio clips 86b corresponding to the video of each video clip 85.

[0186] In the timeline setting process of step S63, each video timeline 80 is automatically set so that, according to the settings in the cut division data D1, the video / audio clips 85 and 86a for the lip-sync cuts are synchronized and placed on the video / audio tracks 83 and 84. Details of the timeline setting process (S63) will be described later.

[0187] The control unit 210 outputs management data (S64) including the results of the folder configuration process (S62) and the timeline setting process (S63), and terminates the process shown in the flowchart of Figure 17. The control unit 210 stores the management data in, for example, the storage unit 220.

[0188] Based on the export function processing described above, the information support terminal 200 of this system 10 can provide a video folder 70 and a video timeline 80 that facilitate the process of editing various video clips taken by the user in accordance with a scenario on the video editing PC 300 (S62, S63).

[0189] For example, according to the video timeline 80 (Figure 18) for lip-syncing provided by this system 10, the video material for each cut is placed on the video track 83 in sync with the audio clip 86a of the music audio track 89. This allows the user to edit videos with synchronized video and audio without having to run the functions of editing software for synchronization.

[0190] In this system 10, the video folder 70 can adopt various configurations. For example, it is not necessary to have an OK folder 73 and a KEEP folder 72 for every cut. Also, in this system 10, a folder may be further provided in the video folder 70 to store videos of takes that received an "NG" rating.

[0191] 2.3.2. Timeline Setup Process The details of the timeline setting process in step S63 of Figure 17 will be explained using Figures 20 and 21.

[0192] Figure 20 is a flowchart illustrating the timeline setting process (S63) in this system 10. Figure 21 illustrates the data structure of the management data D3 generated by the timeline setting process (S63). The management data D3 illustrated in Figure 21 includes the video folder 70 generated by the folder configuration process (S62) and the video timeline 80 generated by the timeline setting process (S63).

[0193] First, the control unit 210 generates separate video timelines 80 for lip-syncing and B-roll, respectively, for arranging video materials as shown in Figures 18 and 19 (S80). For example, the control unit 210 generates a video timeline 80 for lip-syncing so that audio clips 86a are placed on a music audio track 89 based on the music's sound source data. The control unit 210 determines the length of the music audio track 89, etc., based on information indicating the range of music to be included in the video work input in the scenario creation function (S11 in Figure 5). In the video timeline 80, the timing for starting music playback is set, for example, to the start timing of the video timeline 80.

[0194] Furthermore, the control unit 210, for example, refers to the cut division data D1 and sets the length of the video timeline 80 for the B-roll as the sum of the video durations of all cuts in the B-roll (S80).

[0195] Next, the control unit 210 selects one cut folder 71 from among multiple cut folders 71 based on the management data D3 for the video folder 70 obtained by, for example, the folder configuration process (S62) (S81). Step S81 is a process for determining the placement of video material on the video timeline 80 for each cut, and is performed, for example, in ascending order of cut number.

[0196] Based on the cut division data D1, the control unit 210 determines whether the cut type of the cut corresponding to the selected cut folder 71 is lip-sync or not (S82).

[0197] If the cut type of the selected cut is lip-sync (YES in S82), the control unit 210 places a video file corresponding to the music range of the cut on the video timeline 80 for lip-sync (Figure 18) based on the cut division data D1 (S83). For example, the control unit 210 extracts the video file with the most recent recording date and time from the video files of takes that have an evaluation of "OK" or "KEEP" in the cut, as the target to be placed on the video timeline 80.

[0198] The control unit 210, for example, after each execution of step S84, associates the video track 83 with the selected lip-sync cut in ascending order from track number "1" on the video timeline 80 for lip-sync (Figure 18). The control unit 210 also performs initial settings for each video track 83 regarding the playback of the preview image 64 (Figure 16) along the video timeline 80. For example, the video track 83 with track number "1" and the music audio track 89 are set to enable playback display and audio output, respectively, while the other video tracks 83 are set to disable playback display.

[0199] The control unit 210 determines the placement of video material on the video track 83 for the selected cut, based on management data D3 for the video folder 70 in addition to cut division data D1 (S83). For example, the control unit 210 first determines various timing information for the latest and highest-rated video material for that cut. For example, the control unit 210 refers to the timer period of the video material and determines the start timing to be adopted in the video timeline 80 for that video material. For example, if the timer period is "5 seconds", the start timing to be adopted is determined to be 5 seconds after the beginning of the video material.

[0200] Furthermore, the control unit 210 sets the start and end timings for playback of the video material in the lip-sync video timeline 80 based on the music range associated with the selected cut in the cut division data D1 (S83). For example, for a lip-sync cut, the start timing of video material playback is set to the start timing of the corresponding music range. The end timing of video material playback is set, for example, to a timing that is the playback duration of the video material after the start timing. If there is a timer period, the playback period is managed by subtracting the timer period from the total length of the video, such as the video duration.

[0201] The control unit 210 places video clips 85 of the video material corresponding to the extracted video file on the video track 83 so as to reflect the various timing settings described above (S83).

[0202] The control unit 210 refers to, for example, the shooting completion flag in the cut division data D1 to determine whether or not the placement settings for all cuts to be placed on the video timeline 80 have been completed (S85).

[0203] If the control unit 210 has cuts for which placement settings are not yet completed (NO in S85), it repeats the processing from step S81 onwards for the cuts that have not been set.

[0204] If the cut type of the cut selected in step S81 is not lip-sync (NO in S82), the control unit 210 places the video file of the cut on the video timeline 80 for B-roll (Figure 19) (S84). The control unit 210 manages the time range of the cut on the video timeline 80 by referring to the detailed information of the extracted video file, for example, in the same way as in step S83. The time range of the cut on the video timeline 80 is managed, for example, as the playback period of the video file to be placed.

[0205] In step S84, the control unit 210 performs initial settings for each audio track 84 regarding the playback of preview images 64 along the video timeline 80, for example, as shown in Figure 19. For example, audio track 84 with track number "1" is set to enable audio output, while the other audio tracks 84 are set to disable it. In each video timeline 80 in Figures 18 and 19, this playback display / audio output enable / disable can be switched as appropriate by, for example, the video editing PC 300 according to user operations on the video editing screen.

[0206] The control unit 210 determines the placement of video material in the video track 83 for the relevant cut, for example, based on the management data D3 of the video folder 70 (S84). For example, for each video material, the control unit 210 determines the start timing to be adopted in the video timeline 80 by referring to the timer period, similar to step S83. The control unit 210 further sets the playback start timing and playback end timing for the video material in the video timeline 80 for the B-roll (S84).

[0207] For example, the playback start timing for the first cut is set to the start timing of the video timeline 80 for the B-roll, and for the second and subsequent cuts, it is set to the end of the time range of the previous cut, i.e., the playback end timing of the video file for that cut. The playback end timing is set to a timing that is the playback duration of the video file after the playback start timing. The control unit 210 sets the placement of the audio clip 86b of the video material on the audio track 84 in the same way as setting the video clip 85 (S84).

[0208] The control unit 210 places video / audio clips 85 and 86b of the video material corresponding to the extracted video file on video / audio tracks 83 and 84 so as to reflect the various timing settings described above (S84).

[0209] Once the placement settings for all cuts are complete (YES in S85), the control unit 210 generates management data D3 for the video timeline 80, for example, by describing it in XML language (S86).

[0210] In step S86, the control unit 210, for example as shown in Figure 21, sets up tags for lip sync and B-roll in the video timeline 80 in management data D3, and sets up tags for each video and audio. Furthermore, the control unit 210 sets up track-specific tags under each video and audio tag. Under the lip sync tag, the control unit 210 sets up, for example, a tag for the final video track 90 in addition to the track number tag for the video. In the tag for each track number, the control unit 210 describes information regarding the placement settings, such as the video file name and timing information of the video clip 85 determined in step S83 for the corresponding lip sync cut.

[0211] For example, timing information includes a timer period and a video period, in addition to the playback start timing and playback end timing. In the timing information for lip-sync cuts, the control unit 210 describes the playback start timing according to the start time of the song range in the cut division data D1, based on, for example, the start timing of the video timeline 80 for lip-sync. This allows, for example, the video clips 85 of each cut to be placed on the music audio track 89 according to the song range set for that cut in the video timeline 80 for lip-sync (Figure 18) corresponding to the duration of the song. The timing information may also be described by the number of frames based on the frame rate of the video file.

[0212] Furthermore, in step S86, the control unit 210 sets up a tag for the music audio track 89 under the lip-sync tag, describing, for example, the file name of the music's sound source data and the range of the music in the video work.

[0213] Furthermore, under the B-roll tags, the control unit 210 provides one track tag for video and tags for audio corresponding to various microphones, etc. For each track tag, the control unit 210 describes information regarding placement settings, such as the video file name and timing information of the video / audio clips 85, 86b determined in step S84, for each cut.

[0214] With the generation of management data D3 for these 80 video timelines, the control unit 210 terminates the timeline setting process (S63) and proceeds to step S64 in Figure 17.

[0215] According to the timeline setting process (S63) described above, the information support terminal 200 of this system 10 can prepare a video timeline 80 in which video / audio clips 85 and 86 are arranged according to the scenario, before the video editing process, so as to reflect the cut settings. As a result, the user can start video editing without the trouble of arranging video material using the cut shooting function according to the scenario, and at that time obtain a video timeline 80 in which the video for each cut is arranged in sync with the audio of the music (S83, see Figure 18).

[0216] In the example above, one take of video material was placed for each cut in each video timeline 80 (see S83, S84, Figures 18 and 19). The control unit 210 may place multiple takes of video material for each cut. For example, in the video timeline 80 for lip-sync, the video material for each take may be placed in video track 83 with consecutive track numbers for each cut. Also, in the video timeline 80 for B-roll, multiple takes of video material may be placed consecutively in the direction of the time axis 81, for example, in video / audio tracks 83 and 84 for each cut. The time range management in step S84 may be performed, for example, based on the playback period of the video material placed in each cut.

[0217] 2.4. Operation of the video editing PC Based on the management data D3 output by the export function described above, the operation when editing a video work is performed on the video editing PC300 will be explained using Figures 22 and 23.

[0218] Figure 22 is a flowchart illustrating the operation of the video editing PC 300 in the imaging system 10. Figure 23 is a diagram illustrating the operation of the video editing PC 300. The processing in this flowchart is executed by the control unit of the video editing PC 300 after the management data D3 is output from the information support terminal 200 via the export function in the imaging system 10 (S64 in Figure 17).

[0219] First, the video editing PC 300 receives management data D3, etc. (S91). For example, the video editing PC 300 obtains management data D3 and music sound source data from the information support terminal 200 via the communication unit, and further obtains video data of the shooting result from the digital camera 100. The management data D3 may be input to the video editing PC 300 from a memory card 142, etc., or it may be transferred from the information support terminal 200 to the digital camera 100 and input to the video editing PC 300 together with the video data.

[0220] The video editing PC 300 loads video data and music audio data into the editing software based on management data D3, and displays a video editing screen as shown in Figure 16, for example (S92). Based on management data D3 for video timeline 80 (Figure 21), the video editing PC 300 synchronizes and places the video clip 85 of the lip-sync cut with the audio clip 86a of the music audio track 89 on video timeline 80.

[0221] The video editing PC300 accepts various editing operations via its control panel on the displayed video editing screen (S93). Figure 23 shows an example of a video timeline 80 for lip-syncing, where the lip-sync tab 87 is selected on the video editing screen (Figure 16) and editing operations have been performed.

[0222] For example, as shown in Figure 23, in the video timeline 80 for lip-syncing, various cuts of video clips 85 are placed on the final video track 90 through editing operations. In this example, the video clips 85 and corresponding audio clips 86 of the B-roll cuts can be copied from the video timeline 80 for B-roll to the video timeline 80 for lip-syncing through editing operations.

[0223] According to the operation of the video editing PC300 as described above, editing operations can be performed on the video timeline 80 with the video and audio synchronized by the management data D3, making the video editing process easier.

[0224] 3. Summary As described above, in this embodiment, an information support terminal 200, which is an example of an electronic device, manages video in a scenario that includes multiple cuts. The information support terminal 200 includes a display unit 240 that displays information, an operation unit 230, which is an example of an input unit that receives user operations, and a control unit 210 that controls the display unit 240 according to the operations input in the operation unit 230. The control unit 210 displays a setting screen (see Figures 6 and 7) related to sound source data associated with the scenario on the display unit 240, and acquires cut division data D1, which is an example of audio setting information that associates cuts in multiple cuts with a song, which is an example of audio indicated by the sound source data, based on user operations in the operation unit 230 (S10~S17). Based on the acquired cut division data D1, the control unit 210 generates management data D3, which manages the positional relationship in which video information representing each video is arranged in chronological order in the song, on a video editing screen (Figure 16), which is an example of an editing screen for editing multiple videos associated with multiple cuts (S63~S64).

[0225] According to the information support terminal 200 described above, the management data D3, which manages the positional relationship of video information arranged chronologically in the music on the video editing screen, makes it easier for the user to perform video editing processes according to the music for each cut of video footage. Furthermore, in this system 10, such management data D3, including the video timeline 80, can improve the processing efficiency of the process by, for example, the video editing PC 300.

[0226] In the information support terminal 200 of this embodiment, the control unit 210, in accordance with user operation on the operation unit 230, sets a song range in the cut division data D1 for a lip-sync cut (an example of a cut associated with a song) among multiple cuts, which is an example of a time range associated with the video of the cut in the song in the scenario (S12~S14). Based on the cut division data D1 set for each cut in the multiple cuts (see Figure 8), the control unit 210 generates management data D3 on the video editing screen so that the video information of the lip-sync cut is placed within the time range set in the time series of the sound source data (S63, S83, S86). This makes it easier to perform video editing according to the song, for example, by placing the lip-sync cut within the corresponding song range in the time series of the song indicated by the sound source data.

[0227] In the information support terminal 200 of this embodiment, the control unit 210 further sets B-roll cuts (an example of cuts not associated with music) among multiple cuts in the cut division data D1 according to user operation on the operation unit 230 (S12, S15). Based on the cut division data D1 set for each cut (see Figure 8), the control unit 210 generates management data D3 to identify lip-sync cuts and B-roll cuts among multiple cuts (S63, S82~S84, S86). This makes it possible to arrange, for example, each cut on the video timeline 80 according to the lip-sync and B-roll settings, and further facilitates video editing (see Figures 16~18).

[0228] In this embodiment, management data D3 manages a video timeline 80, which is an example of a timeline in which video clips 85 and audio clips 86b (each an example of video information) are arranged chronologically on the video editing screen (see Figures 18, 19, and 21). The control unit 210 generates management data D3 so that the video clips 85 of the lip-sync cuts are arranged on the video timeline 80 according to the music range set in the chronological order of the sound source data (S63, S82~S83, S86). This makes it possible to arrange, for example, the audio clips 86a of the music and the video clips 85 of the lip-sync cuts in the video timeline 80 for lip-syncing in sync according to the music range set for the cut.

[0229] The information support terminal 200 of this embodiment further includes a speaker 270, which is an example of an audio output unit that outputs audio. The control unit 210 controls video shooting for each cut based on the cut division data D1 (S21-S27), and when shooting video for a lip-sync cut, it outputs audio to the speaker 270 to play music, for example, according to the set range of the music (S25, S42-S45). This makes it possible to shoot video for a lip-sync cut while playing audio from the set range of the music. This also makes it easier to shoot videos that match the music, and easier to edit videos according to the music.

[0230] In the information support terminal 200 of this embodiment, the control unit 210 sets a timer period, which is an example of a predetermined period, according to the user operation on the operation unit 230 (S44). When shooting a video of a lip-sync cut, the control unit 210 controls the speaker 270 to output audio from before the range of the music set for that cut, according to the timer period (S45, see Figure 14). This makes it easier to synchronize the timing when shooting a video of that cut, for example, and makes it easier to shoot a video that is synchronized with the music.

[0231] In the information support terminal 200 of this embodiment, the control unit 210 receives a user operation in the operation unit 230 to associate one or more cuts from a plurality of cuts with a given time range in a song, and acquires cut division data D1 (S13-S14, S17). This allows, for example, one or more lip-sync cuts to be set, associated with overlapping time ranges within a song.

[0232] In this embodiment, a video management method is provided for managing videos in a scenario that includes multiple cuts. In this method, the control unit 210 of the information support terminal 200 displays a setting screen (see Figures 6 and 7) related to sound source data associated with the scenario on the display unit 240, and, through user operation on the operation unit 230, acquires cut division data D1, which is an example of audio setting information that associates cuts in multiple cuts with a song, which is an example of the audio indicated by the sound source data (S10-S17). Based on the acquired cut division data D1, the control unit 210 generates management data D3 (S63-S64) that manages the positional relationship in which video information representing each video is arranged in chronological order in the song, on a video editing screen (Figure 16), which is an example of an editing screen for editing multiple videos associated with multiple cuts.

[0233] In this embodiment, a program is provided to cause the control unit 210 to execute the above video management method. This video management method makes it easier to edit videos in a scenario that includes multiple cuts associated with sound source data. The various information management for each cut as described above is not necessarily limited to a single cut unit, but may also be performed in units of a set of cuts that include multiple cuts.

[0234] In this embodiment, the video editing PC 300, an example of a video editing device, includes, for example, a display unit that displays a video editing screen (see Figure 23), an example of an editing screen for editing multiple videos, similar to the configuration of the information support terminal 200 shown in Figure 3; a communication unit, an example of a data input unit for inputting data; an operation unit that accepts user operations; and a control unit that controls the display unit based on the data input from the data input unit. The control unit acquires from the data input unit video data indicating videos shot in association with each cut in a scenario containing multiple cuts, sound source data associated with the scenario, and management data D3 that associates the cuts in the multiple cuts with a song, an example of the sound indicated by the sound source data (S91). The management data D3 manages the positional relationship in which the video information indicating each video associated with multiple cuts is arranged in chronological order in the song on the video editing screen (see Figures 21 and 23). Based on the management data D3, the control unit displays a video timeline 80 on the video editing screen, which is an example of a timeline in which video information of lip-sync cuts (an example of cuts associated with the music) in multiple cuts is arranged in the timeline of the music (S92). The control unit 210 accepts user operations on the video editing screen to edit each video indicated by the video information via the operation unit (S93).

[0235] According to the video editing PC300 described above, the management data D3 allows for the arrangement of video information for each cut and the time range in the music on the video timeline 80 of the video editing screen, making it easier for the user to perform video editing according to the music. Furthermore, it can improve the processing efficiency of this process on the video editing PC300.

[0236] (Other embodiments) As described above, Embodiment 1 has been presented as an example of the technology disclosed in this application. However, the technology in this disclosure is not limited to this and can be applied to embodiments that are modified, substituted, added, or omitted as appropriate. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiment.

[0237] In Embodiment 1 described above, an information support terminal 200 was described as an example of an electronic device separate from the imaging device, but this disclosure is not limited thereto. The electronic device in this embodiment may be integrated with the imaging device that performs video recording. In this embodiment, for example, a digital camera 100 may have various functions similar to those of the information support terminal 200 described above. Such a digital camera 100 can also provide information support to the user through a scenario creation function, an export function, etc., similar to Embodiment 1.

[0238] In each of the embodiments described above, a cut selection screen including a cut list 30 has been illustrated, but the selection screen of this disclosure is not limited thereto. The selection screen of this embodiment does not have to include a cut list 30, and may include multiple cuts in a display configuration different from the cut icon 3. Furthermore, the selection screen of this embodiment may be a dialog box display, or it may be displayed superimposed on various display screens. In this embodiment, the cut list 30 may be just one example of a selection screen. In this embodiment, the selection screen of the information support terminal 200 may identify and display whether or not video recording has been completed for each cut in various display configurations not limited to the examples described above.

[0239] In the embodiments described above, three types of evaluation information were explained: "OK," "KEEP," and "NG." However, the evaluation information is not limited to these. In this embodiment, the evaluation information may be three different types of evaluations, or it may be two or four or more types, not just three. Also, in this embodiment, the evaluation information may be a continuous score. The electronic device of this embodiment may receive user input of such various types of evaluation information and manage video recording for each cut. For example, by appropriately setting criteria for whether or not video recording for a cut has been completed, identification and display can be performed. Furthermore, the information support terminal 200 of this embodiment may set priorities in the management data D3 to reflect these various types of evaluations.

[0240] In each of the embodiments described above, an example was given in which the management data D3 includes a video folder 70 and a video timeline 80. In this embodiment, the management data may be generated by omitting, for example, the video folder 70. Also, in each of the embodiments described above, an example was given in which the management data D3 is written in XML language. In this embodiment, the management data D3 is not limited to XML language, but may be written in various markup languages ​​or data description languages ​​that are supported by the editing software used in the video editing process. The management data D3 may also be various metadata that manages the video in various cuts.

[0241] Furthermore, in each of the above embodiments, an information support terminal 200 was described as an example of an electronic device, and a video editing PC 300 was described as an example of an external device. In this embodiment, the electronic device may be a video editing PC 300, and the video editing PC 300 may have the various functions of the information support terminal 200 described above. In this embodiment, the editing screen is not limited to an external device, but may be displayed on the electronic device. Also, if an external display device is used with the video editing PC 300, the external device in this embodiment may include such a display device.

[0242] Furthermore, in each of the above embodiments, a digital camera 100 equipped with an optical system 110 and a lens drive unit 112 was provided as an example. The imaging device of this embodiment does not necessarily have to be equipped with an optical system 110 and a lens drive unit 112, and may be, for example, a camera with interchangeable lenses.

[0243] Furthermore, while a digital camera was described as an example of an imaging device in each of the embodiments described above, the invention is not limited to this. The imaging device of this disclosure may be any electronic device having an image capture function (e.g., a video camera, smartphone, tablet terminal, etc.). Moreover, the electronic device of this disclosure does not need to have an image capture function in particular, and may be any type of electronic device.

[0244] (Example of form) The following are examples of various aspects of this disclosure.

[0245] The first aspect of this disclosure is an electronic device for managing video in a scenario that includes multiple cuts. The electronic device manages video in a scenario that includes multiple cuts. The electronic device comprises a display unit for displaying information, an input unit for inputting user operations, and a control unit for controlling the display unit in accordance with the operations input in the input unit. The control unit displays a setting screen for sound source data associated with the scenario on the display unit, and acquires audio setting information that associates cuts in the multiple cuts with the time range of the audio indicated by the sound source data based on user operations in the input unit. Based on the acquired audio setting information, the control unit generates management data that manages the positional relationship in which video information representing each video is arranged in chronological order in the sound source data on an editing screen for editing multiple videos associated with multiple cuts.

[0246] In the second embodiment, in the electronic device described in the first embodiment, the control unit sets a time range in the audio setting information to associate the audio of a cut associated with an audio from among multiple cuts with the video of that cut in the audio source data, and generates management data on the editing screen such that the video information of the cut associated with the audio is placed within the time range set in the time series of the audio source data, based on the audio setting information set for each cut in the multiple cuts.

[0247] In a third embodiment, in the electronic device described in the second embodiment, the control unit further sets in the audio setting information cuts that are not associated with audio among a plurality of cuts according to user operation in the input unit, and generates management data to identify cuts that are associated with audio and cuts that are not associated with audio among the plurality of cuts based on the audio setting information set for each cut.

[0248] In the fourth aspect, in the electronic device described in the second or third aspect, the management data manages a timeline in which video information is arranged chronologically on the editing screen. The control unit generates the management data such that the video information of cuts associated with audio in multiple cuts is arranged on the timeline according to the time range set in the chronological order of the recorded audio data.

[0249] In the fifth embodiment, the electronic device described in any of the second to fourth embodiments further comprises an audio output unit that outputs audio. The control unit controls the shooting of video for each cut in a plurality of cuts based on audio setting information, and causes the audio output unit to output audio to the video output unit in accordance with a set time range during the shooting of video for cuts associated with audio.

[0250] In the sixth embodiment, in the electronic device described in the fifth embodiment, the control unit sets a predetermined period of time according to user operation in the input unit, and in video recording of cuts associated with the audio, controls the audio output unit to output audio from before the time range set in the audio of the sound source data, according to the predetermined period.

[0251] In the seventh embodiment, in the electronic device described in any of the first to sixth embodiments, the control unit receives a user operation in the input unit to associate one or more cuts from a plurality of cuts with a given time range in the audio, and acquires audio setting information.

[0252] The eighth aspect is a video management method for managing videos in a scenario that includes multiple cuts. In this method, the control unit of an electronic device displays a setting screen for sound source data associated with the scenario on the display unit, and acquires sound setting information that associates cuts in multiple cuts with the audio indicated by the sound source data through user operation on the input unit. Based on the acquired sound setting information, the control unit generates management data that manages the positional relationship in which video information representing each video is arranged in chronological order in the audio of the sound source data on an editing screen for editing multiple videos associated with multiple cuts.

[0253] In the ninth aspect, a program is provided for causing a control unit to execute the video management method described in the eighth aspect.

[0254] The tenth embodiment is a video editing device. The video editing device comprises a display unit that displays an editing screen for editing multiple videos, a data input unit for inputting data, an operation unit that accepts user operations, and a control unit that controls the display unit based on the data input from the data input unit. The control unit acquires from the data input unit video data indicating videos shot in association with each cut in a scenario containing multiple cuts, sound source data associated with the scenario, and management data that associates the cuts in the multiple cuts with the audio indicated by the sound source data. The management data manages the positional relationship in which the video information indicating each video associated with the multiple cuts is arranged in chronological order in the audio of the sound source data on the editing screen. Based on the management data, the control unit displays on the display unit a timeline in which the video information of the cuts associated with the audio in the multiple cuts is arranged in chronological order in the audio source data, and accepts user operations on the editing screen to edit each video indicated by the video information via the operation unit.

[0255] As described above, embodiments have been explained as examples of the technology in this disclosure. For this purpose, accompanying drawings and a detailed description have been provided. Therefore, among the components described in the accompanying drawings and detailed description, there may be not only components that are essential for solving the problem, but also components that are not essential for solving the problem, in order to illustrate the above technology. [Industrial applicability]

[0256] This disclosure is applicable to various uses of video recording that includes multiple cuts associated with audio such as music. [Explanation of Symbols]

[0257] 10 Imaging System 100 Digital Cameras 200 Information Support Terminals 210 Control Unit 220 Memory Department 230 Operations Department 240 indicates the section 250 Ministry of Communications

Claims

1. An electronic device for managing video in a scenario that includes multiple cuts, A display unit that displays information, An input section for user input, The system comprises a control unit that controls the display unit in accordance with the operation input to the input unit, The control unit, The display unit displays a settings screen for the sound source data associated with the scenario, and the user, through operation on the input unit, obtains audio setting information that associates the cuts in the multiple cuts with the audio indicated by the sound source data. Based on the acquired audio setting information, in the editing screen for editing multiple videos associated with the multiple cuts, management data is generated to manage the positional relationship in which the video information representing each video is arranged in chronological order in the audio of the sound source data. electronic equipment.

2. The control unit, In accordance with user operation in the input unit, the audio setting information sets a time range in the audio of the sound source data to associate with the video of the cut associated with the audio among the multiple cuts, Based on the audio setting information set for each cut in the aforementioned multiple cuts, the management data is generated in the editing screen so that the video information of the cut associated with the audio is placed within the set time range in the time series of the sound source data. The electronic device according to claim 1.

3. The control unit, In accordance with the user operation in the input unit, the audio setting information further sets the cuts among the plurality of cuts that are not associated with the audio, Based on the audio setting information set for each cut, the management data is generated to identify which cuts are associated with the audio and which are not associated with the audio in the plurality of cuts. The electronic device according to claim 2.

4. The aforementioned management data manages the timeline in which the video information is arranged chronologically on the editing screen. The control unit generates the management data such that the video information of the cuts associated with the audio in the plurality of cuts is placed on the timeline according to the set time range in the time series of the sound source data. The electronic device according to claim 2.

5. It further includes an audio output unit that outputs audio, The control unit controls the video recording of each cut in the plurality of cuts based on the audio setting information, and in the video recording of the cut associated with the audio, it causes the audio to be output to the audio output unit according to the set time range. The electronic device according to claim 2.

6. The control unit, A predetermined period is set according to the user operation in the input unit. In the video recording of the cut associated with the aforementioned audio, the audio output unit is controlled to output audio from the audio source data starting before the set time range, according to the predetermined period. The electronic device according to claim 5.

7. The control unit receives a user operation at the input unit to associate one or more cuts from the plurality of cuts with a time range in the audio, and acquires the audio setting information. The electronic device according to claim 1.

8. A video management method for managing videos in a scenario that includes multiple cuts, The control unit of the electronic device, The settings screen for the sound source data associated with the scenario is displayed on the display unit, and by user operation on the input unit, audio setting information is obtained that associates the cuts in the multiple cuts with the audio indicated by the sound source data. Based on the acquired audio setting information, in the editing screen for editing multiple videos associated with the multiple cuts, management data is generated to manage the positional relationship in which the video information representing each video is arranged in chronological order in the audio of the sound source data. Video management methods.

9. A program for causing the control unit to execute the video management method described in claim 8.

10. A display unit that shows an editing screen for editing multiple videos, A data input section for entering data, An operating unit that accepts user input, The system comprises a control unit that controls the display unit based on data input from the data input unit, The control unit obtains from the data input unit video data showing video footage shot in association with each cut in a scenario including multiple cuts, sound source data associated with the scenario, and management data relating the cuts in the multiple cuts to the audio indicated by the sound source data. The management data manages the positional relationship in the editing screen where video information representing each video associated with the multiple cuts is arranged in chronological order in the audio of the sound source data. The control unit, Based on the management data, the editing screen displays a timeline in which the video information of the cuts associated with the audio in the multiple cuts is arranged in chronological order of the audio source data. In the aforementioned editing screen, the operation unit accepts user operations to edit each video indicated by the video information. Video editing equipment.

Images