Information processing device, information processing method, and program

Abstract

To appropriately control camera work on the basis of changes during an actual performance relative to a rehearsal.SOLUTION: A first camerawork, which is the camerawork during rehearsal imaging, and first audio information, which is audio information during rehearsal imaging, are acquired. Second audio information, which is audio information during actual imaging corresponding to the rehearsal imaging, is acquired. A second camerawork, which is the camerawork during actual imaging, is determined on the basis of the first camerawork, the first audio information, and the second audio information.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, there are known methods for determining camera work in response to changes in the situation in live music broadcasts, etc. For example, Patent Document 1 proposes a method for switching camera work based on music score information when the progress of a musical performance reaches a specific position in the score. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6753460 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the prior art disclosed in the above-mentioned Patent Document 1 determines when a specific position in the score (musical score) has been reached, and therefore does not allow for camerawork that takes into account the tempo leading up to that position or the excitement of the venue (cheers). Specifically, when filming a live music concert, it is not possible to adjust the speed of the camerawork to match the tempo. Also, when filming a live music concert, the camera is usually shaken slightly to match the excitement of the venue (volume of cheers), but the way the camera is shaken does not reflect the excitement of the venue (volume of cheers).

[0005] An object of the present invention is to appropriately control camerawork based on changes in the actual performance compared to the rehearsal. [Means for solving the problem]

[0006] In order to achieve the object of the present invention, for example, an information processing device according to one embodiment has the following configuration: a first acquisition means for acquiring first camerawork that is camerawork during rehearsal shooting and first sound information that is sound information during the rehearsal shooting, a second acquisition means for acquiring second sound information that is sound information during actual shooting relative to the rehearsal shooting, and a determination means for determining second camerawork that is camerawork during the actual shooting based on the first camerawork, the first sound information, and the second sound information. [Effects of the Invention]

[0007] To appropriately control camerawork based on changes in the actual performance compared to the rehearsal. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram of a system including an information processing device according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration of each device according to the first embodiment. [Figure 3] FIG. 1 is a block diagram showing an example of the functional configuration of an information processing device according to a first embodiment. [Figure 4] 10 is a flowchart showing an example of processing during rehearsal imaging. [Figure 5] FIG. 10 is a diagram showing an example of a UI for acquiring user input. [Figure 6] 10A and 10B are diagrams showing an example of camera work and sound information during rehearsal filming. [Figure 7] 5 is a flowchart showing an example of processing during actual image capture according to the first embodiment. [Figure 8] FIG. 10 is a diagram showing an example of sound information (pattern 1) during actual imaging according to the first embodiment. [Figure 9] FIG. 10 is a diagram showing an example of sound information (pattern 2) during actual imaging according to the first embodiment. [Figure 10] FIG. 10 is a schematic diagram of a system including an information processing device according to a second embodiment. [Figure 11]FIG. 10 is a block diagram showing an example of the hardware configuration of each device according to the second embodiment. [Figure 12] FIG. 10 is a block diagram showing an example of the functional configuration of an information processing device according to a second embodiment. [Figure 13] 10A and 10B are diagrams showing an example of camera work and sound information during rehearsal filming. [Figure 14] 10 is a flowchart showing an example of processing during actual image capture according to the second embodiment. [Figure 15] FIG. 10 is a diagram showing an example of sound information (pattern 1) during actual imaging according to the second embodiment. [Figure 16] FIG. 10 is a diagram showing an example of sound information (pattern 2) during actual imaging according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] [Embodiment 1] The information processing device according to the first embodiment determines the camerawork for the actual shooting based on the camerawork for the rehearsal shooting, the sound information for the rehearsal shooting, and the sound information for the actual shooting relative to the rehearsal shooting. For example, assuming that a live music concert will be shot, the information processing device can determine the camerawork for the actual shooting based on the tempo for the rehearsal shooting and the tempo for the actual shooting.

[0011] FIG. 1 is a schematic diagram showing an example of the configuration of a control system including an information processing device according to this embodiment. The information processing device 100 is a device that performs processing of the control system. The control system shown in FIG. 1 includes a stage 101, audience seats 102, performers 103 and 105, musical instruments 104, a microphone 106 that captures sounds from the stage, audience members 107 and 108, and cameras 109, 110, and 111. The information processing device 100 can control each device via a network 112. The network 112 may connect the information processing device 100 to other devices via wired communication or wireless communication. A user 113 is a user who uses the information processing device 100 in the control system.

[0012] 1, the control system includes multiple cameras, but the control system may include only one camera. In this case, the camera ID is always set to 1 in the processing described below.

[0013] 2 is a block diagram showing an example of the hardware configuration of the information processing device 100, cameras 109, 110, and 111, and microphone 106 according to this embodiment. The information processing device 100 has a CPU 210, a ROM 211, a RAM 212, a communication unit 213, a storage medium 214, and a display / input unit 215, and these components are connected via a bus 216.

[0014] The CPU 210 is a control unit consisting of at least one processor or circuit, and controls the entire information processing device 100. The ROM 211 is an electrically erasable and recordable memory that stores constants, programs, etc. for the operation of the CPU 210. The program in this embodiment refers to a computer program for executing the processes related to various flowcharts described below. The RAM 212 loads constants, variables, etc. for the operation of the CPU 210, or programs read from the ROM 211. The communication unit 213 is an interface for communicating with external devices such as network devices or USB devices, and performs data communication via a network or sending and receiving data to and from external devices. The storage medium 214 is a recording medium such as a memory card, and is composed of a semiconductor memory, etc. The display / input unit 215 is composed of buttons or a touch panel, etc., and a display device such as an LCD monitor, and accepts user input and displays the results of the operation.

[0015] The cameras (109, 110, and 111) each have a CPU 220, a ROM 221, a RAM 222, a communication unit 223, a storage medium 224, an imaging unit 225, and a camera control unit 226, and the respective components are connected via a bus 227. Here, the cameras 109, 110, and 111 are imaging devices each having the same configuration and function, and hereinafter, when simply referred to as "camera," there is no distinction between them. However, the cameras may have different configurations as long as they can be controlled in the same way.

[0016] The CPU 220 is a control unit consisting of at least one processor or circuit, and controls the entire camera. The ROM 221 is an electrically erasable and recordable memory that stores constants or programs for the operation of the CPU 220. The RAM 222 expands constants, variables, or programs read from the ROM 221 for the operation of the CPU 220. The communication unit 223 is an interface for communicating with external devices such as network devices or USB devices, and performs data communication via a network or sending and receiving data to and from external devices. The storage medium 224 is a recording medium such as a memory card, and is composed of semiconductor memory or the like. The imaging unit 225 is an imaging element composed of a CCD or CMOS element that converts optical images into electrical signals. The camera control unit 226 controls the pan, tilt, zoom, and other aspects of the camera.

[0017] The microphone 106 has an audio input unit 230 and a communication unit 231, each of which is connected via a bus 232. The audio input unit 230 is a device that converts external vibration sound into sound data. The communication unit 231 is an interface for communicating with an external device such as a network device or a USB device, and performs data communication via a network or sending and receiving data to and from the external device.

[0018] Note that, in this description, the information processing device 100, the camera, and the microphone 106 are each assumed to be different devices, but the configuration of the control system is not limited to this as long as the information processing device 100 can perform similar processing. For example, the information processing device 100 may have some or all of the functions of the camera and the microphone 106, or the camera may include the microphone 106 within the same device.

[0019] 3 is a block diagram showing an example of the functional configuration of the information processing device 100, cameras 109, 110, and 111, and microphone 106 that constitute the control system, and the logical connection between them. The processing by each functional unit will be described in detail later.

[0020] The information processing device 100 includes a rehearsal camerawork and sound information acquisition unit (first acquisition unit) 301, a performance sound information acquisition unit (second acquisition unit) 303, a camerawork determination unit (determination unit) 305, a performance video distribution unit (video distribution unit) 306, a display and input unit 307, and a communication unit 308. The first acquisition unit 301 acquires camerawork during rehearsal filming and sound information during rehearsal filming as first information 302 indicating the camerawork and sound information during rehearsal filming. The second acquisition unit 303 acquires sound information during the performance as second information 304. The first acquisition unit 301 and the second acquisition unit 303 can acquire sound information acquired via the sound input unit 230 of the microphone 106.

[0021] The determination unit 305 determines the camera work for the actual shooting based on the camera work for the rehearsal shooting, the sound information for the rehearsal shooting, and the sound information for the actual shooting. The camera work according to this embodiment includes the control details of the pan-tilt-zoom (PTZ) of the camera and information (camera ID) indicating the camera that will perform the control (and capture images) at that timing.

[0022] For example, the determination unit 305 can determine the camera work in the second audio section that chronologically follows the first audio section during the rehearsal shooting, based on sound information of the first audio section (hereinafter also referred to as "cut") during the rehearsal shooting and sound information of the first audio section during the actual shooting. Details of such processing performed by the determination unit 305 will be described later with reference to FIGS. 6 to 9.

[0023] The video distribution unit 306 acquires data captured by the camera's imaging unit 225 via the communication units 223 and 213, and distributes the data acquired from the camera with the camera ID to be distributed at the current timing via the communication unit 213.

[0024] The display and input unit 307 displays the processing results or acquires user input via the display and input unit 215. The communication unit 308 transmits and receives information to and from the outside via the communication unit 213.

[0025] 4 is a flowchart showing an example of processing performed by the information processing device 100 according to this embodiment during rehearsal shooting performed before the actual shooting. The processing shown in FIG. 4 is started, for example, when a user instructs the start of the processing.

[0026] In S401, the first acquisition unit 301 acquires camera work and sound information during rehearsal shooting, and ends the processing shown in Fig. 4. The camera work and sound information (first information 302) acquired by the first acquisition unit 301 in this embodiment will be described below.

[0027] The first acquisition unit 301 can acquire the first information 302 based on a user input entered via a screen such as that shown in Fig. 5, for example. Fig. 5 will be described below. Fig. 5 shows a UI displayed on a screen that is used when a user inputs the first information. In this embodiment, a path to a file including the first information 302 is entered in a frame 501, and the file with the entered path is saved in the information processing device 100 by pressing a save button 502.

[0028] FIG. 6 is a diagram showing an example of the first information 302 acquired by the first acquisition unit 301. The first information 302 includes a cut ID indicating a cut to be captured, a camera ID for each cut ID, PTZ control details, and sound information. The cut ID is information indicating a specific audio section (cut) during capture (here, in chronological order), and is shown as an ID numbered for each timing at which the camerawork changes. Here, the PTZ control details include a start position indicating the PTZ position of the camera at the start of the cut, and a movement speed indicating the change per second of the PTZ of the camera from the start position (positive / negative for the direction of movement, and numerical for the amount of movement). Here, the "movement" of the camera includes a change in the camera position, a change in the camera's attitude, a change in the camera's zoom amount, etc. The sound information also includes a tempo (BPM), which is the number of beats per minute (the number of quarter notes), and sound data, which is data representing the change in sound volume and pitch over time as a digital signal. To make it easier to visualize, this table expresses sound data as do-re-mi... In other words, in the example of Figure 6, the range of data that expresses the sounds do-re-mi... as a digital signal is defined as cut ID (1). Similarly, the timing of each cut change (the timing of moving to the section of the next cut ID) is defined as the position in the data that expresses the sound data as a digital signal.

[0029] Note that here, a section for one cut ID is a section set during rehearsal filming, and is a predetermined section that can be defined acoustically, such as one measure of a musical score, or a section separated by recognizing a predefined sound, etc. Note that the lengths of the sections defined by the cut ID may be the same or different.

[0030] 5 has been described assuming that a file specified on the screen is acquired as the first information 302, the acquisition method is not limited to this as long as it is possible to similarly acquire camerawork and sound information during rehearsal shooting. For example, the camerawork included in the first information 302 may be acquired by acquiring the details of camera operation during rehearsal shooting. Furthermore, the sound information included in the first information 302 may be generated from sound information acquired by the microphone 106 during rehearsal shooting.

[0031] 7 is a flowchart showing an example of processing performed by the information processing device 100 according to this embodiment during actual shooting after rehearsal shooting. The processing shown in FIG. 7 is started, for example, when the user instructs the start of processing.

[0032] In S701, the second acquisition unit 303 acquires sound information during actual image capture. Here, the second acquisition unit 303 can acquire sound information collected by the microphone 106 during actual image capture as data represented by a digital signal.

[0033] Fig. 8 is a diagram showing an example of sound information (second information 304) acquired by the second acquisition unit 303. In the example of Fig. 8, sound information up to the end timing of cut ID (1) in the actual shooting is collected by the microphone 106. The example shown in Fig. 8 shows sound information acquired in pattern 1, which will be described later. Here, the sections corresponding to each cut ID and the sounds at the end of each section are defined in advance during the rehearsal shooting, and the second acquisition unit 303 is able to acquire sound information for each section by recognizing the sounds at the end of such sections in S701.

[0034] In S702, the determination unit 305 calculates the rate of change in tempo between the rehearsal and the actual performance based on the first information 302 and the second information 304. Here, for example, the determination unit 305 can calculate the rate of change in tempo between the rehearsal and the actual performance by comparing the tempo during the rehearsal shooting with the actual tempo during the actual shooting. Hereinafter, when simply referred to as a "rate of change in tempo," it refers to the rate of change in tempo between the rehearsal and the actual performance for the same cut. Here, the determination unit 305 calculates the rate of change in tempo by comparing data expressed as digital sound signals between the sound information during the rehearsal shooting and the sound information during the actual shooting, and aligning the data with the same sound.

[0035] In S703, the determination unit 305 determines camerawork based on the tempo change rate calculated in S702. Here, the determination unit 305 determines camerawork for the actual shooting by correcting the camerawork used during the rehearsal shooting based on the calculated tempo change rate (ratio). Here, for sections of the sound information used during the actual shooting that cannot be synchronized with the sound information used during the rehearsal shooting (sections that cannot be associated with the rehearsal shooting), the determination unit 305 determines camerawork to perform fixed tracking or fixed control. Furthermore, the timing of cut transitions is calculated by aligning the sound used during the actual shooting with the cut transition sound used during the rehearsal. For example, the determination unit 305 can determine camerawork for the actual shooting by correcting the camerawork used during the rehearsal shooting for cut ID (i+1) based on the tempo change rate for cut ID (i). Here, a certain cut is represented by cut ID (i), and the cut chronologically following that cut is represented by cut ID (i+1) (where i = 1, 2, 3, ...). In the following, cut ID (i+1) may be referred to as the current cut, and cut ID (i) may be referred to as the previous cut.

[0036] In S704, the communication unit 308 issues a control instruction via the communication units 213 and 223 to the camera control unit 226 of the camera with the camera ID corresponding to the cut ID so that control based on the camerawork determined in S703 is performed.

[0037] In S705, the determination unit 305 determines whether or not to continue the process. If the process is to continue, the process returns to S701; if not, the process in FIG. 7 ends. Here, the process may be ended, for example, when a user performs an end operation, or when S701 to S704 have been executed a predetermined number of times or for a predetermined period of time. Also, for example, during actual shooting, sound information may be transmitted from the microphone 106 to the information processing device 100 for each cut (at the end of each cut), and one loop of S701 to S704 may be started upon receipt of such data.

[0038] Hereinafter, two possible patterns, pattern 1 and pattern 2, will be described for the processing according to FIG.

[0039] [Pattern 1: When processing is performed at the timing of a scene change] Pattern 1 is a case where the second acquisition unit 303 acquires sound information for one cut (up to the timing of switching to the cut of the next cut ID) in S701. That is, here, sound information for one section of such a cut is acquired using a cut that is a section that has been set in advance in the rehearsal shooting and the actual shooting. In pattern 1, it is assumed that sound information is acquired up to the end timing of cut ID (i) (the timing of switching to cut ID (i+1)).

[0040] In the example of FIG. 8, sound information from the actual shooting up to the end timing of cut ID (1) has been acquired, and data has been recorded up to the row of cut ID (1). The actual tempo (BPM) is the tempo calculated for the time within that cut. The actual tempo may be calculated, for example, based on the tempo value from the rehearsal shooting and using the ratio of the time required to execute the same sound during the actual shooting. Similar to the sound data recorded in FIG. 6, the sound data is data that expresses the changes in sound volume and pitch over time as digital signals. This data is acquired as second sound information 304 in S701 in pattern 1.

[0041] In S702 of pattern 2, the determination unit 305 calculates the rate of change in tempo between the rehearsal and the actual performance by comparing the tempo during the rehearsal shooting with the actual tempo during the actual shooting. Here, an example will be described in which the determination unit 305 determines the camerawork during the actual shooting for cut ID (2) by correcting the camerawork during the rehearsal shooting for cut ID (2) based on the rate of change in tempo for cut ID (1).

[0042] The tempo change rate for cut ID (i) can be calculated by dividing the actual tempo of cut ID (i) by the tempo of cut ID (i) during the rehearsal. For example, the tempo change rate for cut ID (1) is 75 ÷ 50 = 1.5. This indicates that at the end of cut ID (1) in the actual performance, the performance is being performed at a tempo 1.5 times faster than in the rehearsal.

[0043] In S703, the determination unit 305 determines the camerawork based on the rate of change calculated in S702. Here, the PTZ movement speed for cut ID (2) during the rehearsal shooting is multiplied by the rate of change to calculate the PTZ movement speed for cut ID (2) during the actual shooting. Through this processing, the camera movement speed during the rehearsal shooting can be changed to a speed that matches the tempo of the actual shoot. Specifically, the following calculation is performed. Movement speed of cut ID (i+1) during actual shooting = Movement speed of cut ID (i+1) during rehearsal shooting × Tempo change rate of cut ID (i) Movement speed (P) of cut ID (2) during actual shooting = 4 × 1.5 = 6 The movement speed (T) of cut ID (2) during actual shooting = 2 × 1.5 = 3 The movement speed (Z) of cut ID (2) during actual shooting = -0.2 × 1.5 = -0.3

[0044] In S704, the communication unit 308 issues a control instruction to the camera control unit 226 of the camera with the camera ID corresponding to the cut ID, via the communication units 213 and 223, so that control based on the camerawork determined in S703 is performed. Since it is now the timing to switch cuts, the determination unit 305 instructs the video distribution unit 306 on the camera ID that will capture the image at the current timing.

[0045] In the description of pattern 1, an example was given in which the section with cut ID (1) was used as the first audio section and the section with cut ID (2) was used as the second audio section. However, the first and second audio sections may be sections previously set as cuts, or a section corresponding to one set of processing for which sound information was acquired may be used as the (first / second) cut. The division method is not particularly limited as long as a specific audio section is extracted based on a digital sound signal and associated with the rehearsal shooting and the actual shooting. For example, if sound information is acquired up to the middle of a section with cut ID 1, the partial section of the entire section with that cut ID up to the point where sound information was acquired within that section may be set as the first audio section, and the partial section from the point where sound information was acquired to the end of the section with that cut ID may be set as the second audio section. An example of acquiring sound information up to the middle of such a cut will be described below with reference to pattern 2.

[0046] [Pattern 2: When processing is performed during cutting] Pattern 2 is a case where the second acquisition unit 303 acquires sound information up to the middle of one cut in S701. Fig. 9 is a diagram showing an example of second information 304 acquired in such pattern 2. The items included in the sound information shown in Fig. 9 are the same as those in Fig. 8.

[0047] In the example of FIG. 9, sound information is acquired up to the middle of cut ID (2) (the timing when the Fa-So notes are executed out of the Fa-So-La notes). In pattern 2, the tempo change rate is calculated as in pattern 1, and the subsequent camerawork is determined. Here, because sound information is acquired up to the middle of the cut, the camerawork for the remaining part of the cut is determined based on the change rate of the acquired range. Here, the provisional value of the actual tempo for cut ID (2) (the provisional value up to the So note) is calculated as 100. Specifically, if it took 4 seconds to execute the Fa-So note data in the rehearsal and 2 seconds to execute the same notes in the actual performance, since the BPM for cut ID (2) during the rehearsal was 50, the BPM for the actual performance will be 50 × 4 ÷ 2 = 100. Note that while the actual tempo is calculated using a ratio here, any method for evaluating the tempo of audio, such as calculating the number of beats per minute (number of quarter notes) by using musical score data in combination, may also be used.

[0048] In the example of pattern 2 shown in FIG. 9, the following calculations are specifically performed. The movement speed (P) of the (remaining) cut ID (2) during actual shooting = 4 × 2 = 8 The movement speed (T) of the (remaining) cut ID (2) during actual shooting = 2 × 2 = 4 The movement speed (Z) of the (remaining) cut ID (2) during actual shooting = -0.2 × 2 = -0.4

[0049] According to this processing, camerawork during the rehearsal shooting, sound information during the rehearsal shooting, and sound information during the actual shooting are acquired, and camerawork during the actual shooting can be determined based on these. In particular, by calculating the rate of change based on the sound information during the rehearsal shooting and the sound information during the actual shooting, and correcting the camerawork during the rehearsal shooting based on this rate of change, camerawork during the actual shooting can be determined. Therefore, it is possible to more appropriately control camerawork based on the change during the actual shooting compared to the rehearsal.

[0050] Note that, although the explanation has been given here assuming that the PTZ control speed is used as the camera control content (movement speed) included in the camerawork, this is not particularly limited as long as it is a camera control amount. For example, when capturing images while moving (for example, sliding) a camera whose position can be moved, the same processing may be performed on the movement speed calculated for the movement of such a camera.

[0051] [Embodiment 2] In the first embodiment, an example was described in which tempo is used as sound information and camera work is determined based on the rate of change of the tempo. The information processing device 100 according to this embodiment determines camera work during actual shooting using volume as sound information. In particular, an example will be described in which camera work during actual shooting is determined based on changes in volume relative to a reference volume due to excitement in the audience seats.

[0052] Fig. 10 is a schematic diagram showing the configuration of a control system according to embodiment 2. The components shown in Fig. 10 are the same as those shown in Fig. 1, except that a microphone 1001 is added. The microphone 1001 has a similar configuration to the microphone 106 and is capable of measuring sound volume.

[0053] FIG. 11 is a block diagram showing an example of the hardware configuration of the information processing device 100, cameras 109, 110, and 111, and microphone 106 according to this embodiment. FIG. 11 is the same as that shown in FIG. 2 except that a microphone 1001 having a configuration similar to that of the microphone 106 is added. FIG. 12 is a block diagram showing an example of the functional configuration of the information processing device 100, cameras 109, 110, and 111, and microphones 106 and 1001 constituting a control system, and their logical connection. FIG. 12 is the same as that shown in FIG. 3 except that a microphone 1001 having a configuration similar to that of the microphone 106 is added.

[0054] The information processing device 100 according to this embodiment has the same configuration as the information processing device 100 according to the first embodiment and can execute the same processes, so a duplicated description will be omitted. Below, differences between the information processing device 100 according to this embodiment and the first embodiment will be described.

[0055] In S401, the first acquisition unit 301 according to this embodiment acquires camerawork and sound information during rehearsal filming. Here, the camerawork includes the starting position and movement speed shown in FIG. 6 as well as the amplitude of the camera shake operation and the shaking period of the operation. FIG. 13 is a diagram showing an example of the first information 302 acquired according to this embodiment, including such camerawork. In FIG. 13, the amplitude (based on 50 dB) of the periodic camera shake operation and the shaking period (per second) (for the operation) are recorded as information included in the camerawork for each cut ID. Here, the amplitude of the camera operation (based on 50 dB) is the amplitude of the camera shake when moving the camera. Here, 50 dB is set as the standard volume (volume set during a rehearsal without an audience). The amplitude can be set for each PTZ control, for example, Z: ±0.5.

[0056] The camera shaking cycle indicates how many times per second a single shake of the set shake amplitude will be executed. For example, if the shake amplitude Z is set to ±0.5 and the shaking cycle is set to 3 ( / sec), Z will shake 3 times per second from the current position: 0 → 0.5 → 0 → -0.5 → 0.

[0057] In this example, the reference volume is assumed to be 50 dB across all cuts, but the reference volume may be different for each cut. For example, the reference volume may be set for each cut based on the volume of sound collected for each cut during rehearsal filming.

[0058] Fig. 14 is a flowchart showing an example of processing performed by the information processing device 100 according to this embodiment during actual shooting after rehearsal shooting. The processing shown in Fig. 14 is performed in the same way as that shown in Fig. 7, except that S1401 to S1403 are performed instead of S701 to S703, so duplicated explanations will be omitted.

[0059] In S1401, the second acquisition unit 303 acquires sound information (second information 304) during actual image capture. The second information 304 acquired here is the same information as that acquired in S701 of the first embodiment, except that it additionally includes information indicating the volume during actual image capture.

[0060] In S1402, the determination unit 305 calculates the rate of change in tempo between the rehearsal and the actual performance, and the rate of change in volume in the actual performance relative to the reference (hereinafter, this may be simply referred to as the "rate of change in volume"), based on the first information 302 and the second information 304. The rate of change in tempo is calculated in the same manner as in S702. The rate of change in volume in the actual performance relative to the reference is calculated here as the ratio of the volume during actual shooting acquired in S1401 to a volume that is set in advance as the reference (here, shown as 50 dB in the first information 302). Specific processing examples in S1402 will be described in Pattern 1 and Pattern 2 below.

[0061] In S1403, the determination unit 305 determines camerawork based on the rate of change of tempo calculated in S1402 and the rate of change of volume from the reference volume to the actual performance. Here, the camera movement speed during actual shooting based on the rate of change of tempo is determined in the same manner as in S703. The determination unit 305 can also determine camerawork during actual shooting by correcting the camerawork used during rehearsal shooting based on the rate of change of volume. Here, the determination unit 305 determines the amplitude and swing period of camera operation during actual shooting, which are included in the camerawork, by correcting the amplitude and swing period of camera operation during rehearsal shooting based on the rate of change of volume. The subsequent processes S704 and S705 are performed in the same manner as in FIG. 7.

[0062] Hereinafter, two possible patterns, pattern 1 and pattern 2, will be described for the processing according to FIG.

[0063] [Pattern 1: When processing is performed at the timing of a scene change] Pattern 1 is a case similar to pattern 1 according to embodiment 1. Fig. 15 is a diagram showing an example of sound information (second information 304) acquired by the second acquisition unit 303. In the example of Fig. 15, sound information up to the end timing of cut ID (1) in the actual shooting is collected by the microphone 106.

[0064] In S1401 of pattern 1, the actual volume (of the cheering microphone) is acquired via microphone 1001 as information included in second information 304. This actual volume is assumed to be the volume of the sound from the audience seats acquired by microphone 1001. Note that this actual volume may be, for example, the average or maximum volume value during the sound collection period, or any other statistical value may be used.

[0065] In S1402 of pattern 1, the determination unit 305 calculates the rate of change in the volume of the actual performance relative to the reference volume by comparing the reference volume with the actual volume obtained in S1401. Here, an example will be described in which the determination unit 305 determines the camera work to be used during the actual shooting of cut ID (2) by correcting the camera work used during the rehearsal shooting of cut ID (2) based on the rate of change in the volume of cut ID (1).

[0066] The volume change rate for cut ID (i) can be calculated by dividing the actual volume of cut ID (i) by the reference volume. For example, the volume change rate for cut ID (1) is 75 ÷ 50 = 1.5. This indicates that at the end of cut ID (1) in the actual performance, the performance is being performed at a volume 1.5 times the expected reference volume.

[0067] In S1403, the determination unit 305 determines the camerawork based on the change rates calculated in S702. Here, the shake amplitude and shake cycle of the camera operation for cut ID (2) during the rehearsal shooting are multiplied by the change rate of the volume, respectively, to calculate the shake amplitude and shake cycle of the camera operation for cut ID (2) during the actual shooting. Through this processing, the shake amplitude and shake cycle of the camera operation during the rehearsal shooting can be changed to match the volume of the actual performance (i.e., the excitement expected from the volume of cheers, etc.). Specifically, the following calculations are performed. The fluctuation range of the camera operation for cut ID (i+1) during the actual shooting = The fluctuation range of cut ID (i+1) during the rehearsal shooting × The volume change rate of cut ID (i) Camera operation shaking cycle for cut ID (i+1) during actual shooting = Shaking cycle for cut ID (i+1) during rehearsal shooting × Volume change rate for cut ID (i) When we perform specific calculations using the values ​​in Figures 13 and 15, Shake range of camera operation for cut ID (2) during actual shooting = ±0.5 × 1.5 = ±0.75 The camera shake cycle for cut ID (2) during actual shooting = 3 x 1.5 = 4.5

[0068] [Pattern 2: When processing is performed during cutting] Pattern 1 is a case similar to pattern 1 according to embodiment 1. Fig. 16 is a diagram showing an example of second information 304 acquired in such pattern 2. The items included in the sound information shown in Fig. 16 are the same as those in Fig. 8.

[0069] In the example of FIG. 16, sound information has been acquired up to the middle of cut ID (2) (the timing when the notes Fa-So are executed out of Fa-So-La). In pattern 2, the rate of change in volume is calculated as in pattern 1, and the subsequent camera work is determined. Here, sound information has been acquired up to the middle of the cut, so the camera work for the remaining part of the cut is determined based on the rate of change in the acquired range. Here, the provisional value of the actual volume for cut ID (2) (provisional value up to the note So) is calculated as 100. The rate of change in tempo, etc. is the same as in the example of FIG. 9. Here, the reference volume for cut ID (2) during rehearsal filming is 50 dB, and the volume up to the note So in cut ID (2) during actual filming is 100 dB, so it is conceivable that the volume is twice as loud as expected.

[0070] In the example of pattern 2 shown in FIG. 16, the following calculations are specifically performed. The camera operation fluctuation range for the (remaining) cut ID (2) during actual shooting = ±0.5 × 2 = ±1.0 The camera operation shaking cycle for the (remaining) cut ID (2) during actual shooting = 3 x 2 = 6

[0071] This type of processing allows camerawork for the actual shoot to be determined based on a reference volume and the volume during the actual shoot. In particular, by calculating the rate of change between the reference volume and the volume during the actual shoot and correcting the amplitude and period of camera shake during the rehearsal shoot based on this rate of change, camerawork for the actual shoot can be determined. Therefore, it is possible to estimate how loud the volume has become due to cheers, etc., i.e., how excited the audience is, based on the reference volume, and to control the amount of camera movement in accordance with this excitement.

[0072] The disclosure of this specification includes the following information processing device, information processing method, and program. (Item 1) a first acquisition means for acquiring first camera work that is camera work during rehearsal shooting and first sound information that is sound information during the rehearsal shooting; a second acquisition means for acquiring second sound information, which is sound information during actual shooting relative to the rehearsal shooting; a determining means for determining a second camerawork, which is a camerawork to be used during the actual shooting, based on the first camerawork, the first sound information, and the second sound information; An information processing device comprising: (Item 2) a first calculation means for calculating a rate of change of the tempo of the second sound information relative to the tempo of the first sound information; The information processing device described in item 1, characterized in that the determination means determines the second camera work by correcting the first camera work based on the rate of change calculated by the first calculation means. (Item 3) 3. The information processing device according to item 1 or 2, wherein the second camera work includes PTZ control content of a camera that captures images. (Item 4) the first sound information is sound information in a first audio section during the rehearsal imaging, and the second sound information is sound information in the first audio section during the actual imaging, The information processing device described in any one of items 1 to 3 is characterized in that the determination means determines the second camera work in the second audio section during the actual shooting by correcting the first camera work in the second audio section that chronologically follows the first audio section during the rehearsal shooting based on the first sound information and the second sound information. (Item 5) 5. The information processing device according to item 4, wherein the first audio section and the second audio section are sections that are set in advance during the rehearsal imaging. (Item 6) The information processing device described in item 4 or 5, characterized in that the second acquisition means acquires, as the first sound information, sound information up to the sound at the end of the first audio section, which was set in advance during the rehearsal imaging. (Item 7) Item 5. The information processing device according to item 4, wherein the first audio section and the second audio section are each a partial section of a section that is set in advance during the rehearsal imaging. (Item 8) The information processing device described in any one of items 4 to 7, characterized in that the first audio section and the second audio section are sections extracted in correspondence with each other based on digital audio signals during rehearsal shooting and actual shooting. (Item 9) 9. The information processing device according to any one of items 1 to 8, wherein the second camera work includes a shaking amplitude and a shaking period of an operation for periodically shaking the camera during the actual shooting. (Item 10) the first camerawork includes information on a shaking amplitude and a shaking period of an operation of periodically shaking a camera during the rehearsal imaging; the second sound information includes a volume during the actual shooting, a setting means for setting a reference volume; and a second calculation means for calculating a rate of change of the volume during the actual shooting relative to the reference volume, The information processing device described in item 9 is characterized in that the determination means determines the shaking amplitude and shaking period of the operation of periodically shaking the camera during the actual shooting, which is included in the second camerawork, by correcting the shaking amplitude and shaking period based on the change rate calculated by the second calculation means. (Item 11) Item 11. The information processing device according to item 10, wherein the volume during the actual shooting is the volume of cheers. (Item 12) The information processing device described in any one of items 1 to 10 is further characterized in that the determination means determines the second camerawork so that, in a section of the actual shooting that cannot be matched based on the digital sound signals between the rehearsal shooting and the actual shooting, the camera performs fixed tracking or is controlled to perform fixed control. (Item 13) acquiring first camera work that is camera work during rehearsal shooting and first sound information that is sound information during the rehearsal shooting; acquiring second sound information which is sound information during actual shooting relative to the rehearsal shooting; determining a second camerawork, which is a camerawork during the actual shooting, based on the first camerawork, the first sound information, and the second sound information; An information processing method comprising: (Item 14) A program for causing a computer to function as each means of the information processing device described in any one of items 1 to 12.

[0073] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0074] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0075] 100: Information processing device, 106: Microphone, 109: Camera, 110: Camera, 111: Camera, 112: Network

Claims

1. a first acquisition means for acquiring first camera work that is camera work during rehearsal imaging and first sound information that is sound information during the rehearsal imaging; a second acquisition means for acquiring second sound information, which is sound information during actual shooting relative to the rehearsal shooting; a determining means for determining a second camerawork, which is a camerawork to be used during the actual shooting, based on the first camerawork, the first sound information, and the second sound information; An information processing device comprising:

2. a first calculation means for calculating a rate of change of the tempo of the second sound information relative to the tempo of the first sound information; The information processing device according to claim 1 , wherein the determining means determines the second camera work by correcting the first camera work based on the rate of change calculated by the first calculating means.

3. 2. The information processing apparatus according to claim 1, wherein the second camerawork includes PTZ control content of a camera that captures images.

4. the first sound information is sound information in a first sound section during the rehearsal imaging, and the second sound information is sound information in the first sound section during the actual imaging, 2. The information processing device according to claim 1, wherein the determination means determines the second camera work in the second audio section during the actual shooting by correcting the first camera work in the second audio section that chronologically follows the first audio section during the rehearsal shooting based on the first sound information and the second sound information.

5. The information processing device according to claim 4 , wherein the first audio section and the second audio section are sections that are set in advance during the rehearsal imaging.

6. 5. The information processing device according to claim 4, wherein the second acquisition means acquires, as the first sound information, sound information up to the sound at the end of the first sound section, which is set in advance during the rehearsal imaging.

7. The information processing device according to claim 4 , wherein the first audio section and the second audio section are each a partial section of a section that is set in advance during the rehearsal imaging.

8. 5. The information processing device according to claim 4, wherein the first audio section and the second audio section are sections extracted in association with each other based on digital audio signals during rehearsal filming and actual filming, respectively.

9. The information processing device according to claim 1 , wherein the second camerawork includes a shaking width and a shaking period of an operation for periodically shaking the camera during the actual image capture.

10. the first camerawork includes information on a shaking amplitude and a shaking period of an operation of periodically shaking a camera during the rehearsal imaging; the second sound information includes a volume during the actual image capture, a setting means for setting a reference volume; a second calculation means for calculating a rate of change of the volume during the actual shooting with respect to the reference volume, The information processing device described in claim 9, characterized in that the determination means determines the shaking amplitude and shaking period of the operation of periodically shaking the camera during the actual shooting, which is included in the second camerawork, by correcting the shaking amplitude and shaking period based on the change rate calculated by the second calculation means.

11. The information processing device according to claim 10, wherein the volume during the actual image capture is the volume of cheers.

12. 2. The information processing device according to claim 1, further characterized in that the determining means determines the second camerawork so as to cause a camera to perform fixed tracking or fixed control in a section of the actual shooting that cannot be matched based on the digital sound signals between the rehearsal shooting and the actual shooting.

13. acquiring first camera work that is camera work during rehearsal imaging and first sound information that is sound information during the rehearsal imaging; acquiring second sound information, which is sound information during actual shooting relative to the rehearsal shooting; determining a second camerawork, which is a camerawork for the actual shooting, based on the first camerawork, the first sound information, and the second sound information; An information processing method comprising:

14. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 12.

Images