Control device, control method thereof, program, and recording medium

Abstract

To provide an electronic device with which a user can suitably select sound to be used at the time of shooting a moving image.SOLUTION: A control device (smartphone 100) capable of communicating with a plurality of sound collecting devices (microphones 312 to 316) includes display means, image acquisition means for acquiring a captured image from an imaging device, position acquisition means for acquiring position information of the plurality of sound collecting devices with respect to the captured image, display control means for controlling the display means to display display items corresponding to the plurality of sound collecting devices together with the image acquired by the image acquisition means on the basis of the position information acquired by the position acquisition means, and control means for displaying a display item corresponding to a sound collecting device that is not included in the range of the image on the display means, acquiring audio information from the sound collecting device selected by a user, and controlling to associate the image with the audio information and record the image and the audio information to a recording device.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a method for acquiring audio data during image capture. [Background technology]

[0002] Conventionally, when shooting a video, there has been a need to collect and record not only the sound near the electronic device with a microphone built into the electronic device having an imaging function, but also the sound near a subject (the subject being filmed) in the direction the camera is facing. Patent Document 1 discloses a method of automatically selecting a microphone near a subject in the direction of the video being filmed by the user, acquiring the sound collection information of that microphone, and recording it in association with the video data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-107971 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Document 1, the imaging device automatically selects the microphone to pick up sound, and it is sometimes impossible to use a microphone built into the imaging device or a microphone located somewhere other than the main subject being imaged by the user depending on the situation. This results in the automatic selection of a microphone contrary to the user's intention, and there are cases where the user is unable to record the sound that he or she desires.

[0005] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to allow a user to suitably select the audio to be used when shooting a moving image. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention The control device of A control device capable of communicating with a plurality of sound collecting devices, comprising: a display unit; an image acquisition means for acquiring an image captured by an imaging device from the imaging device; a position acquisition means for acquiring position information of the plurality of sound collecting devices; a display control means for controlling the display unit to display a plurality of display items corresponding to the plurality of sound collecting devices respectively together with the image acquired by the image acquisition means based on the position information acquired by the position acquisition means; a selection means for selecting a display item from the plurality of display items displayed on the display unit by a user operation; and a control means for acquiring audio information from the sound collecting device corresponding to the display item selected by the selection means, and recording the image and the audio information in a recording device in association with each other, wherein the display control means controls the display unit to display, for a sound collecting device located outside the range of the shooting angle of view of the image, a directional item indicating the direction of the position of the sound collecting device located outside the range of the shooting angle of view of the image as the display item corresponding to the sound collecting device. Effect of the Invention

[0007] According to the present invention, the user can suitably select the audio to be used when shooting a moving image. [Brief description of the drawings]

[0008] [Figure 1] 1 is an external view of a smartphone 100 according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram showing a configuration of a smartphone 100 according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram showing the positional relationship between the smartphone 100 and a sound collection device (microphone). [Figure 4] FIG. 13 is a diagram showing an example of a selection screen for a sound collection device (microphone) displayed on a display 104. [Diagram 5] 13 is an example of a display of the status of a sound collection device (microphone). [Figure 6] FIG. 11 is a control flowchart relating to selection of a sound collection device (microphone). [Figure 7] FIG. 13 is a diagram showing another display example of the screen for selecting a sound collection device (microphone). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0010] 1(a) and (b) show external views of a smartphone 100 as an example of a device (electronic device) to which the present invention can be applied. FIG. 1(a) is a front view of the smartphone 100, and FIG. 1(b) is a rear view of the smartphone 100. In FIG. 1, a display 104 is a display unit provided on the front of the smartphone for displaying images and various information. A touch panel 106a is a touch operation member, and can detect a touch operation on the display surface (operation surface) of the display 104 (touch detectable). The smartphone 100 can display a live view image (LV image) captured by an in-camera 116, an out-camera 117, or both, on the display 104.

[0011] The power button 106b is an operating member included in the operation unit 106, and can switch on / off the display 104. By continuing to press (long press) the power button 106b for a certain period of time, for example, three seconds, the power of the smartphone 100 can be switched on / off.

[0012] The volume plus button 106c and the volume minus button 106d are volume buttons that control the volume of the sound output from the audio output unit 112. Pressing the volume plus button 106c increases the volume, and pressing the volume minus button 106d decreases the volume. In addition, when the camera is in a recording standby state while in use, pressing the volume plus button 106c or the volume minus button 106d also functions as a shutter button or a REC button that instructs shooting of still images or starting / stopping recording of videos. The user can also set it as desired so that a specific function is executed when the power button 106b and the volume minus button 106d are pressed simultaneously, or when the volume minus button 106d is pressed quickly several times.

[0013] The home button 106e is an operation button for displaying a home screen, which is a startup screen of the smartphone 100, on the display 104. When various applications have been started and used on the smartphone 100, pressing the home button 106e can temporarily close the various running applications and display the home screen. Note that the home button 106e is assumed to be a physically pressable button, but it may be a touchable button displayed on the display 104 and having a similar function, rather than a physical button. Also, a function similar to that of the home button 106e may be executed by performing a specific gesture on the touch panel 106a.

[0014] Audio output terminal 112a is an earphone jack, and is a terminal for outputting audio to earphones, an external speaker, etc. Sub-speaker 112b and main speaker 112c are speakers built into the main body that output audio. When a terminal for outputting audio to audio output terminal 112a, for example, an earphone cord is not attached, and it is assumed that the user holds smartphone 100 to the user's ear, such as during a voice call, audio is output from sub-speaker 112b. In other situations, for example, when the user starts playing a video without attaching earphones, audio is output from main speaker 112c.

[0015] The sound collection unit 113 is a microphone built into the smartphone 100, and collects sounds around the smartphone 100. For example, it can collect the voice of a user when the user makes a call using the smartphone 100, or it can associate the sound collected by the sound collection unit 113 with a video recorded using the in-camera 116 or the out-camera 117. In FIG. 1(a) of this embodiment, the smartphone 100 has one built-in sound collection unit 113, but it may have multiple built-in sound collection units.

[0016] 2 is a block diagram showing an example of the configuration of the smartphone 100 according to the present embodiment. A CPU 101, a memory 102, a non-volatile memory 103, a display image processing unit 105, a display 104, an operation unit 106, a storage medium I / F 107, an external I / F 109, and a communication I / F 110 are connected to an internal bus 150. In addition, a sound output unit 112, a sound collection unit 113, a posture detection unit 114, a GPS 115, an in-camera 116, an out-camera 117, an infrared camera 118, and a camera image processing unit 119 are connected to the internal bus 150. Each unit connected to the internal bus 150 is configured to be able to exchange data with each other via the internal bus 150.

[0017] The CPU 101 is a control unit that controls the entire smartphone 100, and is composed of at least one processor or circuit. The memory 102 is composed of, for example, a RAM (a volatile memory using semiconductor elements, etc.). The CPU 101 controls each unit of the smartphone 100 using the memory 102 as a work memory according to a program stored in, for example, the nonvolatile memory 103. The nonvolatile memory 103 stores image data, audio data, other data, various programs for the operation of the CPU 101, and the like. The nonvolatile memory 103 is composed of, for example, a flash memory, a ROM, etc.

[0018] Display 104 displays images, GUI screens constituting a GUI (Graphical User Interface), and the like under the control of CPU 101. CPU 101 generates a display control signal according to a program, and controls each unit of smartphone 100 to generate a video signal for display on display 104 and output it to display 104. Display 104 displays an image based on the output video signal. Note that the configuration of smartphone 100 itself is limited to an interface for outputting a video signal for display on display 104, and display 104 may be configured as an external monitor (such as a television).

[0019] The display image processing unit 105 analyzes images and GUIs displayed on the display 104. The display image processing unit 105 can also detect a banner display (described later) displayed on the display 104. Like the display 104, the display image processing unit 105 may also be configured to be attached to an external monitor (such as a television).

[0020] The operation unit 106 is an input device for receiving user operations, including a character information input device such as a keyboard, a pointing device such as a mouse or a touch panel, a button, a dial, a joystick, a touch sensor, a touch pad, etc. The touch panel is an input device that is configured in a planar manner by being overlaid on the display 104, and outputs coordinate information according to the touched position. The operation unit 106 includes the above-mentioned touch panel 106a, power button 106b, volume plus button 106c, volume minus button 106d, and home button 106e.

[0021] The recording medium I / F 107 allows a recording medium 108 such as a memory card, CD, or DVD to be attached, and reads data from the attached recording medium 108 and writes data to the recording medium 108 based on the control of the CPU 101. The recording medium 108 may be an internal storage incorporated in the smartphone 100.

[0022] The external I / F 109 is an interface for connecting to an external device via a wired cable or wirelessly, and for inputting and outputting video signals and audio signals.

[0023] The communication I / F 110 is connected wirelessly or via a wired cable, and transmits and receives video signals and audio signals to and from the sound collection device 150. The communication I / F 110 can also be connected to a wireless LAN (Local Area Network) or the Internet 111. The communication I / F 110 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication I / F 110 can transmit video images (including live images) captured by the in-camera 116 or the out-camera 117, or both, and images recorded in the recording medium 107, to a cloud storage 250 or the like. The communication I / F 110 can also receive image data and various other information from external devices.

[0024] Internet 111 refers to wireless LAN communications such as Wifi and 4G / 5G communications.

[0025] The audio output unit 112 performs noise reduction processing and amplification processing on the audio data input from the sound collection unit 113. In addition, the audio of video and music data, operation sounds, ringtones, various notification sounds, etc. are output via an audio output terminal 112a for connecting earphones, etc., speakers 112b and 112c, wireless communication, etc.

[0026] The attitude detection unit 114 detects the attitude of the smartphone 100 with respect to the direction of gravity and the inclination of the attitude with respect to each axis of yaw, roll, and pitch. Based on the attitude detected by the attitude detection unit 114, it is possible to determine whether the smartphone 100 is held horizontally, vertically, facing up, facing down, or at an angle. This makes it possible to detect the direction in which the in-camera 116 or the out-camera 117 is facing. As the attitude detection unit 114, at least one of an acceleration sensor, a gyro sensor, a geomagnetic sensor, a direction sensor, an altitude sensor, and the like can be used, and a combination of two or more sensors can also be used.

[0027] The GPS 115 is a system that uses artificial satellites to measure geographical information. A signal is sent to an artificial satellite, and the geographical position is determined by the time difference until the response is received. Hereinafter, this position is called the shooting position information. The shooting position information is longitude, latitude, and time information.

[0028] The front camera 116 is a camera arranged on the same surface of the housing of the smartphone 100 as the display 104 .

[0029] The rear camera 117 is a camera arranged on the side of the housing of the smartphone 100 opposite the display 104. Although only one camera is depicted for each of the front camera 116 and rear camera 117 in Fig. 1(a) and (b), this is not limited thereto. In other words, multiple cameras with different focal lengths may be mounted.

[0030] Live view images captured by the in-camera 116, the out-camera 117, or both can be displayed on the display 104. Depending on whether the in-camera 116 or the out-camera 117 is used, it is possible to select whether to capture a scene in front of the user or to capture a self-portrait of the user. When both the in-camera 116 and the out-camera 117 are selected, LV images captured by both cameras are displayed on the display 104.

[0031] The infrared camera 118 is a camera arranged on the same surface of the housing of the smartphone 100 as the display 104, and is arranged on the opposite side of the front camera 116 with the speaker 112b at the center. The infrared camera 118 is used to analyze data of the user's face to create a depth map and an infrared image of the face. This can also be used to change the smartphone 100 from a locked state to an unlocked state.

[0032] Camera image processing unit 119 performs various image processing and recognizes displayed items on the screen displayed on display 104 under the control of CPU 101. The image processing unit can also perform various image processing on images stored in non-volatile memory 103 or recording medium 108, video signals acquired via external I / F 109, images acquired via communication I / F 110, and the like. A finger or pen that has not been touching the touch panel 106a touches the touch panel 106a again, that is, the start of touching (hereinafter referred to as touch-down). The touch panel 106a is in a state where a finger or a pen is touching the touch panel 106a (hereinafter referred to as Touch-On). A finger or a pen is moved while touching the touch panel 106a (hereinafter referred to as Touch-Move). The finger or pen that has been touching the touch panel 106a is removed from the touch panel 106a, that is, the touch ends (hereinafter referred to as "touch-up"). A state in which nothing is touching the touch panel 106a (hereinafter referred to as Touch-Off) When touch-down is detected, touch-on is also detected at the same time. After touch-down, touch-on will usually continue to be detected unless touch-up is detected. If touch-move is detected, touch-on is also detected at the same time. Even if touch-on is detected, touch-move will not be detected if the touch position does not move. When it is detected that all fingers or pens that were touching have touched up, touch-off is detected.

[0033] These operation states and the position coordinates of the touch panel 106a touched by a finger or pen are notified to the CPU 101 via an internal bus. The CPU 101 determines what kind of operation (touch operation) has been performed on the touch panel 106a based on the notified information. Regarding touch-move, the moving direction of the finger or pen moving on the touch panel 106a can also be determined for each vertical and horizontal component on the touch panel 106a based on the change in the position coordinates. If a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 106a, quickly moved a certain distance, and then released is called a flick.

[0034] In other words, a flick is a movement operation in which a finger is quickly traced over the touch panel 106a as if flicking it. When a touch-move is detected over a predetermined distance or more at a predetermined speed or more, and then a touch-up is detected, it can be determined that a flick has been performed (a slide operation followed by a flick can be determined).

[0035] Furthermore, a touch operation in which multiple points (for example, two points) are touched simultaneously and the touch positions are brought closer together is called a pinch in, and a touch operation in which the touch positions are moved away from each other is called a pinch out. Pinch out and pinch in are collectively called a pinch operation (or simply pinch). The touch panel 106a may be any of various touch panels, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type.

[0036] There are methods for detecting a touch by contact with the touch panel, and methods for detecting a touch by the approach of a finger or pen to the touch panel, and either method may be used.

[0037] The sound collection device 150 is a device connected to the Internet that can communicate with the smartphone 100 via the communication I / F 110, and includes a sound collection unit 151, a GPS 152, and a sound processing unit 153. When sound data collected by the sound collection unit 151 is selected by a user operation, the sound data and location information acquired by the GPS 152 can be acquired via the communication I / F 110 and saved in association with a video captured by the outer camera 117 or the like.

[0038] Cloud storage 250 can exchange information such as image data with smartphone 100 via communication I / F 110. For example, when a user is taking a video using rear camera 117, it is also possible to save the video in real time in cloud storage 250 instead of saving the video in recording medium 108 built into smartphone 100. At this time, as described above, audio information and position information acquired by sound collection unit 113 or sound collection device 150 are associated with the video and saved in cloud storage 250.

[0039] A display example of sound collection devices (microphones) when shooting a moving image using the smartphone 100 of this embodiment and a method for selecting the sound collection device will be described with reference to FIGS. 3 to 7. FIG.

[0040] FIG. 3 is a diagram showing an example of the positional relationship between a subject in an LV image and a plurality of selectable sound collection devices (microphones) in this embodiment. FIG. 3 shows a scene in which the smartphone 100 is held by the user 302 and is held to shoot a video with the out-camera 117. A range 301 indicates the range of the shooting angle of view captured by the out-camera 117 of the smartphone 100. Subjects 303 to 305 are subjects that exist on the opposite side of the smartphone 100 from the user 302 and on the side where the out-camera 117 is placed, with the smartphone 100 as a reference. The subject 303 is a subject (dog) that the user 302 is trying to shoot as a main subject. The subject 304 is a subject (small bird) that is reflected behind the subject 303, which is the main subject. The subject 305 is a subject (chicken) that is on the right side in the shooting direction, outside the range of the shooting angle of view (range 301) of the out-camera 117.

[0041] Microphone 312 is a microphone capable of collecting sound via sound collecting unit 113 built into smartphone 100, and can collect sounds around smartphone 100, i.e., user 302. The devices capable of collecting sound represented by microphones 313 to 316 are wireless microphones or the like that are connected to the Internet, and when a user connects to the microphones via a device that can connect to the Internet, such as smartphone 100, he or she can freely obtain sounds.

[0042] The microphone 313 is a wireless microphone located near the subject 303 and can collect sounds around the subject 303. The microphone 314 is a wireless microphone located near the subject 304 and can collect sounds around the subject 304. The microphone 315 is a sound collecting device that can collect sounds around the subject 305. The microphone 316 is a wireless microphone located outside the range 301. There is no subject nearby, but it can be selected arbitrarily. The microphones 313 to 316 represent microphones that can be selected by the user, and by communicating via the communication I / F 110 of the smartphone 100, audio information collected by each microphone can be acquired and stored in the recording medium 108 or the cloud storage 250. Two-way communication is possible with the microphones 313 to 316 via the communication I / F 110.

[0043] Not only one microphone 313-316 but multiple microphones can be selected. When multiple microphones are selected, the system control unit 50 can automatically determine the proportion of the audio to be used by analyzing the acquired audio, or the user can arbitrarily determine the proportion of the audio. Note that the microphones 313-316 only indicate wireless microphones that the smartphone 100 has detected via the Internet, and the display method is not limited to this. Instead of an icon shaped like a microphone, an icon shaped like a speaker may also be used. The microphones 312-316 are displayed in the same way regardless of the method of connection to the smartphone 100, but may be changed.

[0044] FIG. 4 is an example in which a sound collecting device (hereinafter, referred to as a wireless microphone) capable of collecting sound via the Internet is superimposed on an LV image captured by the out-camera 117 and displayed on the display 104 in the state shown in FIG. 3. An LV 400 of the range 301 is displayed on the display 104. A display item 402 indicates that the microphone is built into the smartphone 100 (built-in sound collecting item). A display item 403 indicates that the microphone 313 is currently selected by the user and that audio information collected by the microphone 313 is being used. Display items 405 and 406 indicate that the microphones 315 and 316 are outside the range 301, which is the shooting angle of view. Both display items 405 and 406 inform the user that they are on the right side outside the range 310, and the size of the arrow indicates how far they are from the range 310. In FIG. 4, the arrow of the display item 406 is smaller (shorter) than that of the display item 405, so it can be seen that the microphone 316 is closer to the range 301 than the microphone 315. Display items 405 and 406 allow the user to visually recognize sound collection devices that exist outside the shooting angle of view, and therefore the user can select the desired sound from among a larger number of sound collection devices. In the state shown in FIG. 4, the user can select / change the target microphone from which sound information to be associated with and saved in association with the video is acquired by selecting one of microphones 312-316 by touch operation. Display item 407 is a level meter display showing the sound collection information of the currently selected microphone, and when sound collection information from one microphone is used, the upper row shows the L side and the lower row shows the R side. When sound collection information from multiple microphones is used, the upper and lower rows of display item 407 show level meters of sound collection information from different microphones.

[0045] FIG. 5 shows an example of a display form according to the state of the microphone. For a selected (selected) microphone, a display showing a wave motion as if sound is coming from a microphone-shaped icon is performed on the microphone icon. This indicates that sound is being collected from the microphone. For a microphone that is not selected (not in a selected state), no wave-like display is performed. Before an instruction to shoot a video (described later) is given in S620 of FIG. 6 (i.e., when video shooting is not being performed), the sound picked up by the microphone selected by the user is output from one of the audio output units 112. After an instruction to shoot a video is given (i.e., during video shooting), the sound picked up by the microphone selected by the user is associated with the video being shot and saved. At this time, the sound from the selected microphone is not output. Note that the display according to the state of the microphone is not limited to the display form shown in FIG. 5. It is sufficient for the user to visually recognize that the microphone state is different, and the color of the microphone may be changed or an animation in which the microphone icon shakes may be used.

[0046] A selectable microphone is displayed as a microphone icon. By displaying the microphone icon, the user can visually confirm that a microphone (sound collection device) is present at the display position of the icon. If a selectable microphone is outside the currently captured image field (range), it is displayed as being outside the selectable microphone screen area. A directional item such as an arrow indicates which side of the image field the microphone is outside of, from the image field. A microphone in error is selectable if no error occurs, but is a display item that is displayed when, for example, a temporary communication error prevents connection, or when the wireless microphone is not turned on (powered off) due to a dead battery, etc., and sound cannot be collected. In other words, it can be said to be an indication of the sound collection status of the wireless microphone and the connection status with communication. In addition, a microphone in error is displayed even when the microphone cannot be used because it has already been used (selected) by another user.

[0047] The microphone representation is not limited to this. The size of the microphone icon may represent the audio level, or auxiliary information such as a level meter may be added near the microphone to aid in user selection. Although an arrow is used to represent the microphone position outside the field of view, any directional item that indicates a direction may be used. Also, the length or thickness of the arrow may be used to represent the distance, or the color of the arrow may be changed to represent the recommended level based on the sound pickup situation. It is also possible to select multiple wireless microphones. For example, touching a non-selected microphone makes the pickup information the acquisition target, and touching another microphone does not switch the microphone, but by selecting multiple microphones, it is possible to associate multiple pieces of pickup information with a video. Touching a microphone that has already been selected again will deselect the microphone, making it possible to pick up only the desired microphone. To make microphone switching smoother, the pickup information may be switched by cross-fading when performing an operation such as dragging and dropping from one microphone to another.

[0048] FIG. 6 is a flowchart showing the control of the smartphone 100 in this embodiment.

[0049] 6 is a control flowchart of the video shooting mode process that starts when digital camera 100 is started (powered on) and in a video shooting standby state. This control process is realized by system control unit 50 expanding a program stored in non-volatile memory 56 into system memory 52 and executing it.

[0050] In S601, the system control unit 50 processes an image captured through the in-camera 116 or the out-camera 117 using the image processing unit 24, and displays the image on the display 104 as a live view image.

[0051] In S602, the system control unit 50 detects connectable wireless microphones that are present within a predetermined range via the communication I / F 110. At this time, GPS information of the wireless microphones is also acquired.

[0052] In S603, the system control unit 50 inputs the number of wireless microphones detected in S602 into the variable Num, and stores it in the system memory 52.

[0053] In S604, the system control unit 50 determines whether or not the variable Num is 0. If it is 0, the process proceeds to S617, and if it is not 0, the process proceeds to S605.

[0054] In S605, the system control unit 50 initializes a variable i to 0 and stores the variable i in the system memory 52. ​​The variable i will be described later in S606.

[0055] In S606, the system control unit 50 acquires position information of the i-th detected wireless microphone among the wireless microphones detected in S602. The position information acquired in this step is absolute position information.

[0056] In S607, the system control unit 50 acquires position information of the smartphone 100 using the GPS 115. The system control unit 50 calculates the relative position of the i-th wireless microphone with respect to the smartphone 100 from the position information of the smartphone 100, the position information of the wireless microphone acquired in S606, and the direction of the camera currently capturing images. From the relative positional relationship between the smartphone and the microphone calculated in this step, the position of the microphone detected in the LV image can be displayed in S609, which will be described later. In other words, it can be said that the relative position calculated in this step is information corresponding to the position information of the wireless microphone.

[0057] In S608, the system control unit 50 acquires information on the shooting angle of view from the shooting direction of the camera currently capturing images and the specification information of the camera capturing images, and calculates the shooting range. An example of the positional relationship at this time is shown in FIG. 3. The shooting range acquired in S608 is set as range 301, the subjects detected in the live view image are set as subjects 303 to 305, and the wireless microphones detected in S602 are set as microphones 313 to 315. The range shown in range 301 is the range that can be captured by the out-camera 117, and is the range that is displayed on the display 104. In other words, subjects 303 and 304 can be captured, and microphones 313 and 314 are wireless microphones included in the shooting angle of view. Subject 305 is not included in the shooting angle of view. Microphones 315 and 316 are also not included in the shooting angle of view (outside range 301), but the user can arbitrarily select microphones 315 and 316 to acquire audio information, just like microphones 313 and 314.

[0058] Microphone 312 indicates a microphone (sound collection unit 113) built into smartphone 100, and similarly, audio information collected by microphone 312 can be acquired and selected. These microphones 312 to 316 are not limited to using audio information collected from only one microphone, and audio information collected from multiple microphones can also be used. When using multiple pieces of audio information, the user may be allowed to arbitrarily select the ratio of audio to be used from each microphone, or the system control unit 50 may automatically determine the ratio from the multiple pieces of audio information acquired and synthesize them.

[0059] In S609, the system control unit 50 displays display items as shown in Fig. 5 superimposed on the LV image based on the relative position of the wireless microphone calculated in S607 and the shooting range calculated in S608. An example of how this is displayed on the display 104 of the smartphone 100 is shown in Fig. 4. As shown in Fig. 4, a live view image (LV400) of the range indicated by range 301 is displayed on the display 104. In Fig. 4, it can be seen from display item 403 that audio information picked up by microphone 313 is being used / selected.

[0060] In S610, the system control unit 50 refers to the system memory 52 and determines whether the relationship between the variable i and the variable Num is i < Num. If i < Num, it proceeds to S611; otherwise, it proceeds to S612.

[0061] In S611, the system control unit 50 stores the variable i as i + 1 in the system memory 52 and returns to S606.

[0062] In S612, the system control unit 50 displays a display item representing the built-in microphone (sound collection unit 113) of the smartphone 100 on the LV image of the display 104. That is, the display item 402 in FIG. 4 is superimposed and displayed on LV400.

[0063] In S613, the system control unit 50 determines whether the user has selected a microphone different from the currently recording microphone via the operation unit 70. If selected, it proceeds to S614; otherwise, it proceeds to S615.

[0064] In S614, the system control unit 50 changes the icon indicating the wireless microphone selected by the user in S613 to a display form indicating that it is the selected microphone, and controls it to be displayed on the display 104. When the microphone 314 is selected by the user operation from the state shown in FIG. 4, the display item 403 is displayed for the microphone 314.

[0065] In S615, the system control unit 50 determines whether any microphone among the wireless microphones is selected. If selected, it controls the transition to S616; if not selected, it controls the transition to S617.

[0066] In S616, the system control unit 50 acquires the sound collection information of the selected microphone. As described above in the description of the display item 407 in FIG. 4, when video shooting is not being performed, the sound collected by the selected microphone is output from the audio output unit 112, and when video shooting is in progress, it is saved in association with the video without being output.

[0067] In S617, since the result of the determination in S615 is No, the system control unit 50 displays an icon indicating the built-in microphone as a selected display on the display 104. That is, the display item 403 is displayed for the microphone 312 in FIG.

[0068] In S618, the system control unit 50 uses the sound collection unit 113 to obtain sound collection information of the built-in microphone.

[0069] In S619, the system control unit 50 draws an audio level meter based on the sound pickup information acquired in S616 or S618, and displays it on the display 104. For example, like display item 407 in Fig. 4, a level meter for the audio information acquired by the microphone selected by the user is displayed.

[0070] In S620, the system control unit 50 determines whether or not there is an instruction to start shooting a moving image. If there is an instruction to start, the process proceeds to S621, and if there is no instruction, the process proceeds to S625. In this embodiment, it is assumed that there is an instruction to start shooting when the start icon for shooting a moving image displayed on the display 104 is touched on or the volume buttons 106c and 106d are pressed while the LV image is being displayed on the display 104 in a shooting standby state.

[0071] In S621, the system control unit 50 starts video shooting. That is, a video file is created in the recording medium 108 or the cloud storage 250, and the video shot by the in-camera 116 or the out-camera 117 with the current settings is saved.

[0072] In S622, the system control unit 50 associates and stores the audio information together with the video that has been started to be shot in S621. At this time, the video is stored in the recording medium 108 or the cloud storage 250 via the communication I / F 110 according to the user's settings. By storing the video in the cloud storage 250 on the Internet 111, it becomes possible to record a video with a longer recording time than when stored in the recording medium 108, or a video with a large capacity to which a lot of audio information is added. Note that, although the video is stored in the recording medium 108 or the cloud storage 250 here, the video and the associated audio signal may be output via the external I / F 109 and received by an external signal receiving device. In other words, the video may be output to an external recording device (for example, an HDMI (registered trademark) having a recording device) connected by wire / wireless.

[0073] In S623, the system control unit 50 determines whether or not an instruction to stop video recording has been given. If an instruction to stop has been given, the process proceeds to S624, and if not, the process returns to S613. An instruction to stop video recording specifically refers to a touch operation on the video recording stop icon displayed on the display 104, or pressing the volume buttons 106c and 106d or the home button 106e. If the home button 106e is pressed while video recording is not being performed, the screen returns to the home screen and video recording is stopped at the same time.

[0074] In S624, the system control unit 50 stops the video shooting. When the shooting stops, the system control unit 50 performs a process of closing the video file created in the recording medium 108 or the cloud storage 250 (such as adding attribute information).

[0075] In S625, the system control unit 50 determines whether the user has performed an operation to end the video, and if so, ends the process, and if not, controls to transition to S602.

[0076] In this embodiment, a wireless microphone in the vicinity of the smartphone 100 is detected and displayed superimposed on the LV image. The method of acquiring audio data from the microphone linked to the microphone icon by the user selecting the displayed display item (microphone icon) by touch operation or the like and storing the audio data in association with the video has been described, but the present invention is not limited to this. The microphone can be selected by other operation methods such as operation with a physical button such as a cross key in addition to touch. Although an example of selecting a microphone superimposed on the LV image has been shown, a radar display as shown in FIG. 7 may be displayed separately from the LV image and the user may select the microphone. When a microphone near the main subject is selected in shooting, the sound pickup sensitivity of the microphone near the main subject may be changed according to the proportion of the main subject in the shooting angle of view to create a sense of realism. In the case of stereo sound, the balance between L and R may be corrected during sound pickup based on the relationship between the orientation of the camera and the selected microphone, thereby allowing the sound to be picked up more faithfully.

[0077] In addition to the above-mentioned method, when selecting multiple wireless microphones, the following method is also available. For example, the user can change the volume of the collected sound by sliding up and down or left and right or by performing gesture operations such as pinching in and out near the wireless microphone for which the user wants to change the volume of the collected sound among the multiple selected microphones. The user can select the wireless microphone for which the user wants to change the volume by tapping, long pressing, or double tapping, regardless of the vicinity of the wireless microphone, and then change the volume by performing gesture operations as described above. In addition, for example, when two wireless microphones are selected and a double tap is performed on the display item 407 in FIG. 4, the sound collection levels (volumes) of the two wireless microphones are swapped. In other words, when the volume of microphone 313 in FIG. 4 is 70 and the volume of microphone 312 is 15, a double tap can be performed to set the volume of microphone 313 to 15 and the volume of microphone 312 to 70 (both volumes can be selected up to 100). In addition to swapping, a double tap may be performed to set all the volumes obtained from the wireless microphones selected by the user to the same volume (same sound collection level). Instead of double tapping, the volume of all the wireless microphones may be set to the same volume by double clicking (quickly pressing twice) either operation unit 106c or 106d or pressing both at the same time.

[0078] The adjusted volume can be displayed by increasing or decreasing the wavy line of display item 403 or by changing the display form of display item 407, so that the user can see from which microphone and at what volume the sound is being acquired. If the microphone volume is high, the display may be made with a low transparency (less transparent), and if the volume is low, the display may be made with a high transparency (more transparent). The size of the microphone icon may also be changed in conjunction with the volume.

[0079] Similarly, the display form according to the state of the microphone described above in FIG. 5 can be changed regardless of whether the microphone is selected or not, or whether the microphone is connected or not. For example, regardless of the volume adjustment of the wireless microphone by the user, the display form of the display item (microphone icon) may be changed according to the volume of the sound picked up by the microphone. Specifically, the color of the microphone icon picking up a loud sound is changed. The color of the microphone icon is changed to blue when a loud sound is picked up, and light blue when a smaller but relatively loud sound is picked up, and when the microphone is connected but not picking up sound, it is displayed in gray like the microphone in error in FIG. 5. In addition to changing the color, the microphone picking up sound may be displayed in a flashing manner, and the flashing speed may be increased as the picked up sound becomes louder. Also, when the sound is loud, the number of wavy lines in the display item 403 may be increased to notify the user that more sound is being acquired.

[0080] By using the above-described method, if you want to pick up the sound of a dog, which is the main subject, while shooting a video, you can select a microphone nearby and pick up the sound of the dog. If you want to shoot a video with the voice of the person shooting the dog as the subject, you can select the built-in microphone and associate the user's desired sound with the captured image and save / record it. Note that the above method may be used to change the sound output from the built-in speaker of smartphone 100 even when not shooting a video but waiting to shoot.

[0081] As described above, in this embodiment, the audio used for video shooting can be selected from audio picked up by a microphone located away from the smartphone, not limited to audio around the user shooting. This allows the user to shoot a video together with audio from a location far away from the location where the user is shooting, making it possible to shoot a more realistic video. In addition, by connecting to a microphone located away from the smartphone via the Internet, the user can select a desired microphone from a large number of options without having to take the time and cost to place a microphone. This allows the user to shoot a video as desired.

[0082] The various controls described above as being performed by the system control unit 50 may be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0083] In addition, although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined.

[0084] In the above-mentioned embodiment, the present invention is applied to a smartphone, but the present invention is not limited to this example and can be applied to any electronic device that has an imaging means and is capable of two-way communication. In other words, the present invention can be applied to personal computers, PDAs, digital cameras, portable image viewers, music players, game consoles, electronic book readers, etc.

[0085] The present invention can be applied not only to the electronic device itself, but also to a control device that communicates with an electronic device (including a network camera) via wired or wireless communication and remotely controls the electronic device. Examples of devices that remotely control electronic devices include mobile phone terminals, smartphones, tablet PCs, and desktop PCs. The electronic device can be remotely controlled by notifying the electronic device of commands to perform various operations and settings from the control device side based on operations performed on the control device side or processes performed on the control device side. In addition, a live view image captured by the electronic device may be received via wired or wireless communication and displayed on the control device side.

[0086] Also, a control device may exist separately from the electronic device, and the control device may acquire live view images and sounds from a camera (imaging means) or a wireless microphone (sound collection means) provided in the electronic device via wired or wireless communication. That is, the present embodiment may be applied to a system in which a control device that instructs control execution, an imaging device for acquiring images, and a sound collection device for acquiring sounds are separately present and connected via wired or wireless communication. In this case, the electronic device is an imaging device. Specifically, the control device executes the control flow chart of FIG. 6. The control device transmits a command to the electronic device (imaging device) to send an LV image captured through a camera arranged in the electronic device to the control device, and the LV image is acquired from the imaging device (receives image information) and displayed on the control device (S601). At this time, the control device also acquires angle of view information such as the position of the imaging device and the shooting method. In the control of S607 to S609, the control device transmits a command to the imaging device or the wireless microphone, receives and acquires necessary information from each device, and generates, calculates, and displays the relative position. When the user switches microphones, the control device transmits a command to the selected wireless microphone, receives information from the wireless microphone, and acquires the audio pickup information. When an instruction to start recording video is given in S620, the control device transmits a command to start recording video to the imaging device, and video recording begins using the camera of the imaging device. The video and audio information received and acquired are associated when they are received by the control device, and are recorded on a recording medium or cloud storage.

[0088] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium on which the program is stored constitute the present invention.

Claims

1. A control device capable of communicating with a plurality of sound collecting devices, A display unit; an image acquisition means for acquiring an image captured by an imaging device from the imaging device; A position acquisition means for acquiring position information of the plurality of sound collecting devices; a display control means for controlling the display unit to display a plurality of display items corresponding to the plurality of sound collecting devices, together with the image acquired by the image acquisition means, based on the position information acquired by the position acquisition means; a selection means for selecting a display item from the plurality of display items displayed on the display unit by a user operation; a control means for controlling the display device to acquire audio information from a sound collection device corresponding to the display item selected by the selection means, and to record the image and the audio information in a recording device in association with each other; having The control device is characterized in that the display control means controls the display unit to display, for a sound collecting device located outside the range of the shooting angle of view of the image, a directional item indicating the direction of the position of the sound collecting device located outside the range of the shooting angle of view of the image as a display item corresponding to the sound collecting device.

2. The control device according to claim 1, characterized in that each of the multiple display items is at least one of an icon indicating the sound collection device, a display item related to the audio information, and a display item indicating the direction of a position of a sound collection device outside the range of the shooting angle of view of the image.

3. The control means The control device according to claim 1 or 2, characterized in that a display item corresponding to a sound collecting device located within the range of the shooting angle of view of the image and a display item corresponding to a sound collecting device located outside the range of the shooting angle of view of the image are displayed in different display formats.

4. The display control means 4. The control device according to claim 1, wherein, when the control device is provided with a sound collecting device, a built-in sound collecting item indicating the sound collecting device provided in the control device is displayed on the display unit.

5. A control device as described in any one of claims 1 to 4, characterized in that the display control means changes the display form of the display item corresponding to the sound collection device depending on the state of the sound collection device.

6. The state of the sound collection device is:

6. The control device according to claim 5, characterized in that the state of communication between the sound collection device and the control device is at least one of a connection state of the sound collection device and a state of the sound collected by the sound collection device.

7. The control means When at least one of the plurality of sound collecting devices is selected by the selection means before the start of the recording, control is performed so that sound information collected by the selected sound collecting device is output from the control device; A control device as described in any one of claims 1 to 6, characterized in that when at least one of the multiple sound collecting devices is selected by the selection means after the recording has started, the control device controls the sound collected by the selected sound collecting device to be recorded in the recording device without outputting it from the control device.

8. The control device further includes an imaging device having an imaging unit, The control device according to claim 1 , wherein the image acquisition means acquires an image captured by the imaging unit.

9. The image acquisition means includes: The control device according to claim 1 , wherein the control device communicates with the imaging device, which is a device different from the control device, by wired or wireless communication, and acquires an image captured by the imaging device.

10. A control method for a control device capable of communicating with a plurality of sound collecting devices, comprising: an image acquisition step of acquiring an image captured by the imaging device from the imaging device; A position acquisition step of acquiring position information of the plurality of sound collecting devices; a display control step of controlling a display unit to display a plurality of display items corresponding to the plurality of sound collection devices, together with the image acquired by the image acquisition step, based on the position information acquired by the position acquisition step; a selection step of selecting a display item from the plurality of display items displayed on the display unit by a user operation; a control step of acquiring audio information from a sound collection device corresponding to the display item selected in the selection step, and controlling the image and the audio information to be recorded in a recording device in association with each other; having A control method for a control device, characterized in that in the display control step, for a sound collecting device located outside the range of the shooting angle of view of the image, a directional item indicating the direction of the position of the sound collecting device located outside the range of the shooting angle of view of the image is controlled to be displayed on the display unit as a display item corresponding to the sound collecting device.

11. A program for causing a computer to function as each of the means of the control device according to any one of claims 1 to 9.

12. A computer-readable recording medium storing a program for causing a computer to function as each of the means of the control device according to any one of claims 1 to 9.

Images