Electronic device, control method thereof, and recording medium

Abstract

The present invention discloses an electronic device, a control method thereof, and a recording medium. The electronic device enables a user to appropriately select sounds to be collected related to moving image shooting. The electronic device can communicate with a plurality of sound collection devices, and includes: a display control unit that, for each of the plurality of sound collection devices, based on the respective position information of each sound collection device acquired by a position acquisition unit, displays a display item corresponding to each sound collection device, and displays an image acquired by an image acquisition unit together with the plurality of displayed display items, wherein the display control unit, for each sound collection device, displays the corresponding display item even if the display item corresponds to a sound collection device not within the shooting range of the acquired image; and a control unit that associates and records the acquired image and the acquired sound information in a recording device.

Description

Technical Field

[0001] The present disclosure relates to a method for acquiring sound data related to image shooting. Background Art

[0002] In moving image shooting, in addition to collecting sounds near an electronic device through a microphone built in the electronic device including a shooting function, it may also be necessary to collect and record sounds near a subject (object to be shot) in the direction pointed by a camera. Japanese Patent Laid-Open No. 2020-107971 discloses a technique of automatically selecting a microphone near a subject in the direction of a moving image shot by a user, acquiring sound collection information of the microphone, and recording the sound collection information in association with video data.

[0003] In the technique disclosed in Japanese Patent Laid-Open No. 2020-107971, a shooting device can automatically select a microphone to collect sounds. Therefore, depending on the situation, a choice of using a microphone built in the shooting device or a microphone located at a position different from a position near a main subject being shot by a user cannot be provided to the user for sound collection.

[0004] Therefore, sometimes a microphone that the user does not want will be automatically selected for sound collection, so that in some cases, the sounds that the user wants cannot be recorded. Summary of the Invention

[0005] The present disclosure provides improved techniques and mechanisms that enable a user to appropriately select sounds to be collected related to moving image shooting.

[0006] According to aspects of the present disclosure, there is provided an electronic device communicable with a plurality of sound collection devices. The electronic device includes: a display unit; an image acquisition unit configured to acquire an image shot by the shooting device from the shooting device; a position acquisition unit configured to acquire, for each of the plurality of sound collection devices, position information relative to the image shot by the shooting device; a display control unit configured to, for each of the plurality of sound collection devices, based on the position information of each sound collection device acquired by the position acquisition unit, control the display unit to display a display item corresponding to each sound collection device, and display the image acquired by the image acquisition unit together with the plurality of displayed display items, wherein the display control unit is configured to, for each sound collection device, control to display the corresponding display item even if the display item corresponds to a sound collection device not within the shooting range of the acquired image; and a control unit configured to control to acquire sound information from the sound collection device selected by the user from the plurality of sound collection devices, and control to associate and record the acquired image and the acquired sound information in a recording device.

[0007] Other features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1A and Figure 1B are schematic views of the appearance of a smart phone according to an exemplary embodiment of the present disclosure.

[0009] Figure 2 is a block diagram illustrating the configuration of a smart phone according to an exemplary embodiment of the present disclosure.

[0010] Figure 3 is a schematic view illustrating the positional relationship between a smart phone and a sound collection device (microphone) according to an embodiment of the present disclosure.

[0011] Figure 4 is a schematic view illustrating an example of a screen for selecting a sound collection device (microphone) displayed on a display according to an embodiment of the present disclosure.

[0012] Figure 5 Illustrates an example of the display of the state of a sound collection device (microphone) according to an embodiment of the present disclosure.

[0013] Figure 6A and Figure 6B is a flowchart illustrating the control for selecting a sound collection device (microphone) according to an embodiment of the present disclosure.

[0014] Figure 7 is a schematic view illustrating another example of the display of a screen for selecting a sound collection device (microphone) according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] Exemplary embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0016] Figure 1A and Figure 1B are schematic views of the appearance of a smart phone 100 as an example of a device (electronic device) applicable to the embodiments of the present disclosure. Figure 1A is a front view of the smart phone 100, and Figure 1B is a rear view of the smart phone 100. In Figure 1A and Figure 1BIn [description], the display 104 is a display unit provided on the front surface of the smart phone 100 to display images and various information. The touch panel 106a is a touch operation member and can detect a touch operation on the display surface (operation surface) of the display 104 (capable of performing touch detection). The smart phone 100 can display a live view image (LV image) captured by the built-in camera (in-camera) 116 or the external camera (out-camera) 117 or both of them on the display 104.

[0017] The power button 106b is an operation member included in the operation unit 106 and can switch the on / off state of the display 104. Continuously pressing the power button 106b for a certain period of time (long press), for example, 3 seconds, can switch the on / off state of the smart phone 100.

[0018] The volume plus button 106c and the volume minus button 106d are volume buttons for controlling the volume of the sound output from the sound output unit 112. When the volume plus button 106c is pressed, the volume will increase, and when the volume minus button 106d is pressed, the volume will decrease. Further, in the recording standby state during the use of any camera, the volume plus button 106c and the volume minus button 106d are used as a shutter button for indicating still image shooting and a record (REC) button for indicating the start / stop of moving image recording. When the power button 106b and the volume minus button 106d are pressed simultaneously or when the power minus button 106d is pressed quickly multiple times, the user can selectively set a specific function to be executed.

[0019] The home button 106e is an operation button for displaying the home screen of the smart phone 100, which is the startup screen, on the display 104. In the case of starting and using various application programs in the smart phone 100, when the home button 106e is pressed, the various application programs in use can be temporarily closed to display the home screen. It is assumed that the home button 106e is a physically pressable button. However, the home button 106e may not be a physical button but a touchable button with a similar function displayed on the display 104. Optionally, in response to a specific gesture performed on the touch panel 106a, a function similar to that of the home button 106e can be performed.

[0020] The sound output terminal 112a is a headphone jack and is a terminal for outputting sound to headphones, external speakers, etc. The sub-speaker 112b and the main speaker 112c are built-in speakers each for outputting sound. In a case where an output sound terminal (for example, a headphone cable) is not connected to the sound output terminal 112a and in a case where the user puts the smart phone 100 to his / her ear for making a call, sound is output from the sub-speaker 112b. In other cases, for example, when the user starts playing a moving image without wearing headphones, sound is output from the main speaker 112c.

[0021] The sound collection unit 113 is a microphone built in the smart phone 100 and collects the sound around the smart phone 100. For example, when the user makes a call using the smart phone 100, the sound collection unit 113 can collect the user's voice, or the sound collected by the sound collection unit 113 can be associated with a moving image recorded by using the built-in camera 116 or an external camera 117. According to the present exemplary embodiment Figure 1A , one sound collection unit 113 is built in the smart phone 100. However, a plurality of sound collection units 113 may also be built in the smart phone 100.

[0022] Figure 2 is a block diagram illustrating a configuration example of the smart phone 100 according to the present exemplary embodiment. The central processing unit (CPU) 101, the memory 102, the non-volatile memory 103, the display image processing unit 105, the display 104, the operation unit 106, the recording medium interface (I / F) 107, the external I / F 109, and the communication I / F 110 are connected to the internal bus 120. Further, the sound output unit 112, the sound collection unit 113, the posture detection unit 114, the global positioning system (GPS) 115, the built-in camera 116, the external camera 117, the infrared camera 118, and the camera image processing unit 119 are connected to the internal bus 120. The units connected to the internal bus 120 can exchange data with each other through the internal bus 120.

[0023] The CPU 101 is a control unit that controls the entire smart phone 100 and includes at least one processor or circuit. The memory 102 includes a random access memory (RAM) (for example, a volatile memory using a semiconductor device). For example, the CPU 101 controls the respective units of the smart phone 100 by using the memory 102 as a working memory based on the programs stored in the non-volatile memory 103. The non-volatile memory 103 stores image data, sound data, other data, various programs for the operation of the CPU 101, etc. The non-volatile memory 103 includes a flash memory or a read only memory (ROM).

[0024] The display 104 displays images and a GUI that constitutes a graphical user interface (GUI) screen under the control of the CPU 101. The CPU 101 controls the units of the smart phone 100 to generate a display control signal based on a program and generate and output a video signal to be displayed on the display 104. The display 104 displays video based on the output video signal. Note that the smart phone 100 itself may include an interface for outputting a video signal to be displayed on the display 104, and the display 104 may be constituted by an external monitor (e.g., a television).

[0025] The display image processing unit 105 analyzes the images and GUI displayed on the display 104. The display image processing unit 105 may also detect an icon (banner) displayed on the display 104. Similar to the display 104, the display image processing unit 105 may be built into an external monitor (e.g., a television).

[0026] The operation unit 106 is an input device that receives user operations and includes a character information input device such as a keyboard, an indicating device such as a mouse and a touch panel, buttons, a dial, a joystick, a touch sensor, and a touchpad. The touch panel is an input device that is flatly configured to be superimposed on the display 104 and outputs coordinate information corresponding to the touch position. The operation unit 106 includes the touch panel 106a, a power button 106b, a volume plus button 106c, a volume minus button 106d, and a home button 106e described above.

[0027] A recording medium 108 such as a memory card, a compact disc (CD), or a digital versatile disc (DVD) can be mounted on the recording medium I / F 107. The recording medium I / F 107 reads data from the mounted recording medium 108 and writes data into the recording medium 108 under the control of the CPU 101. The recording medium 108 may be a built-in memory included in the smart phone 100.

[0028] The external I / F 109 is an interface that is connected to an external device via a wired cable or radio and performs input / output of video signals and sound signals.

[0029] The communication I / F 110 establishes a connection via radio or a wired cable and sends video signals and sound signals to the sound collection device 150 or receives video signals and sound signals from the sound collection device 150. The communication I / F 110 can establish a connection with a wireless local area network (LAN) and the Internet 111. Further, the communication I / F 110 can pass through or Communicate with external devices with low power consumption. The communication I / F 110 can send moving images (including live images) captured by the built-in camera 116 or the external camera 117 or both of them, as well as the images recorded in the recording medium 108, to the cloud storage 250 or the like. Further, the communication I / F 110 can receive image data and various other information from external devices.

[0030] The Internet 111 indicates wireless LAN communications such as Wi-Fi, and public wireless communications such as fourth-generation (4G) communication and fifth-generation (5G) communication.

[0031] The sound output unit 112 performs noise reduction processing and amplification processing on the sound data input from the sound collection unit 113. Further, the sound output unit 112 outputs the sound of moving images and music data, operation sounds, ringing alarms, various notification sounds, etc. from the sound output terminal 112a connected to, for example, headphones, and from the speakers 112b and 112c through wireless communication or the like.

[0032] The posture detection unit 114 detects the posture of the smartphone 100 in the direction of gravity, as well as the inclination of the posture on the yaw axis, roll axis, and pitch axis. It can be determined whether the smartphone 100 is held horizontally, vertically, upward, downward, or in an inclined posture based on the posture detected by the posture detection unit 114. This can also detect the direction of the built-in camera 116 or the external camera 117. At least one of an acceleration sensor, a gyro sensor, a geomagnetic sensor, an azimuth sensor, an altitude sensor, and other sensors can be used as the posture detection unit 114, and multiple sensors can be used in combination.

[0033] The GPS 115 is a system that measures geographical information by using artificial satellites. The GPS 115 sends signals to artificial satellites and specifies the geographical location according to the time difference until the response of the signal is returned. Hereinafter, the said location is referred to as camera location information. The camera location information includes longitude information, latitude information, and time information.

[0034] The built-in camera 116 is a camera configured on the same surface as the display 104 in the housing of the smartphone 100.

[0035] The external camera 117 is a camera configured on the side opposite to the display 104 in the housing of the smartphone 100. In Figure 1A and Figure 1B One built-in camera 116 and one external camera 117 are illustrated, but the number of built-in cameras 116 and the number of external cameras 117 are not limited to this. Multiple cameras with different focal lengths can be installed.

[0036] A live view image captured by the built-in camera 116, the external camera 117, or both of them can be displayed on the display 104. Based on whether the built-in camera 116 or the external camera 117 is used, it is possible to select whether to capture an image of the scene seen by the user or an image of the user himself. In the case where both the built-in camera 116 and the external camera 117 are selected, the LV images captured by each camera are displayed on the display 104.

[0037] The infrared camera 118 is a camera configured on the same surface as the display 104 in the housing of the smart phone 100, and is configured on the side of the sub-speaker 112b opposite to the built-in camera 116. Through the infrared camera 118, data of the user's face can be analyzed to create a depth map and an infrared image of the face. This can also be used to unlock the smart phone 100.

[0038] The camera image processing unit 119 performs various image processing and identification of display items on the screen displayed on the display 104 under the control of the CPU 101. The camera image processing unit 119 can perform various image processing on images stored in the non-volatile memory 103 or the recording medium 108, video signals obtained through the external I / F 109, images obtained through the communication I / F 110, etc.

[0039] - An operation in which a finger or a stylus that has not been in contact with the touch panel 106a re-contacts the touch panel 106a, that is, touch start (hereinafter referred to as "Touch-Down").

[0040] - A state in which a finger or a stylus is in contact with the touch panel 106a (hereinafter referred to as "Touch-On").

[0041] - An operation in which a finger or a stylus moves on the touch panel 106a when in contact with the touch panel 106a (hereinafter referred to as "Touch-Move").

[0042] - An operation in which a finger or a stylus in contact with the touch panel 106a separates from the touch panel 106a, that is, touch end (hereinafter referred to as "Touch-Up").

[0043] - A state in which nothing is in contact with the touch panel 106a (hereinafter referred to as "Touch-Off").

[0044] When "touch down" is detected, "touch ongoing" will be detected simultaneously. Unless "touch up" is detected after "touch down", "touch ongoing" will be continuously detected. When "touch move" is detected, "touch ongoing" will also be detected simultaneously. Even if "touch ongoing" is detected, when the touch position does not move, "touch move" will not be detected. When "touch up" of all the fingers and the stylus in contact is detected, "touch off" will be detected.

[0045] These operations, states, and the position coordinates of the contact between the finger or the stylus and the touch panel 106a will be notified to the CPU 101 via the internal bus 120. The CPU 101 will determine which operation (touch operation) has been performed on the touch panel 106a based on the notified information.

[0046] For "touch move", based on the change in the position coordinates, the moving direction of the finger or the stylus moving on the touch panel 106a can also be determined for each vertical component and horizontal component on the touch panel 106a. In the case where "touch move" of a predetermined distance or more is detected, it is determined that a swipe operation has been performed. An operation of quickly moving a finger a certain distance and then separating the finger when the finger is in contact with the touch panel 106a is called a flick.

[0047] In other words, a flick is a moving operation of quickly moving a finger on the touch panel 106a like a flip. When "touch move" of a predetermined distance or more at a predetermined speed or faster is detected and then "touch up" is detected, it can be determined that a flick has been performed (it can be determined that a flick has been performed after a swipe operation).

[0048] Furthermore, a touch operation of simultaneously touching multiple positions (for example, 2 positions) and making the touch positions close to each other is called a pinch-in, and a touch operation of making the touch positions away from each other is called a pinch-out.

[0049] Pinch-out and pinch-in are collectively called pinch operations (or simply called "pinch"). For the touch panel 106a, various types of touch panels such as a resistive film type, a capacitance type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type can be used.

[0050] There are methods for detecting a touch when the touch panel is actually touched and methods for detecting a touch when a finger or a stylus approaches the touch panel. The touch panel can adopt any one of the methods.

[0051] The sound collection device 150 is a device connected to the Internet and capable of exchanging data with the smart phone 100 through the communication I / F 110, and includes a sound collection unit 151, a GPS 152, and a sound processing unit 153. In the case where the sound data collected by the sound collection unit 151 is selected by a user operation, the sound collection device 150 can acquire the sound data and the position information etc. obtained by the GPS 152 through the communication I / F 110, and can store the sound data and the position information etc. in association with the moving image captured by the external camera 117 or other cameras.

[0052] The cloud storage 250 can exchange information such as image data with the smart phone 100 through the communication I / F 110. For example, in the case where the user captures a moving image by using the external camera 117, the moving image may not be stored in the recording medium 108 built in the smart phone 100, but exists in the cloud storage 250 in real time. At this time, as described above, the moving image will be stored in the cloud storage 250 in association with the sound information and the position information obtained by the sound collection unit 113 or the sound collection device 150.

[0053] will be referred to Figures 3 to 7 Describe the display example of the sound collection device (microphone) and the method of selecting the sound collection device when capturing a moving image by using the smart phone 100 according to the present exemplary embodiment.

[0054] Figure 3 is a schematic diagram exemplifying the positional relationship between the subject in the LV image and a plurality of selectable sound collection devices (microphones) according to the present exemplary embodiment. Figure 3 Exemplifies a scene where the user 302 holds the smart phone 100 to capture a moving image by using the external camera 117. The range 301 represents the range of the shooting view angle captured by the external camera 117 of the smart phone 100. The subjects 303 to 305 are located on the side of the smart phone 100 opposite to the user 302 and on the side where the external camera 117 is placed. The subject 303 is the subject (puppy) to be captured by the user 302 as the main subject. The subject 304 is the subject (bird) located behind the subject 303 which is the main subject. The subject 305 is the subject (chick) located outside the shooting view angle (range 301) of the external camera 117 and on the right side of the shooting direction.

[0055] The microphone 312 is a microphone that can collect sounds through the sound collection unit 113 built into the smart phone 100, and is capable of collecting sounds around the smart phone 100 (i.e., around the user 302). Devices capable of collecting sounds, as exemplified by microphones 313 to 316, are devices such as wireless microphones. Microphones 313 to 316 are connected to the Internet, and the user can freely obtain sounds by connecting a device such as the smart phone 100 that can be connected to the Internet to microphones 313 to 316.

[0056] The microphone 313 is a wireless microphone close to the subject 303, and is capable of collecting sounds around the subject 303. The microphone 314 is a wireless microphone close to the subject 304, and is capable of collecting sounds around the subject 304. The microphone 315 is a sound collection device capable of collecting sounds around the subject 305. The microphone 316 is a wireless microphone outside the range 301. No subject appears near the microphone 316, but the microphone 316 is arbitrarily optional. Microphones 313 to 316 are microphones that can be selected by the user. By communicating via the communication I / F 110 of the smart phone 100, the sound information collected by each of microphones 313 to 316 can be obtained and stored in the recording medium 108 or the cloud storage 250. The smart phone 100 performs two-way communication with each of microphones 313 to 316 through the communication I / F 110.

[0057] Among microphones 313 to 316, not only can one microphone be selected but also multiple microphones can be selected. In the case of selecting multiple microphones, the CPU 101 can automatically determine the ratio of the sounds to be used by analyzing the acquired sounds, or the user can arbitrarily determine the ratio of the sounds. For microphones 313 to 316, wireless microphones detected by the smart phone 100 through the Internet are exemplified, but the display method is not limited thereto. Instead of the microphone icon, a speaker icon can be displayed. Microphones 312 to 316 are displayed in the same way regardless of the connection method to the smart phone 100, but they can also be displayed in different ways.

[0058] Figure 4 It is exemplified that Figure 3In the illustrated state, an example is shown where a sound collection device (hereinafter, a wireless microphone) capable of collecting sound via the Internet is displayed on the display 104 by being superimposed on an LV image captured by the external camera 117. The LV image 400 of the range 301 is displayed on the display 104. The display item 402 represents a microphone (built-in sound collection item) built into the smart phone 100. The display item 403 indicates that the microphone 313 is currently selected by the user and the sound information collected by the microphone 313 is currently being used. The display items 405 and 406 indicate that the microphones 315 and 316 are outside the range 301 which is the camera shooting angle range. Both the display items 405 and 406 notify the user that the microphones 315 and 316 are on the right side outside the range 301, and the size of the arrow indicates the distance from the range 301. In Figure 4 the arrow of the display item 406 is smaller (shorter) than the arrow of the display item 405. This indicates that the position of the microphone 316 is closer to the range 301 than the microphone 315. The display items 405 and 406 enable the user to visually identify the sound collection devices located outside the camera shooting angle. Therefore, the user can select the desired sound from a large number of sound collection devices. In Figure 4 the illustrated state, when the user selects any one of the microphones 312 to 316 by a touch operation, the microphone from which the sound information to be stored in association with the moving image is obtained can be selected or changed. The display item 407 is a level meter indicating the amount of sound collection information on the currently selected microphone. In the case of using the sound collection information of one microphone, the upper part represents the left (L) side, and the lower part represents the right (R) side. In the case of using the sound collection information of multiple microphones, the upper part and the lower part of the display item 407 represent the level meters of the sound collection information of different microphones.

[0059] Figure 5 An example of a display mode depending on the state of the microphone is illustrated. In the case of the currently selected microphone (selected state), a display indicating that sound waves are output from the microphone icon is given to the corresponding microphone icon. This indicates that sound is collected from the corresponding microphone. In the case of an unselected microphone (unselected state), a display of waves is not given. Before the shooting of the moving image in step S620 described below (that is, when no moving image is being shot), the sound collected by the microphone selected by the user is output from the sound output unit 112. After the shooting of the moving image is instructed (that is, when shooting a moving image), the sound collected by the microphone selected by the user will be stored in association with the moving image being shot. At this time, the sound of the selected microphone is not output. Note that the display depending on the state of the microphone is not limited to Figure 6B In the illustrated state, when the user selects any one of the microphones 312 to 316 by a touch operation, the microphone from which the sound information to be stored in association with the moving image is obtained can be selected or changed. The display item 407 is a level meter indicating the amount of sound collection information on the currently selected microphone. In the case of using the sound collection information of one microphone, the upper part represents the left (L) side, and the lower part represents the right (R) side. In the case of using the sound collection information of multiple microphones, the upper part and the lower part of the display item 407 represent the level meters of the sound collection information of different microphones. Figure 5The display modes exemplified therein. As long as the user can visually recognize the different states of the microphone, the color of the microphone can be changed or animations such as vibrations of a microphone icon can be displayed.

[0060] The selectable microphone is displayed as a microphone icon. Displaying the microphone icon enables the user to visually recognize the microphone (sound collection device) located at the position of the displayed icon. In the case where the selectable microphone is outside the current camera shooting view angle (range), a display item of the selectable microphone outside the screen area will be displayed. The direction (position) of the microphone outside the camera shooting view angle is exemplified by a direction item such as an arrow. For example, in the case where the microphone is temporarily unable to connect due to a communication error, or in the case where the wireless microphone cannot be turned on (off) due to battery exhaustion etc. and the wireless microphone cannot collect sound, even if the microphone is selectable when no error occurs, a display item of the error microphone will be displayed. In other words, the display item of the error microphone indicates the sound collection state or communication connection state of the wireless microphone. Further, for example, in the case where the microphone is unavailable because it has been used (selected) by another user, a display item of the error microphone will be displayed.

[0061] The representation method of the microphone is not limited to this. The sound level can be represented by the size of the microphone icon, or auxiliary information such as a quantity gauge for user selection can be displayed near the microphone. Although an arrow is used to represent the position of the microphone outside the view angle, the direction item representing the direction is not limited to an arrow. Further, the distance can be represented by the length or thickness of the arrow, or the recommendation degree can be represented by changing the color of the arrow based on the sound collection condition. In addition, multiple wireless microphones can be selected. For example, when touching an unselected microphone, the microphone will be set as the sound collection information acquisition target. Also, even when touching other microphones, the microphone is not switched and is added to the sound collection information acquisition target. This can associate multiple pieces of sound collection information with the moving image. When touching the selected microphone again, the selection will be cancelled, which enables sound collection from any microphone. To switch microphones more smoothly, in response to an operation such as a drag-and-drop from one microphone to another, the sound collection information can be switched in a cross-fading manner.

[0062] Figure 6A and Figure 6B are flowcharts illustrating the control of the smart phone 100 according to the exemplary embodiment of the present example.

[0063] Figure 6A and Figure 6BThis is a control flowchart of the moving image shooting mode process that starts in the moving image shooting standby state after the smart phone 100 is started (turned on). When the CPU 101 loads the program stored in the non-volatile memory 103 into the memory 102 and executes the program, the control process is implemented.

[0064] In step S601, the CPU 101 causes the camera image processing unit 119 to perform image processing on the image captured by the built-in camera 116 or the external camera 117, and displays the generated image as a live view image on the display 104.

[0065] In step S602, the CPU 101 detects connectable wireless microphones located within a predetermined range through the communication I / F 110. At this time, the CPU 101 also acquires the GPS information of each wireless microphone.

[0066] In step S603, the CPU 101 inputs the number of wireless microphones detected in step S602 to the variable Num, and stores the variable Num in the memory 102.

[0067] In step S604, the CPU 101 determines whether the variable Num is 0. If the variable Num is 0 (the "Yes" in step S604), the process proceeds to step S617. If the variable is not 0 (the "No" in step S604), the process proceeds to step S605.

[0068] In step S605, the CPU 101 initializes the variable i to 0, and stores the variable i in the memory 102. The variable i will be described in step S606 below.

[0069] In step S606, the CPU 101 acquires the position information of the i-th detected wireless microphone among the wireless microphones detected in step S602. The position information acquired in this step is absolute position information.

[0070] In step S607, the CPU 101 acquires the position information of the smart phone 100 by using the GPS 115. The CPU 101 calculates the relative position of the i-th wireless microphone with respect to the smart phone 100 from the position information of the smart phone 100, the position information of the wireless microphone acquired in step S606, and the direction of the camera of the currently captured image. From the relative position relationship between the smart phone 100 and the microphone calculated in this step, the position of the detected microphone can be displayed on the LV image to be described in step S609 below. In other words, the relative position calculated in this step is information corresponding to the position information of the wireless microphone.

[0071] In step S608, the CPU 101 obtains the shooting view angle information from the shooting direction and specification information of the camera that captures the current image, thereby calculating the shooting range. Figure 3 An example of the positional relationship at this time is illustrated. Assume that the shooting range obtained in step S608 is range 301, the subjects detected in the live view image are subjects 303 to 305, and the wireless microphones detected in step S602 are microphones 313 to 315. Range 301 is the range that can be captured by the external camera 117 and is the range displayed on the display 104. More specifically, subjects 303 and 304 can be imaged, and microphones 313 and 314 are wireless microphones included within the shooting view angle. Subject 305 is not included within the shooting view angle. Microphones 315 and 316 are also not included within the shooting view angle (outside range 301); however, in a manner similar to microphones 313 and 314, microphones 315 and 316 can be arbitrarily selected by the user, and sound information can be obtained from microphones 315 and 316.

[0072] Microphone 312 represents the microphone (sound collection unit 113) built into the smart phone 100, and the sound information collected by microphone 312 is similarly obtainable and selectable. The sound information to be used is not limited to the sound information collected by only one of microphones 312 to 316, and the sound information collected by multiple microphones can also be used. In the case of using the sound information collected by multiple microphones, the user can arbitrarily select the ratio of the sound of each microphone to be used, or the CPU 101 can automatically determine the ratio from the obtained sound information collected by multiple microphones and synthesize the sound.

[0073] In step S609, based on the relative positions of the wireless microphones calculated in step S607 and the shooting range calculated in step S608, the CPU 101 displays by superimposing display items on the LV image Figure 5 the display items illustrated in. Figure 4 An example of what is displayed on the display 104 of the smart phone 100 at this time is illustrated. As Figure 4 illustrated, the live view image (LV image 400) of range 301 is displayed on the display 104. In Figure 4 it can be seen from display item 403 that the sound information collected by microphone 313 is being used and selected.

[0074] In step S610, the CPU 101 refers to the memory 102 and determines whether the variable i is less than the variable Num. If the variable i is less than the variable Num (the "Yes" in step S610), the process proceeds to step S611. Otherwise (the "No" in step S610), the process proceeds to step S612.

[0075] In step S611, the CPU 101 increments the variable i by 1 and stores the variable i in the memory 102. Then, the process returns to step S606.

[0076] In step S612, the CPU 101 displays a display item representing the built-in microphone (sound collection unit 113) of the smart phone 100 on the LV image of the display 104. In other words, the CPU 101 superimposes and displays Figure 4 the display item 402 in.

[0077] In step S613, the CPU 101 determines whether the user has selected a microphone different from the microphone currently collecting sound through the operation unit 106. If the user has selected a microphone different from the microphone currently collecting sound (Yes in step S613), the process proceeds to step S614. Otherwise (No in step S613), the process proceeds to step S615.

[0078] In step S614, the CPU 101 changes the icon representing the wireless microphone selected by the user in step S613 to a display item representing the selected microphone, and displays the display item on the display 104. In Figure 4 the illustrated state, when the microphone 314 is selected by the user operation, the display item 403 is assigned to the microphone 314.

[0079] In step S615, the CPU 101 determines whether any wireless microphone has been selected. If any wireless microphone has been selected (Yes in step S615), the process proceeds to step S616. If no wireless microphone has been selected (No in step S615), the process proceeds to S617.

[0080] In step S616, the CPU 101 acquires the sound collection information of the selected microphone. As described above for Figure 4 the display item 407 in, when no moving image is being shot, the sound collected by the selected microphone is output from the sound output unit 112. When a moving image is being shot, the sound is stored in association with the moving image without being output.

[0081] In step S617, since the determination result in step S615 is No, the CPU 101 displays the icon representing the built-in microphone as the selected microphone on the display 104. In other words, in Figure 4 it, the display item 403 is assigned to the microphone 312.

[0082] In step S618, the CPU 101 acquires the sound collection information of the built-in microphone by using the sound collection unit 113.

[0083] In step S619, the CPU 101 draws a sound quantity gauge based on the sound collection information acquired in step S616 or step S618, and displays the sound quantity gauge on the display 104. For example, as shown in the display item 407 in Figure 4 it shows a quantity gauge of the sound information acquired by the microphone selected by the user.

[0084] In step S620, the CPU 101 determines whether to instruct the start of moving image shooting.

[0085] In the case where the start of moving image shooting is instructed (Yes in step S620), the process proceeds to step S621. In the case where the start of moving image shooting is not instructed (No in step S620), the process proceeds to step S625. In the present exemplary embodiment, in the camera standby state, when the LV image is displayed on the display 104, in the case of "touch operation" of the start moving image shooting icon displayed on the display 104 or when the volume button 106c or 106d is pressed, it is determined that the start of moving image shooting is instructed.

[0086] In step S621, the CPU 101 starts moving image shooting. More specifically, the CPU 101 creates a moving image file in the recording medium 108 or the cloud storage 250, and stores the moving image captured by the built-in camera 116 or the external camera 117 with the currently set content.

[0087] In step S622, the CPU 101 stores the moving image started shooting in step S621 in association with the sound information. At this time, based on the user's settings, the CPU 101 stores the moving image and the sound information in the recording medium 108 or the cloud storage 250 through the communication I / F 110. Compared with the case of using the recording medium 108, using the cloud storage 250 on the Internet 111 can record moving images with a long recording time and large-capacity moving images including a large amount of sound information. In this example, the moving image and the associated sound information are stored in the recording medium 108 or the cloud storage 250; however, for example, the moving image and the associated sound information can be output through the external I / F109, and the external signal receiving device can receive the moving image and the associated sound information. In other words, the moving image and the associated sound information can be output to an external recording device (e.g., recording device) connected by a wired cable or radio.

[0088] In step S623, the CPU 101 determines whether to instruct the stop of moving image shooting. If the stop of moving image shooting is instructed (Yes in step S623), the process proceeds to step S624. If the stop of moving image shooting is not instructed (No in step S623), the process returns to step S613. More specifically, the instruction to stop moving image shooting indicates a touch operation on the stop moving image shooting icon displayed on the display 104, or a press of any of the volume buttons 106c and 106d and the main button 106e.

[0089] During the shooting of a moving image, when the main button 106e is pressed, the screen returns to the home screen, and at the same time, the shooting of the moving image stops.

[0090] In step S624, the CPU 101 stops the moving image shooting. When the moving image shooting is stopped, the CPU 101 performs a process of closing the moving image file created in the recording medium 108 or the cloud storage 250 (adding attribute information, etc.).

[0091] In step S625, the CPU 101 determines whether the user has performed an operation to end the moving image shooting. If the user has performed an operation to end the moving image shooting (Yes in step S625), the process ends. If the user has not performed an operation to end the moving image shooting (No in step S625), the process returns to step S601.

[0092] In the present exemplary embodiment, a wireless microphone around the smartphone 100 is detected, and a display item of the detected wireless microphone is superimposed and displayed on the LV image. In the above description, the user selects the displayed display item (microphone icon) by a touch operation to obtain sound data from the microphone associated with the selected microphone icon, and the sound data is stored in association with the moving image. However, the method is not limited thereto.

[0093] The microphone can be selected not only by a touch operation but also by other operations such as an operation by a physical button (e.g., cross keys). Further, an example of selecting a microphone superimposed and displayed on the LV image is described; however, for example, as Figure 7 shown, a radar can be separately displayed from the LV image for the user to select a microphone. During shooting, when a microphone near the main subject is selected, the sound collection sensitivity of the microphone near the main subject can be changed based on the ratio of the main subject occupying the shooting view angle, thereby creating a sense of presence, etc. In the case of stereo, the left-right (LR) balance during sound collection can be corrected based on the relationship between the direction of the camera and the selected microphone to collect sound more faithfully.

[0094] When multiple wireless microphones are selected, in addition to the above methods, microphones can also be selected by the following method. For example, among the multiple selected microphones, the user can change the volume of the sound to be collected by swiping a finger in the vertical direction or left-right direction or performing a gesture operation such as pinching and separating near the wireless microphone whose sound volume is to be changed. The operation is not limited to the operation near the wireless microphone. After selecting the wireless microphone whose volume is to be changed by a tap operation, long press, or double tap, the user can change the volume by performing the above gesture operation. For example, when two wireless microphones are selected, when double tapping on the display item 407 in Figure 4 , the sound collection quantity (volume) of the two wireless microphones can be switched. In other words, when the volume of the microphone 313 in Figure 4 is set to 70 and the volume of the microphone 312 is set to 15, when double tapping, the volume of the microphone 313 can be changed to 15 and the volume of the microphone 312 can be changed to 70 (the selectable volume does not exceed 100). The operation is not limited to switching. When double tapping, the volume of the sound obtained from the wireless microphone selected by the user can be changed to the same volume (the same sound collection quantity). Not limited to double tapping, in response to double clicking (quickly pressing twice) either of the volume buttons 106c and 106d or pressing both the volume buttons 106c and 106d simultaneously, the volume of the wireless microphone can be changed to the same volume.

[0095] In addition, the adjusted volume of the sound obtained from each microphone is displayed by increasing / decreasing the number of wavy lines of the display item 403 or changing the display mode of the display item 407. When the volume of the microphone is large, the microphone icon can be displayed with a low transmittance (reducing transparency), while when the volume of the microphone is small, the microphone icon can be displayed with a high transmittance (increasing transparency). In addition, the size of the microphone icon can change with the volume.

[0096] Regardless of whether a microphone is selected and whether there is a connection, the display mode corresponding to the state of the microphone described above with reference to Figure 5 can be changed. For example, regardless of the user's adjustment of the volume of the wireless microphone, the display mode of the display item (microphone icon) can be changed based on the volume of the sound collected by the microphone. More specifically, the color of the microphone icon that collects a large sound can be changed. For example, when the microphone collects a large sound, the color of the microphone icon can be changed to blue. Even if the sound is smaller than the sound collected by the microphone displayed in blue, when the microphone collects a relatively large sound, the color of the microphone icon can be changed to light blue. When the microphone is connected but no sound is being collected, the color of the microphone icon can be changed to be like Figure 5The error in is the same gray color as the microphone. The change in the display mode is not limited to changing the color. The microphone icon of the microphone that collects sound may blink, and when the collected sound is louder, the blinking speed may increase. Optionally, in the case where the collected sound is loud, the number of wavy lines in the display item 403 may increase to notify the user that the corresponding microphone has collected a large amount of sound.

[0097] In moving image shooting, in order to collect the voice of a dog as the main subject, the above method can be used to select the microphone near the dog and collect the voice of the dog. In order to shoot the image of the dog as the subject together with the voice of the photographer who shoots the image of the dog, selecting the built-in microphone can store and record the voice the user wants in association with the captured image. Even in the camera standby state rather than in moving image shooting, the above method can also be used to change the sound output from the speaker built into the smart phone 100.

[0098] As described above, in the present exemplary embodiment, as the sound for moving image shooting, it is not limited to the sound around the user who performs shooting, and the sound the user wants can be selected from the sounds collected by the microphone at a position far from the smart phone. This can shoot a moving image together with the sound at the position where the image is shot far from the user, and can shoot the moving image with a higher sense of presence. In addition, by connecting a microphone far from the smart phone through the Internet, the user can select the desired microphone from multiple options without the labor and cost of installing the microphone. Thus, the user can perform the desired moving image shooting.

[0099] Note that the above various controls performed by the CPU 101 can be performed by one piece of hardware, or multiple pieces of hardware (for example, multiple processors and circuits) can share the processing to control the entire device.

[0100] The present disclosure has been described in detail based on the exemplary embodiment. However, the present invention is not limited to a specific exemplary embodiment, and may include various forms that do not deviate from the scope of the present invention.

[0101] In addition, in the above exemplary embodiment, the case where the features of the present disclosure are applied to a smart phone has been described as an example; however, the present invention is not limited to being applied to a smart phone, and can be applied to other electronic devices including a camera unit and capable of two-way communication, for example. For example, various embodiments of the present invention can be applied to a personal computer (PC), a personal digital assistant (PDA), a digital camera, a mobile image viewer, a music player, a game console, an e-book reader, and the like.

[0102] Not limited to the main body of the electronic device, the embodiments of the present disclosure can also be applied to a control device that can communicate with an electronic device (e.g., a network camera) through wired communication or wireless communication and can remotely control the electronic device. Examples of the device (to which the embodiments of the present disclosure can be applied) that can remotely control such an electronic device include a mobile phone terminal, a smartphone, a tablet computer, and a desktop computer. In the embodiments, such a control device can remotely control such an electronic device by sending a command that causes the electronic device to perform various operations and settings based on the operations performed by the control device and the processing performed by the control device. Additionally, in the embodiments, such a control device can receive a live view image captured by such an electronic device through wired communication or wireless communication, and the control device can display the live view image.

[0103] Furthermore, a control device different from the electronic device can obtain a live view image and sound from a camera (imaging unit) or a wireless microphone (sound collection unit) of the electronic device through wired communication or wireless communication. In other words, the present exemplary embodiment can also be applied to a system in which a control device that instructs control execution, an imaging device that obtains an image, and a sound collection device that obtains sound are separately provided and connected through wired communication or wireless communication. In this case, the electronic device serves as the imaging device. More specifically, the control device performs Figure 6A and Figure 6B the processing of the control flow chart in. In step S601, the control device sends a command to cause the electronic device (imaging device) to send the LV image captured by the camera of the electronic device to the control device, obtains the LV image (receives image information) from the imaging device, and displays the LV image. At this time, the control device further obtains view angle information such as the position of the imaging device and the imaging method. In the control of steps S607 to S609, the control device sends a command to the imaging device or the wireless microphone, receives / obtains necessary information from each device, and performs generation, calculation, and display of the relative position. When the user switches the microphone, the control device sends a command to the selected wireless microphone and receives information from the wireless microphone to obtain sound collection information. When step S620 indicates the start of moving image recording, the control device sends a command to start moving image recording to the imaging device, and starts moving image recording by using the camera provided in the imaging device. When the control device receives moving image and sound information, the control device associates the received / acquired moving image with the sound information, and records the moving image and sound information in a recording medium or a cloud storage.

[0104] (Other exemplary embodiments)

[0105] The various embodiments of the present disclosure can be implemented in the following manner: Software (program) that can implement the functions of the above-described exemplary embodiments is provided to a system or device via a network or various storage media, and a computer (or a CPU, a microprocessor unit (MPU), etc.) of the system or device reads and executes the program. In this case, the program and the storage medium storing the program constitute the present disclosure.

[0106] According to an exemplary embodiment of the present disclosure, a user can appropriately select a sound for motion image shooting.

[0107] Other exemplary embodiments

[0108] The various embodiments of the present disclosure can also be implemented by a computer of a system or device that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the above-described (one or more) embodiments and / or includes one or more circuits (e.g., an application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described (one or more) embodiments, and by a method executed by the computer of the system or device that reads and executes the computer-executable instructions from a storage medium, for example, to perform the functions of one or more of the above-described (one or more) embodiments and / or control one or more circuits to perform the functions of one or more of the above-described (one or more) embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)), and may include a network of individual computers or individual processors to read and execute the computer-executable instructions. The computer-executable instructions can be provided to the computer from a network or a storage medium, for example. The storage medium may include, for example, one or more of a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage device of a distributed computing system, an optical disc (such as a compact disc (CD), a digital versatile disc (DVD), or a Blu-ray disc (BD) TM ), a flash device, a memory card, etc.

[0109] The embodiments of the present disclosure can also be implemented by the following method, that is, software (program) that executes the functions of the above-described embodiments is provided to a system or device via a network or various storage media, and a method in which a computer or a central processing unit (CPU) or a microprocessor unit (MPU) of the system or device reads and executes the program.

[0110] Although the exemplary embodiments have been described, it should be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the claims should be given the broadest interpretation to encompass all such variations, equivalent structures, and functions.

Claims

1. An electronic device capable of communicating with a plurality of sound collection devices, the electronic device comprising: a display unit; an image acquisition unit configured to acquire an image captured by the imaging device from the imaging device; a position acquisition unit configured to acquire, for each of the plurality of sound collection devices, position information relative to the image captured by the imaging device; a display control unit configured to, for each of the plurality of sound collection devices, based on the position information of each sound collection device acquired by the position acquisition unit, control the display unit to display a display item corresponding to each sound collection device, and display the image acquired by the image acquisition unit together with the plurality of displayed display items, wherein the display control unit is configured to, for each sound collection device, control the display of the corresponding display item even if the display item corresponds to a sound collection device not within the imaging range of the acquired image; and a control unit configured to control to acquire sound information from the sound collection device selected by the user from the plurality of sound collection devices, and control to associate and record the acquired image and the acquired sound information in a recording device.

2. The electronic device according to claim 1, wherein, each of the display items corresponding to the sound collection devices is at least one of the following: an icon of the respective sound collection device, a display item regarding the respective sound information, or a display item indicating the position direction of the respective sound collection device in the case where the respective sound collection device is not within the imaging range of the acquired image.

3. The electronic device according to claim 1, wherein, the control unit controls the display unit to display the display items corresponding to the sound collection devices within the imaging range of the acquired image and the display items corresponding to the sound collection devices not within the imaging range of the acquired image in different display modes.

4. The electronic device according to claim 1, wherein, the display items corresponding to the sound collection devices within the imaging range of the acquired image and the display items corresponding to the sound collection devices not within the imaging range of the acquired image differ in color, size, or transmittance in the display mode.

5. The electronic device according to claim 1, wherein, the control unit controls the display unit to display a direction item as the display item corresponding to the sound collection device not within the imaging range of the acquired image, the direction item indicating the position direction of the sound collection device not within the imaging range of the acquired image relative to the imaging direction of the imaging device.

6. The electronic device according to claim 1, wherein, in the case where the electronic device includes one sound collection device, the control unit controls the display unit to display a built-in sound collection item indicating the sound collection device configured in the electronic device.

7. The electronic device according to claim 1, wherein, The control unit changes the display mode of each display item displayed for the plurality of sound collection devices based on the states of the respective sound collection devices corresponding to each display item.

8. The electronic device according to claim 7, wherein, the state of each sound collection device is at least one of the following states: the communication connection state between each sound collection device and the electronic device, and the state of the sound collected by each sound collection device.

9. The electronic device according to claim 1, wherein, when the user selects at least one of the sound collection devices before the start of recording, the control unit outputs the sound information collected by the selected sound collection device from the electronic device without recording the sound information in the recording device, and when the user selects at least one of the sound collection devices after the start of recording, the control unit records the sound information collected by the selected sound collection device in the recording device without outputting the sound information from the electronic device.

10. The electronic device according to claim 1, wherein, the control unit controls the display unit to display the display item corresponding to each of the plurality of sound collection devices together with the image captured by the imaging device.

11. The electronic device according to claim 1, further comprising an imaging unit, wherein, the image acquisition unit acquires the image captured by the imaging unit.

12. The electronic device according to claim 1, wherein, the image acquisition unit communicates with the imaging device through wired communication or wireless communication to acquire the image captured by the imaging device, and the imaging device is a device different from the electronic device.

13. A method for controlling an electronic device capable of communicating with a plurality of sound collection devices, the method comprises: performing display; acquiring the image captured by the imaging device from the imaging device; for each of the plurality of sound collection devices, acquiring the position information with respect to the image captured by the imaging device; performing control to, for each of the plurality of sound collection devices, display the display item corresponding to each sound collection device based on the acquired position information of each respective sound collection device, and display the acquired image together with the plurality of displayed display items; performing control to, for each sound collection device, display the corresponding display item even if the display item corresponds to a sound collection device not within the imaging range of the acquired image; and performing control to acquire the sound information from the sound collection device selected by the user among the plurality of sound collection devices, and associating and recording the acquired image and the acquired sound information in the recording device.

14. A computer-readable recording medium storing a program for causing a computer to execute a method for controlling an electronic device capable of communicating with a plurality of sound collection devices, the method comprises: performing display; acquiring the image captured by the imaging device from the imaging device; for each of the plurality of sound collection devices, acquiring the position information with respect to the image captured by the imaging device; Control is performed to display, for each of the plurality of sound collection devices, a display item corresponding to each sound collection device based on the respective position information of each acquired sound collection device, and to display the acquired image together with the plurality of displayed display items; Control is performed to display, for each sound collection device, the corresponding display item even if the display item corresponds to a sound collection device not within the imaging range of the acquired image; And Control is performed to obtain sound information from the sound collection device selected by the user from the plurality of sound collection devices, and to associate and record the acquired image and the acquired sound information in the recording device.

Images