Imaging device, control method thereof, program, and storage medium

Abstract

To provide an imaging device capable of capturing moving images in which the position of an object is associated with its sound without requiring a large-scale system.SOLUTION: An imaging device includes an imaging unit, an audio acquisition unit having a plurality of microphones arranged facing different directions, an object detection unit that detects an object included in the image acquired by the imaging unit and detects the coordinates of the detected object on the image and the direction of the detected object relative to the imaging device, an extraction unit that extracts audio corresponding to the direction of the object relative to the imaging device from the audio acquired by the audio acquisition unit on the basis of the direction of the object relative to the imaging device, and a recording unit that associates the audio extracted by the extraction unit with the coordinates of the object on the image to be recorded together with the image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an imaging device capable of capturing moving images including audio. [Background technology]

[0002] Conventionally, audio in video files has generally been either monaural or stereo.

[0003] In recent years, imaging devices have been developed that can capture video in all directions and record audio using multiple microphones so that the audio can be played back in accordance with the direction the user is facing, thereby achieving stereophonic sound.

[0004] For example, Patent Document 1 discloses the following technology. That is, sound is recorded in all directions around the camera, the image range is divided into multiple regions, and sound corresponding to each divided region is extracted. Then, sound is recorded in association with the position of the divided region, and when playing back, sound is played back at a position corresponding to the direction the user is facing.

[0005] Also, a technology has been proposed in which sound and position information is recorded for each object on an image, and the sound level for each object is adjusted during playback, thereby reproducing sound with a three-dimensional feel. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2017-201743 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, the technology disclosed in the above-mentioned Patent Document 1 divides areas in a specified direction and plays back audio, so it is not possible to play back audio that corresponds to the more precise position and direction of each object in the captured image in order to create a more three-dimensional sound effect.

[0008] In addition, in order to record the sound of each object, a microphone must be placed on each object, which requires a large-scale system, making it difficult for general users to use this technology.

[0009] The present invention has been made in consideration of the above-mentioned problems, and its object is to provide an imaging device that can capture video that associates the position of an object with its sound without requiring a large-scale system. [Means for solving the problem]

[0010] The imaging device of the present invention is characterized in that it comprises an imaging means, an audio acquisition means having a plurality of microphones arranged facing different directions from each other, an object detection means for detecting an object included in an image acquired by the imaging means and detecting the coordinates of the detected object on the image and the direction of the detected object relative to the imaging device, an extraction means for extracting audio corresponding to the direction of the object relative to the imaging device from the audio acquired by the audio acquisition means based on the direction of the object relative to the imaging device, and a recording means for associating the audio extracted by the extraction means with the coordinates of the object on the image and recording it together with the image. Effect of the Invention

[0011] According to the present invention, it is possible to capture video in which the position of an object is associated with sound without requiring a large-scale system. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing the configuration of an imaging apparatus according to a first embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram showing the configuration of a voice recording system. [Diagram 3] FIG. 1 is a diagram showing a specific example of the configuration of a stereophonic microphone. [Figure 4]FIG. 2 is a diagram showing the relationship between an imaging device and an object. [Diagram 5] FIG. 13 is a diagram illustrating the separation of audio for each object and the allocation of coordinate data. [Figure 6] FIG. 13 is a diagram illustrating the results of separating sounds for each detected object. [Figure 7] 5 is a flowchart showing the operation of the imaging apparatus according to the first embodiment. [Figure 8] 5A and 5B are diagrams for explaining a method of calculating the direction of an object with respect to an imaging device. [Figure 9] 10 is a flowchart showing the operation of an imaging apparatus according to a second embodiment. [Figure 10] 11A and 11B are explanatory diagrams relating to determination of correction of sound pressure for each object. [Figure 11] FIG. 4 is a diagram showing a table used for correcting sound pressure. [Figure 12] 1A and 1B are explanatory diagrams illustrating a case where an object moves within and outside the angle of view range of an imaging device. [Figure 13] 10 is a flowchart showing the operation of an imaging apparatus according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] (First embodiment) 1 is a block diagram showing the configuration of an image capturing apparatus 1 according to a first embodiment of the present invention. In FIG. 1, the image capturing apparatus 1 includes an image capturing apparatus body 100 and an imaging optical system 150.

[0015] The photographing optical system 150 includes a lens 151 including a zoom unit, an aperture / shutter unit, and a focus unit. The imaging unit 104 includes an imaging element such as a CMOS sensor or a CCD, and performs photoelectric conversion on an optical image formed by the photographing optical system 150 to output an electrical signal. The lens actuator control unit 153 includes a motor driver IC, and drives various actuators such as the zoom unit, aperture / shutter unit, and focus unit of the lens 151. The various actuators are driven based on actuator drive instruction data from the central control unit 101, which will be described later.

[0016] The sound collection unit (sound acquisition means) 108 is a sound input unit including a microphone (hereinafter, referred to as a microphone). The sound collection unit 108 includes a plurality of microphones, converts a sound signal into an electrical signal, and further converts it into a digital signal for output.

[0017] A central control unit (CPU) 101 controls the entire imaging device 1. An imaging signal processing unit 105 converts an electric signal output from the imaging unit 104 into an image signal for a still image or a moving image. An image signal processing unit 106 performs a predetermined process for recording or display on the image signal output from the imaging signal processing unit 105. The image signal processing unit 106 also performs processing on the image signal according to the application. The processing of the image signal includes electronic image stabilization processing by image cropping and rotation processing, and subject detection processing for detecting a subject. An audio signal processing unit 107 performs audio processing on an audio signal acquired from a sound collection unit 108.

[0018] The operation unit 111 is provided for operating the imaging device 1. The storage unit 112 stores various data such as image signals obtained by shooting, audio data collected simultaneously with shooting of the image signals, and attribute information of objects detected by subject detection processing in association with each other. It also stores a control program executed by the central control unit 101.

[0019] The display unit 113 includes a display such as an LCD, and displays images as necessary based on a signal output from the image signal processing unit 106. The external input / output terminal unit 114 inputs and outputs communication signals and video signals to and from an external device. The audio playback unit 110 includes a speaker, and plays audio based on audio data.

[0020] The power supply unit 115 supplies power according to the purpose to the entirety (each element) of the imaging device 1. The power supply control unit 116 controls the individual start-up and shut-down of different types of power supplies of the power supply unit 115.

[0021] The microphone of the sound collection unit 108 is built into the imaging device body 100, but may also be a wired cable microphone connectable by, for example, a phone plug, and outputs the collected sound as an analog electrical signal. In the case of a wired cable microphone, an analog sound signal is input to the sound signal processing unit 107 via a microphone jack mounted on the imaging device 1. Hereinafter, in this embodiment, the sound collection unit 108 will be described as including a microphone built into the imaging device body 100.

[0022] FIG. 2 is a block diagram showing the configuration of the main parts of the imaging device 1, including the stereophonic microphone 200 and the object sound generator 205. As shown in FIG.

[0023] The sound pickup unit 108 is a microphone built into the imaging device body 100, but may also be configured as a wired cable microphone connectable by, for example, a phone plug, and outputs the picked-up sound as an analog electric signal. In the case of a wired cable microphone, an analog sound signal is input to the sound signal processing unit 107 via a microphone jack mounted on the imaging device 1. Hereinafter, in this embodiment, the sound pickup unit 108 will be described as a microphone built into the imaging device body 100.

[0024] FIG. 2 is a block diagram showing the configuration of a sound recording system including a stereophonic microphone 200 and an object sound generator 205. As shown in FIG.

[0025] The stereophonic microphone 200 is included in the sound collection unit 108 in Fig. 1. The object detection unit 201 and the object coordinate calculation unit 204 are included in the image signal processing unit 106 in Fig. 1. The object sound generation unit 205 and the video generation unit 206 are included in the central control unit 101 in Fig. 1.

[0026] The stereophonic microphone 200 will be described below. The stereophonic microphone 200 is configured with a plurality of microphones having directivity. The plurality of microphones face different directions, and each microphone acquires audio data in the direction in which it faces. Generally, the stereophonic microphone 200 is configured with a plurality of microphones, but in this embodiment, the stereophonic microphone 200 will be described as having four microphones.

[0027] Fig. 3 is a diagram showing a specific example of the configuration of the stereophonic microphone 200. As shown in Fig. 3(a), the microphone 200 has four microphones, namely, microphone A, microphone B, microphone C, and microphone D. Each of the microphones is assumed to be a directional microphone (digital output microphone) having an A / D conversion function. The four microphones A to D each have a directivity of 45° in the up, down, left, and right directions, and are arranged at angular intervals of 90° from each other.

[0028] More specifically, the microphones A to D are provided so as to face the four vertices of a cube shown in Fig. 3. If the space in the figure is defined by the directions shown in Fig. 3(b), microphone A faces the front upper left, microphone B faces the front lower right, microphone C faces the rear lower left, and microphone D faces the rear upper right. The four microphones A to D acquire sound components in their respective directional directions, which makes it possible to collect sound data from 360°. Note that, although this embodiment will be described taking digital output microphones as an example, analog output microphones may also be used.

[0029] Returning to the explanation of Figure 2, the audio signal processing unit 107 performs audio processing for video audio, such as various filter processing, wind cut processing, stereo enhancement processing, drive sound cancellation processing, ALC (Auto Level Control) processing, and compression processing, on the audio data obtained from the stereophonic microphone 200.

[0030] The imaging unit 104 captures an object image formed by the lens 151 in FIG. 1 and outputs image data. When recording a video together with an audio signal from the stereophonic microphone 200, the imaging unit 104 outputs image data of the video. The object detection unit 201 performs an object detection process to detect an object included in each frame of the image data output from the imaging unit 104. The object coordinate calculation unit 204 is configured with an object direction calculation unit 202 and an object distance calculation unit 203. The object direction calculation unit 202 detects the direction in which an object included in each frame of the image data detected by the object detection unit 201 exists. The object distance calculation unit 203 detects the distance from the imaging device 1 to the object based on the size of the object detected by the object detection unit 201, focal length information obtained from the lens 151 in FIG. 1, phase difference information in autofocus processing, and the like.

[0031] The object voice generation unit 205 extracts object voice data, which is voice data for each object, from the voice data acquired by the voice signal processing unit 107 by specifying the direction in which the object acquired by the object coordinate calculation unit 204 exists, and assigns the object voice data to each object. Furthermore, the object voice generation unit 205 generates, for each object, the object voice data and the coordinate data of the object to which the object voice data is assigned, as one piece of object data.

[0032] The video generation unit 206 generates video data by combining the object data generated by the object sound generation unit 205 and the image data acquired by the imaging unit 104. The recording medium 207 records the generated video data.

[0033] Fig. 4 is a diagram showing the relationship between the imaging device and the object. In Fig. 4, coordinates 302, 304 of the imaging device 1 and object A 305 and object B 306 corresponding to times t1, t2, and t3 after the start of video capture are shown. Also shown are screens of the video captured by the imaging unit 104 of the imaging device 1, and waveforms of background sound 300 and sounds 301, 303 of each object, corresponding to times t1, t2, and t3.

[0034] At time t1, the coordinates 302, 304 of object A 305 and object B 306 are (x1, y1, z1) and (x2, y2, z2), respectively, and object A and object B are not making any sound. Note that the background sound 300 is handled as individual sound data in order to record sounds other than those determined as objects.

[0035] At time t2, object A 305 is emitting a sound, and a voice 301 is extracted. Furthermore, object B 306 is not emitting a sound, but the coordinates 304 have changed to (x3, y3, z3), and object B has come closer to the imaging device 1, so that the area that object B occupies on the screen has become larger.

[0036] At time t3, there is no change in object A 305. Also, object B 306 is emitting a sound, and a sound 303 is extracted as object sound data.

[0037] Next, the operation of the imaging device 1 of this embodiment will be described with reference to Fig. 5 to Fig. 7. Fig. 5 is a diagram showing an image captured by the imaging device 1, Fig. 6 is a diagram showing subjects in the image, and Fig. 7 is a flowchart showing the operation of the imaging device 1. The operation of the flowchart in Fig. 7 is realized by the central control unit 101 executing a control program stored in the storage unit 112.

[0038] First, in step S1001 of FIG. 7, when the user turns on the power of the imaging device 1, the central control unit 101 advances the process to step S1002.

[0039] In step S1002, the central control unit 101 initializes the entire device so that the imaging device 1 is capable of capturing moving images.

[0040] In step S1003, the central control unit 101 determines whether or not the user has instructed video shooting through the operation unit 111 provided in the imaging device 1. If video shooting has been instructed, the central control unit 101 proceeds to step S1004, and if not, the central control unit 101 waits.

[0041] In step S1004, the central control unit 101 determines whether or not an object has been detected by the object detection unit 201 from the video captured by the imaging unit 104. If an object has been detected, the central control unit 101 advances the process to step S1005, and if not, the central control unit 101 advances the process to step S1008.

[0042] For example, if the image captured by the imaging unit 104 is an image as shown in Fig. 5(a), the object detection unit 201 detects a person 401, a person 402, and a dog 403 in Fig. 5(b) as objects. Also, if the same type of object exists in the same place, such as a dog 404 in Fig. 5(d), they may be treated as a single object.

[0043] In step S1005, the central control unit 101 calculates the position coordinates in the image of each object detected in step S1004 using the object coordinate calculation unit 204. Furthermore, the central control unit 101 calculates the direction of each object with respect to the imaging device 1 from the position coordinates of each object and information on the focal length of the lens 151.

[0044] For example, the object coordinate calculation unit 204 calculates the coordinates of the center position of each object in each screen of the captured video, as shown in Fig. 5(c). Fig. 5(c) shows that the coordinates on the image of person 401 are A(xa,ya), the coordinates of person 402 are B(xb,yb), and the coordinates of dog 403 are C(xc,yc). The method of calculating the direction of each object with respect to the imaging device 1 using the coordinates of each object and information on the focal length of lens 151 will be described later in detail.

[0045] In step S1006, the central control unit 101 separates the sounds of each object from the multiple sound data (stereophonic data) acquired by the sound signal processing unit 107 using the object sound generation unit 205, based on the direction in which each object exists calculated in step S1005.

[0046] As a method for separating the sound of each object from a plurality of sound data (stereophonic sound data) using the direction in which each object exists, various methods are known, such as a method implemented by commonly known Ambisonics, beamforming, etc. In addition, in order to improve the accuracy of sound separation, a sound filter that matches the sound frequency characteristics of each object detected in step S1004 may be used.

[0047] In step S1007, the central control unit 101 uses the video generating unit 206 to associate the sound data separated for each object in step S1006 with the coordinates for each object calculated in step S1005 as one set of data.

[0048] For example, as shown in FIG. 6, if the coordinates of a person 401 are A(xa, ya) and the audio separated based on the direction calculated from those coordinates is audio data 501, then the coordinates A(xa, ya) and the audio data 501 are associated as one set of data.

[0049] Also, if the coordinates of person 402 are B(xb, yb) and the audio separated based on the direction calculated from those coordinates is audio data 502, then the coordinates B(xb, yb) and the audio data 502 are associated as one set of data.

[0050] Similarly, if the coordinates of dog 403 are C(xc, yc) and the sound separated based on the direction calculated from those coordinates is audio data 503, then the coordinates C(xc, yc) and the audio data 503 are associated as one set of data.

[0051] The following is an example of using an audio filter based on the frequency characteristics of an object when separating audio based on a direction calculated from coordinates.

[0052] For example, if the person 401 is a man, a bandpass filter that passes the frequency band of male voices may be used, and if the person 402 is a woman, a bandpass filter that passes the frequency band of female voices may be used.

[0053] In step S1008, the central control unit 101 determines whether or not the user has instructed to end video shooting via the operation unit 111. If an instruction to end shooting has been given, the central control unit 101 proceeds to step S1009, and if not, returns the process to step S1004.

[0054] In step S1009, the central control unit 101 associates the coordinates and audio data of each object in step S1007 as object data for each object, creates a single video file containing each object data, and records it on the storage medium 207.

[0055] Next, Fig. 8 is a diagram for explaining a method for calculating the direction in which an object exists. Using Fig. 8, a method for calculating the direction of an object with respect to the imaging device 1 from the coordinates of the object in the captured image and the focal length of the lens 102 will be explained.

[0056] 8(a), 601 denotes an object plane on which an object exists, 602 denotes an imaging surface of an imaging element of the imaging unit 104, 603 denotes an object on the object plane 601, and 604 denotes an object image formed by projecting the object 603 onto the imaging surface 602. 605 denotes the focal point of the lens 151, and the distance from the focal point 605 to the imaging surface 602 is the focal length.

[0057] FIG. 8B is a diagram in which FIG. 8A is redrawn so that the line connecting the object plane 601, the imaging plane 602, and the focal point 605 is horizontal.

[0058] 8(b), 610 denotes a line connecting the center of the object plane 601 and the center of the imaging surface 602, 606 denotes a line connecting the object 603, the focal point 605, and the object image 604, and 607 denotes the angle between the line 610 and the line 606. Furthermore, 608 denotes the distance from the center of the imaging surface 602 to the focal point 605, and 609 denotes the distance from the center of the imaging surface 602 to the object image 604. The distance 608 is the focal length. Furthermore, since the pixel pitch of the imaging element is known, the distance 609 can be calculated as the number of pitches of the object image 604 from the center of the imaging element.

[0059] The direction of the object to be found relative to the imaging device 1 is an angle 607, which can be calculated from distances 608 and 609. If the direction of the object is calculated when the subject plane 601 and the imaging plane 602 are viewed from the left-right direction and when viewed from the top-bottom direction, the left-right direction and the top-bottom direction of the object relative to the imaging device 1 can be calculated.

[0060] As described above, according to the above embodiment, it is possible to record objects included in a captured video in association with their sounds, without placing a microphone for each object.

[0061] Second Embodiment Next, a second embodiment of the present invention will be described. In this second embodiment, the configuration of the imaging device and the configuration of the audio recording system are similar to those of the first embodiment shown in Fig. 1 and Fig. 2, so the description will be omitted.

[0062] Fig. 9 is a flowchart showing the operation of the imaging device 1 in the second embodiment. Note that the operation of the flowchart in Fig. 9 is realized by the central control unit 101 executing a control program stored in the storage unit 112.

[0063] First, in step S1101 of FIG. 9, when the user turns on the power of the imaging device 1, the central control unit 101 advances the process to step S1102.

[0064] In step S1102, the central control unit 101 initializes the entire device so that the imaging device 1 is capable of capturing moving images.

[0065] In step S1103, the central control unit 101 determines whether or not the user has instructed video shooting through the operation unit 111 provided in the imaging device 1. If video shooting has been instructed, the central control unit 101 proceeds to step S1104, and if not, the central control unit 101 waits.

[0066] In step S1104, the central control unit 101 determines whether or not an object has been detected by the object detection unit 201 from the video captured by the imaging unit 104. If an object has been detected, the central control unit 101 advances the process to step S1105, and if not, the central control unit 101 advances the process to step S1111.

[0067] In step S1105, the central control unit 101 calculates the position coordinates in the image of each object detected in step S1104 using the object coordinate calculation unit 204. Furthermore, the central control unit 101 calculates the direction of each object with respect to the imaging device 1 from the position coordinates of each object and information on the focal length of the lens 151.

[0068] In step S1106, the central control unit 101 separates the sounds of each object from the multiple sound data (stereophonic data) acquired by the sound signal processing unit 107 using the object sound generation unit 205, based on the direction in which each object exists calculated in step S1105.

[0069] As a method for separating the sound of each object from a plurality of sound data (stereophonic sound data) using the direction in which each object exists, various methods are known, such as a method implemented by commonly known Ambisonics, beamforming, etc. In addition, in order to improve the accuracy of sound separation, a sound filter that matches the sound frequency characteristics of each object detected in step S1104 may be used.

[0070] In step S1107, the central control unit 101 uses the object distance calculation unit 203 to detect the distance from the imaging device 1 to each of the objects detected in step S1104.

[0071] In step S1108, the central control unit 101 determines whether or not the sound of each object is to be corrected for sound pressure based on the distance detected in step S1107. If the sound is to be corrected for sound pressure, the central control unit 101 advances the process to step S1109, and if not, the central control unit 101 advances the process to step S1110.

[0072] In step S1109, the central control unit 101 performs sound pressure correction on the sound data separated for each object in step S1106. The details of the correction will be described later.

[0073] In step S1110, the central control unit 101 uses the moving image generating unit 206 to associate, as one set of data, the sound data separated for each object in step S1106 and whose sound pressure has been corrected, with the coordinates of each object calculated in step S1105.

[0074] In step S1111, the central control unit 101 determines whether or not the user has instructed to end video shooting via the operation unit 111. If an instruction to end shooting has been given, the central control unit 101 proceeds to step S1112, and if not, returns the process to step S1104.

[0075] In step S1112, the central control unit 101 associates the coordinates and audio data of each object in step S1110 as object data for each object, creates a single video file including each object data, and records it on the storage medium 207.

[0076] Next, a process of correcting sound pressure according to an object distance from the imaging device 1 will be described with reference to Fig. 10 and Fig. 11. Fig. 10 is a diagram showing the distance between the imaging device 1 and an object, and Fig. 11 is a diagram showing an example of correction data for sound pressure.

[0077] In general, in a system for recording audio to realize stereophonic sound, a microphone is assigned to each object, and audio data is acquired. Therefore, conditions such as the distance between the microphone and the object are often standardized to a certain extent for each object. In contrast, in this embodiment, a single microphone provided in an imaging device is used to acquire the audio of each object. Therefore, the distance from the microphone to each object differs for each object. Therefore, the sound pressure of the audio data is corrected according to the distance between the object and the microphone.

[0078] 10, 700 indicates the positional relationship between the imaging device 1 and each object when viewed from above the imaging device 1. Also, 701 indicates an image captured by the imaging unit 104 of the imaging device 1 in the positional relationship indicated by 700.

[0079] Reference numeral 702 denotes the angle of view of an image captured by the imaging unit 104 of the imaging device 1 through the lens 151, and the range of application of sound pressure correction. The range of application of sound pressure correction will be described later. Reference numerals 703, 704, 705, and 706 denote objects to be detected by the object detection unit 201. Here, object 704 is in the blind spot of object 703 as seen from the imaging device 1, as shown in 701, and is an object that cannot be detected by the object detection unit 201.

[0080] Auxiliary lines 707, 708, and 709 indicate the directions of objects 703, 705, and 706 detected by the object detection unit 201. The imaging device 1 then performs sound pressure correction corresponding to the distances on the auxiliary lines 707, 708, and 709 for the objects 703, 705, and 706 detected by the object detection unit 201. The sound pressure correction is performed by referring to table data previously recorded in memory. This process will be described with reference to FIG. 11.

[0081] Table 800 in Fig. 11 shows an example of table data stored in memory. Here, the frequency characteristics refer to frequency characteristics of correction data used for sound pressure correction, which corrects the difference in sound pressure of the audio data acquired by stereophonic microphone 200 according to distance.

[0082] Generally, the farther the distance to the microphone, the lower the sound pressure of the acquired voice data. Also, the attenuation of sound pressure according to distance is greater for higher frequencies than for lower frequencies. Therefore, the frequency characteristic 802 applied for correction at a distance of 2 m has a larger sound pressure correction value than the frequency characteristic 801 applied at a distance of 1 m, and the correction value is greater at higher frequencies. In other words, the correction value increases as the distance increases. And in table 800, the correction value of frequency characteristic 803 applied at a distance of Nm is the largest.

[0083] Next, the application range of sound pressure correction will be described. As described above, 702 in Fig. 10 indicates the angle of view of the image captured by the imaging unit 104 of the imaging device 1 via the lens 151 and the application range of sound pressure correction (predetermined threshold value).

[0084] In the sound pressure correction, sound pressure correction processing is performed for each object using frequency characteristics according to the distance to the object. However, there is a problem that the greater the distance, the greater the correction value of the sound pressure, and the greater the correction value, the greater the noise. Here, noise often refers to noise that is picked up at a minimum level, such as floor noise or background noise. Therefore, if it is desired to make the noise in the recorded voice less noticeable, there is a distance limit to the sound pressure correction in the object voice generation unit 205. The ideal distance limit is a range in which the user cannot hear the sound pressure of the noise in the voice data for each object after correction.

[0085] 10, object 706 is farther away from image capture device 1 than objects 703, 704, and 705, and is located outside the effective range of sound pressure correction described above. Therefore, based on the distance of object 706 calculated by object distance calculation unit 203, object 706 is determined to be outside the applicable range of sound pressure correction, and sound correction is not performed. As a result, the sound emitted by object 706 is recorded on recording medium 207 as background sound.

[0086] As described above, according to this embodiment, it is possible to capture a moving image in which the sound of each object is associated with that of the other object without disposing a microphone for each object. Also, it is possible to appropriately correct the volume of the sound for each object.

[0087] (Third embodiment) FIG. 12 is a diagram showing how to find coordinates when an object moves between the inside and outside of the imaging angle range of the imaging device 1. In FIG.

[0088] 12 shows how to express coordinates within the angle of view range and coordinates outside the angle of view range when the imaging device 1 is at the center. An area 908 is within the angle of view range and an area 909 is outside the angle of view range, with an angle of view boundary line 907 as the boundary. Within the angle of view range, coordinate data according to the position of a detected object is assigned, and outside the angle of view range, the coordinates of a ● (center coordinates), which is the center position of an area divided into fixed areas, are assigned.

[0089] For example, we will explain the case where a person 901 within the field of view moves from coordinates (x10, y10, z10) to coordinates (x11, y11, z11), and then from an object-undetected point 902 (x12, y12, z12) in the direction of the arrow to 903 outside the field of view.

[0090] Within the angle of view, the coordinates assigned to the person 901 are (x10, y10, z10), (x11, y11, z11), and (x12, y12, z12) along the trajectory of the person 901's movement.

[0091] Furthermore, if the person 901 continues to emit the same type of sound outside the field of view as that emitted within the field of view, the voice data is continuously associated with the area coordinates outside the field of view (the coordinates of the destination) during that time. The area outside the field of view is determined based on the direction in which the person 901 moved from the object non-detection point 902 to the outside of the field of view. In the example of person 901, as shown in the figure, the outside field of view area coordinates (x92, y92, z92) are assigned to the voice data.

[0092] On the other hand, when a person 904 outside the angle of view moves in the direction of the arrow from object detection point 905 (x13, y13, z13) to 906 within the angle of view, the coordinate data to be assigned is as follows: starting from object detection point 905 (x13, y13, z13), (x14, y14, z14) and (x15, y15, z15) are assigned.

[0093] Also, when the person 904 enters the angle of view from the object detection point 905, it is confirmed from which direction the person 904 moved into the angle of view, and it is determined from which area outside the angle of view the person moved. In the example of the person 904, as shown in the figure, it is determined that the person moved from outside the angle of view area coordinates (x95, y95, z95). Also, if the person 904 emits the same type of sound as the sound emitted within the angle of view at outside the angle of view area coordinates (x95, y95, z95) going back in time, the outside the angle of view area coordinates (x95, y95, z95) are associated with the sound data.

[0094] Also, in FIG. 12, the description is given on the assumption that the detected object is a person. In this case, since it is assumed that the person does not move significantly, the coordinates of the area near the field of view boundary line 907 are assigned to the coordinates when the object moves outside the field of view. However, when the detected object is an object that moves significantly inside and outside the field of view, such as a train, a car, or an airplane, the area near the field of view boundary line 907 in FIG. 9 and the area behind it are assigned to the coordinate area outside the field of view. By controlling in this way, audio data with a more three-dimensional feel can be obtained. Also, in FIG. 12, an example of a coordinate area on a plane is shown, but coordinates indicating a three-dimensional position may be calculated and assigned.

[0095] Fig. 13 is a flowchart showing the operation of the imaging device 1 in the third embodiment. Note that the operation of the flowchart in Fig. 13 is realized by the central control unit 101 executing a control program stored in the storage unit 112.

[0096] First, in step S1201 of FIG. 13, when the user turns on the power of the imaging device 1, the central control unit 101 advances the process to step S1202.

[0097] In step S1202, the central control unit 101 initializes the entire device so that the imaging device 1 is capable of capturing moving images.

[0098] In step S1203, the central control unit 101 determines whether or not the user has instructed video shooting through the operation unit 111 provided in the imaging device 1. If video shooting has been instructed, the central control unit 101 proceeds to step S1204, and if not, the central control unit 101 waits.

[0099] In step S1204, the central control unit 101 determines whether or not an object has been detected by the object detection unit 201 from the video captured by the imaging unit 104. If an object has been detected, the central control unit 101 advances the process to step S1205, and if not, the central control unit 101 advances the process to step S1216.

[0100] In step S1205, the central control unit 101 determines whether an object has been detected within the angle of view, rather than moving from outside the angle of view. If an object has been detected within the angle of view, the central control unit 101 proceeds to step S1206, otherwise, the central control unit 101 proceeds to step S1207.

[0101] In step S1206, the central control unit 101 calculates the position coordinates in the image of the object detected in step S1204 using the object coordinate calculation unit 204. Furthermore, the central control unit 101 calculates the direction of the object with respect to the imaging device 1 from the position coordinates of the object and information on the focal length of the lens 151.

[0102] 12, when a person 901 is detected on an image, the coordinates of the center position of the object are calculated, such as (x10, y10, z10). Then, the direction of the person 901 with respect to the imaging device 1 is calculated from the coordinates and the focal length of the lens 151.

[0103] In step S1207, the central control unit 101 detects the coordinates of the point where the object has entered the angle of view (object detection point 905 in FIG. 12), and the moving direction and moving speed of the object.

[0104] In step S1208, the central control unit 101 separates the sounds of each object from the multiple sound data (stereophonic data) acquired by the sound signal processing unit 107 using the object sound generation unit 205, based on the direction in which each object exists calculated in step S1206 or step S1207.

[0105] There are various known methods for separating the sound of each object from multiple audio data (stereophonic data) using the direction in which each object exists, such as the commonly known Ambisonic method and beamforming.

[0106] In step S1209, the central control unit 101 uses the video generating unit 206 to associate the sound data separated for each object in step S1208 with the coordinates for each object calculated in step S1207 as one set of data.

[0107] In step S1210, the central control unit 101 determines whether or not the user has instructed to end video shooting via the operation unit 111. If an instruction to end shooting has been given, the central control unit 101 proceeds to step S1211, and if not, returns the process to step S1204.

[0108] In step S1211, the central control unit 101 determines whether or not an object has moved from outside the angle of view to inside the angle of view between the start of video shooting in step S1203 and the end instruction in step S1210. If there has been no movement, the central control unit 101 advances the process to step S1212, and if there has been movement, the central control unit 101 advances the process to step S1213.

[0109] In step S1212, the central control unit 101 treats the data obtained in step S1209, which associates the coordinates of each object with the audio data, as object data for each object, creates a single video file including each object data, and records it on the storage medium 207.

[0110] In step S1213, the central control unit 101 determines whether or not the object that has moved from outside the angle of view into the angle of view has emitted the same sound immediately before moving into the angle of view. If the object has emitted the same sound, the central control unit 101 proceeds to step S1214, and if not, the central control unit 101 proceeds to step S1212.

[0111] In step S1214, the central control unit 101 predicts the position of the object outside the angle of view before the movement from the coordinates when the object entered the angle of view from outside the angle of view acquired in step S1207 and the direction and speed of the object's movement after that, and determines a coordinate area outside the angle of view. Then, the central control unit 101 extracts the same sound data of the object in the direction of the coordinate area.

[0112] In step S1215, the central control unit 101 associates the coordinate area outside the angle of view with the audio data extracted in step S1214 during the time period going back from the timing when the object entered the angle of view from outside the angle of view in step S1207 before the end of video shooting. The association time is set to the time during which the same audio of the object can be detected.

[0113] In step S1216, the central control unit 101 determines whether an object that was detected and recognized as an object during video shooting has moved from within the angle of view to outside the angle of view and become undetected. If the object has moved from within the angle of view to outside the angle of view, the central control unit 101 proceeds to step S1217; otherwise, the central control unit 101 proceeds to step S1210.

[0114] The process may proceed to step S1210 if, in step S1216, no object has been detected since the start of video capture, or if an object that was detected during video capture is no longer detectable because it is hidden by the shadow of another object within the angle of view, etc. In this case, the object coordinate data and audio data are not updated, and the process proceeds to step S1210.

[0115] In step S1217, the central control unit 101 detects the coordinates of the location where the recognized object has moved outside the angle of view when the recognized object has moved from within the angle of view to outside the angle of view, as well as the direction and speed of movement.

[0116] In step S1218, the central control unit 101 determines whether or not there is voice data of the same object in the coordinate area direction to which the object that has moved out of the angle of view has moved. If there is voice data of the same object, the central control unit 101 proceeds to step S1219, and if the object has moved out of the angle of view without emitting a voice, the central control unit 101 proceeds to step S1221. Also, if voice data of the same object was once detected even outside the angle of view, but voice is no longer detected, the central control unit 101 proceeds to step S1222.

[0117] In step S1219, the central control unit 101 extracts sound data outside the angle of view of the recognized object.

[0118] In step S1220, the central control unit 101 associates the out-of-view angle coordinate data of the object that has moved outside the angle of view with the out-of-view angle sound data.

[0119] In step S1221, the central control unit 101 associates the coordinate data of the location outside the angle of view with the audio data.

[0120] In step S1222, the central control unit 101 associates the coordinate area of ​​the origin closest to the imaging device 1 with the sound data in all directions with respect to the object.

[0121] That is, in the process from step S1217 onwards, for an object that has moved outside the angle of view, the data of the outside-angle-of-view coordinate area continues to be associated with the outside-angle-of-view audio data while audio data of the same object continues.

[0122] Note that the above-described object detection from the start of video capture in step S1203 to the end of video capture in step S1210 and the control related to association of object coordinate data and audio data are performed at a constant cycle. For example, the object coordinate data is associated with audio data for each frame of the video frame rate.

[0123] In addition, in the above explanation, the imaging device 1 is fixed, and the coordinates of each object are described with the imaging device 1 at the center, but these coordinates may be either relative coordinates or absolute coordinates with the imaging device 1 at the center.

[0124] Similar control is also possible when the imaging device 1 moves. For example, the coordinates of the imaging device 1 at the time when the imaging device 1 starts capturing a video are calculated, and when the imaging device moves, the coordinates of an object detected from the moved position can be expressed as relative coordinates.

[0125] Furthermore, if multiple objects are present within the field of view, applying the same flowchart of FIG. 13 for each object makes it possible to deal with cases where multiple objects are detected at the same time.

[0126] In the embodiment of the present invention, a digital still camera is given as an example of an imaging device, but the present invention can also be applied to various devices having a moving image recording means.

[0127] The disclosure of this specification includes the following imaging device, its control method, program, and storage medium.

[0128] (Item 1) 1. An imaging device, comprising: An imaging means; A voice acquisition means having a plurality of microphones arranged facing in different directions; an object detection means for detecting an object included in an image acquired by the imaging means, and detecting coordinates of the detected object on the image and a direction of the detected object relative to the imaging device; an extraction means for extracting, from the sounds acquired by the sound acquisition means, a sound corresponding to a direction of the object with respect to the imaging device, based on a direction of the object with respect to the imaging device; a recording means for recording the voice extracted by the extraction means together with the image in association with the coordinates of the object on the image; An imaging device comprising:

[0129] (Item 2) 2. The imaging device according to item 1, wherein the object detection means calculates a direction of the object relative to the imaging device based on coordinates of the object on the image.

[0130] (Item 3) 3. The imaging device according to item 2, wherein the object detection means calculates a direction of the object relative to the imaging device based on the coordinates of the object on the image and a focal length of an imaging optical system of the imaging device.

[0131] (Item 4) 4. The imaging device according to any one of items 1 to 3, wherein the extraction means extracts the audio using an audio filter that limits a frequency band in accordance with the object detected by the object detection means.

[0132] (Item 5) 5. The imaging device according to any one of items 1 to 4, characterized in that the object detection means treats the multiple objects detected as a single object when the multiple objects are of the same type and have the same coordinates.

[0133] (Item 6) 6. The imaging device according to any one of items 1 to 5, wherein the recording means records a set of the sound extracted by the extraction means and the coordinates of the object on the image as one object.

[0134] (Item 7) The imaging device described in any one of items 1 to 6, characterized in that the object detection means further calculates a distance from the imaging device of the detected object, and the imaging device further comprises a correction means for correcting a sound pressure of a sound corresponding to the object in accordance with the distance.

[0135] (Item 8) 8. The imaging device according to item 7, wherein the correction means corrects the sound pressure using table data for correcting frequency characteristics of sound.

[0136] (Item 9) 8. The imaging device according to item 7, wherein the correction means does not correct the sound pressure of the sound corresponding to the detected object when the distance of the detected object from the imaging device is greater than a predetermined threshold.

[0137] (Item 10) The imaging device described in item 1, characterized in that when the detected object moves from within the angle of view range of the imaging device to outside the angle of view range and the detected object continues to emit the same type of sound as the sound it was emitting within the angle of view range, the recording means associates the sound with the center coordinates of a region separated by a certain size that is predicted to be the destination of the detected object outside the angle of view range.

[0138] (Item 11) The imaging device described in item 1, characterized in that when the detected object moves from within the angle of view range of the imaging device to outside the angle of view range and the detected object does not continue to emit the same type of sound as the sound it emitted within the angle of view range, the recording means associates the coordinates of the boundary between within the angle of view range and outside the angle of view range with the sound.

[0139] (Item 12) The imaging device described in item 1, characterized in that when the detected object moves from within the angle of view of the imaging device to outside the angle of view and the detected object does not continue to emit the same type of sound as it emitted within the angle of view, the recording means associates the coordinates of the imaging device with the sound in all directions.

[0140] (Item 13) The imaging device described in item 1, characterized in that when the detected object enters the angle of view range of the imaging device from outside the angle of view range and continues to emit the same type of sound as the sound it was emitting outside the angle of view range, the recording means associates the central coordinates of a certain-sized area from which the object outside the angle of view range is predicted to have moved, with the sound.

[0141] (Item 14) 14. The imaging device according to any one of items 1 to 13, wherein the plurality of microphones are arranged at predetermined angular intervals from each other.

[0142] (Item 15) Item 15. The imaging device according to item 14, wherein the multiple microphones are arranged at angular intervals of 90 degrees from each other.

[0143] (Item 16) A method for controlling an imaging device including an imaging means and an audio acquisition means having a plurality of microphones arranged facing in different directions, comprising: an object detection step of detecting an object included in an image acquired by the imaging means, and detecting coordinates of the detected object on the image and a direction of the detected object relative to the imaging device; an extraction step of extracting, from the sounds acquired by the sound acquisition means, a sound corresponding to a direction of the object with respect to the imaging device, based on a direction of the object with respect to the imaging device; a recording step of associating the voice extracted in the extraction step with coordinates on an image of the object and recording the sound together with the image; 13. A method for controlling an imaging apparatus comprising:

[0144] (Item 17) 17. A program for causing a computer to execute each step of the control method according to item 16.

[0145] (Item 18) A computer-readable storage medium storing a program for causing a computer to execute each step of the control method described in item 16.

[0146] (Other embodiments) The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0147] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0148] 1: imaging device, 100: imaging device main body, 101: central control unit, 104: imaging unit, 105: imaging signal processing unit, 106: image signal processing unit, 107: audio signal processing unit, 108: sound collection unit, 109: wireless unit, 110: audio playback unit, 111: operation unit, 112: storage unit, 113: display unit, 114: external input / output terminal unit, 115: power supply unit, 116: power supply control unit, 150: photographing optical system, 151: lens, 153: lens actuator control unit

Claims

1. 1. An imaging device, comprising: An imaging means; A voice acquisition means having a plurality of microphones arranged facing in different directions; an object detection means for detecting an object included in an image acquired by the imaging means, and detecting coordinates of the detected object on the image and a direction of the detected object relative to the imaging device; an extraction means for extracting, from the sounds acquired by the sound acquisition means, a sound corresponding to a direction of the object with respect to the imaging device, based on a direction of the object with respect to the imaging device; a recording means for recording the voice extracted by the extraction means together with the image in association with the coordinates of the object on the image; An imaging device comprising:

2. 2. The imaging device according to claim 1, wherein the object detection means calculates a direction of the object relative to the imaging device based on coordinates of the object on the image.

3. 3. The imaging device according to claim 2, wherein the object detection means calculates a direction of the object relative to the imaging device based on coordinates of the object on the image and a focal length of a photographing optical system of the imaging device.

4. 2. The imaging device according to claim 1, wherein the extraction means extracts the sound using an audio filter that limits a frequency band in accordance with the object detected by the object detection means.

5. 2. The imaging device according to claim 1, wherein said object detection means treats a plurality of detected objects as a single object when the detected objects are of the same type and have the same coordinates.

6. 2. The imaging device according to claim 1, wherein the recording means records a set of the sound extracted by the extraction means and the coordinates of the object on the image as one object.

7. 2. The imaging device according to claim 1, wherein the object detection means further calculates a distance from the imaging device of the detected object, and the imaging device further comprises a correction means for correcting a sound pressure of a sound corresponding to the object in accordance with the distance.

8. 8. The imaging apparatus according to claim 7, wherein the correction means corrects the sound pressure using table data for correcting frequency characteristics of sound.

9. 8. The imaging device according to claim 7, wherein the correction means does not correct the sound pressure of the sound corresponding to the detected object when the distance of the detected object from the imaging device is greater than a predetermined threshold value.

10. The imaging device described in claim 1, characterized in that when the detected object moves from within the angle of view range of the imaging device to outside the angle of view range and the detected object continues to emit the same type of sound as the sound it was emitting within the angle of view range, the recording means associates the central coordinates of an area of ​​a certain size that is predicted to be the destination of the detected object outside the angle of view range with the sound.

11. The imaging device described in claim 1, characterized in that when the detected object moves from within the angle of view range of the imaging device to outside the angle of view range and the detected object does not continue to emit the same type of sound that it was emitting within the angle of view range, the recording means associates the coordinates of the boundary between within the angle of view range and outside the angle of view range with the sound.

12. The imaging device described in claim 1, characterized in that the recording means associates the coordinates of the imaging device with the sound from all directions when the detected object moves from within the angle of view of the imaging device to outside the angle of view and the detected object does not continue to emit the same type of sound that it emitted within the angle of view.

13. The imaging device described in claim 1, characterized in that when the detected object enters the angle of view range of the imaging device from outside the angle of view range and the detected object continues to emit the same type of sound as the sound it was emitting outside the angle of view range, the recording means associates the central coordinates of a certain-sized area from which the object outside the angle of view range is predicted to have moved, with the sound.

14. 2. The imaging device according to claim 1, wherein the plurality of microphones are arranged at predetermined angular intervals from one another.

15. 15. The imaging device according to claim 14, wherein the plurality of microphones are arranged at angular intervals of 90 degrees from each other.

16. A method for controlling an imaging device including an imaging means and an audio acquisition means having a plurality of microphones arranged facing in different directions, comprising: an object detection step of detecting an object included in an image acquired by the imaging means, and detecting coordinates of the detected object on the image and a direction of the detected object relative to the imaging device; an extraction step of extracting, from the sounds acquired by the sound acquisition means, a sound corresponding to a direction of the object with respect to the imaging device, based on a direction of the object with respect to the imaging device; a recording step of associating the voice extracted in the extraction step with coordinates on the image of the object and recording the sound together with the image; 13. A method for controlling an imaging apparatus comprising:

17. A program for causing a computer to execute each step of the control method according to claim 16.

18. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 16.

Images