Image pickup apparatus and method

Abstract

An image pickup apparatus and method includes an image pickup circuit and at least one CPU or at least one circuit configured to implement functions of: a conversion unit configured to convert a sound signal acquired by a microphone including a plurality of microphone elements arranged at respective predetermined angles into stereophonic sound data; an extraction unit configured to extract a sound signal of a specific sound source from the sound signals acquired by the microphone and convert the sound signal of the specific sound source into omnidirectional data of stereophonic sound data; a combination processing unit configured to combine the stereo sound data and the omnidirectional data; and a recording unit configured to record the sound data output from the combination processing unit and the image data acquired by the imaging circuit.

Description

Technical Field

[0001] The present disclosure relates to an imaging device. Background Art

[0002] As stereoscopic image technology improves, 3D moving images are becoming popular as general content. For example, video sharing websites support 3D moving images. In this case, the demand for three-dimensional sound field reproduction is increasing to follow the stereoscopic image technology.

[0003] Ambient stereo is known as a technology for reproducing a sound field at a specific location in space. In ambient stereo, the sound field at the location can be reproduced by reproducing sounds from all directions at the location where a dedicated microphone array is arranged. A signal having directivity in an arbitrary direction at the location can be generated by processing after sound pickup.

[0004] For example, a microphone disclosed in Japanese Patent Laid-Open No. 2020-57987 has been provided as a microphone for acquiring ambient stereo sound data. By connecting a camera and an ambient stereo microphone to perform image capture, content having both a three-dimensional image and a three-dimensional sound field can be provided.

[0005] In recent years, content such as online travel or sports viewing, which aims at a realistic experience in a remote location, has begun to be provided. In recording of such content, the use of the above-mentioned technology can be expected.

[0006] However, since the sound in all directions is recorded three-dimensionally in the related art, the sound such as narration that the shooter intentionally makes becomes a directional sound. This makes the content uncomfortable. Suppose that some viewers want to turn off the narration. It is also assumed that it is necessary to eliminate the sound that accidentally enters the content when the image is shot from the content. Summary of the invention

[0007] The present disclosure provides an imaging apparatus configured to record stereo sound and configured to make specific sound data such as narration or unexpected noise sound omnidirectional or remove the specific sound data.

[0008] According to an embodiment of the present disclosure, a camera device includes a camera unit; a conversion unit, configured to convert a sound signal acquired by a microphone including a plurality of microphone elements arranged at respective predetermined angles into stereo sound data; an extraction unit, configured to extract a sound signal of a specific sound source from the sound signal acquired by the microphone, and convert the sound signal of the specific sound source into omnidirectional data of the stereo sound data; a combination processing unit, configured to combine the stereo sound data acquired by the conversion unit and the omnidirectional data acquired by the extraction unit and output the combined sound data; and a recording unit, configured to record the sound data output from the combination processing unit and the image data acquired by the camera unit.

[0009] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram of an image pickup apparatus according to the first embodiment.

[0011] Figure 2 is a schematic diagram of an image pickup apparatus describing the arrangement of microphones according to the first embodiment.

[0012] Figure 3A and 3B is a schematic diagram of a microphone for acquiring stereo sound according to the first embodiment.

[0013] Figure 4 is a flowchart according to the first embodiment.

[0014] Figure 5 is a block diagram of a sound processing unit according to the first embodiment.

[0015] Figure 6 is a diagram showing an exemplary positional relationship between a user and a camera according to the first embodiment.

[0016] Figure 7 is a diagram for describing sound source extraction according to the first embodiment.

[0017] Figure 8 is a schematic diagram showing the arrangement of microphones according to a modification of the first embodiment.

[0018] Fig. 9 is a block diagram of an image pickup apparatus according to a second embodiment.

[0019] Fig.10 is a schematic diagram describing the arrangement of microphones according to the second embodiment.

[0020] Fig.11 is a block diagram of a sound processing unit according to the second embodiment.

[0021] Fig.12 is a flowchart describing the processing according to the second embodiment.

[0022] Fig.13 is a block diagram of a sound processing unit according to a modification of the second embodiment.

[0023] Fig.14 : is a diagram describing a narration output state when a stereoscopic image is reproduced according to a modification of the second embodiment. DETAILED DESCRIPTION

[0024] Embodiments will be described in detail below with reference to the accompanying drawings.

[0025] First embodiment

[0026] Figure 1 3 is a block diagram showing the configuration of the image pickup apparatus 100 according to the first embodiment. The optical lens 300 is a removable optical lens in which an optical lens and a motor for driving the lens are installed. By controlling the optical lens, various operations such as focusing, zooming, and camera shake correction can be performed on a moving subject during image capture. Various types of optical lenses 300 can be attached to the image pickup apparatus 100. Note that the user can freely replace the optical lens 300 attached to the image pickup apparatus 100 according to the application.

[0027] Since the image acquired by the camera 100 is a stereoscopic image, the optical lens 300 has a binocular configuration with two optical axes. For the stereoscopic image, the binocular parallax as the difference in the image position on the retina of the right eye and the left eye is used. The visual system can recognize the depth of the image by this difference in the image position, thereby realizing stereoscopic vision. The viewing angle range of the optical lens 300 can cover all directions, but is set to 180 degrees in the embodiment.

[0028] The imaging unit 101 converts the optical image of the subject captured by the optical lens 300 into an image signal using an imaging device, and performs analog-to-digital conversion and image adjustment processing on the image signal to generate image data. The imaging device in the imaging unit 101 may be a photoelectric conversion element represented by, for example, a CCD or CMOS sensor. The lens control unit 102 controls the optical lens 300 as needed based on information acquired from the imaging unit 101 and information from the control unit 111 to be described below.

[0029] The sound processing unit 104 is connected to the microphone 113 via the sound input terminal of the camera device 100. The microphone 113 is an external microphone of the camera device 100, and is connected to the camera device 100 through a connection terminal including a sound input terminal. The microphone 113 is an ambient stereo microphone (hereinafter referred to as a microphone) including at least four unidirectional microphone elements. The microphone 113 collects sounds from all directions of the camera device 100 using these microphone elements. The sound signal acquired by the microphone 113 is sent to the sound processing unit 104. The sound processing unit 104 performs, for example, amplification, analog-to-digital conversion, conversion into the B format of the ambient stereo format, and sound signal processing on the sound signal from the microphone 113 to generate a plurality of sound data. In the present embodiment, the microphone 113 is externally connected to the sound input terminal of the camera device 100, but the microphone 113 may be built into the camera device 100. The sound processing unit 104 will be described in detail below.

[0030] The memory 105 temporarily stores image data acquired by the imaging unit 101 and sound data acquired by the sound processing unit 104. The display control unit 106 displays, for example, an image corresponding to the image data acquired by the imaging unit 101, and an operation screen and a menu screen of the imaging apparatus 100 on the display unit 107 or on an external display via an image terminal of an external output unit 115 to be described below.

[0031] The encoding processing unit 108 reads out the image data and the sound data temporarily stored in the memory 105, and performs predetermined encoding on the image data and the sound data to generate compressed image data and compressed sound data. The sound data does not necessarily have to be compressed. The compressed image data can be obtained by any compression method such as MPEG2 or H.264 / MPEG4-AVC. The compressed sound data can also be obtained by any compression method such as AC3AAC, ATRAC or ADPCM.

[0032] The recording / reproducing unit 109 records the compressed image data generated by the encoding processing unit 108, the compressed sound data generated by the encoding processing unit 108, or the sound data and various data in the recording medium 110. The recording medium 110 is any type of recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, as long as the recording medium can record, for example, image data and sound data.

[0033] The control unit 111 can control each block in the image pickup device 100 by sending a control signal to each block. The control unit 111 includes, for example, at least one CPU or at least one circuit and a memory to perform various controls on these blocks. The memory used in the control unit 111 includes, for example, a read-only memory (ROM) for storing various control programs and a random access memory (RAM) for calculation processing, and also includes an external memory of the control unit 111.

[0034] The operation unit 112 includes, for example, various buttons, various dials, and a touch panel. In response to the user's operation of the operation unit 112, an instruction signal corresponding to the operation is sent to the control unit 111. The operation unit 112 includes, for example, an image capture button for instructing the start or end of moving image recording and a manual volume for adjusting the volume when sound is reproduced. The data bus 118 sends various data such as sound data and image data and various control signals to each block in the imaging device 100.

[0035] The normal operation of the image pickup apparatus 100 according to the present embodiment will be described. The image pickup apparatus 100 according to the present embodiment supplies power to each block therein from a power supply unit (not shown) in response to a user's instruction to turn on the power via the operation unit 112. Upon receiving power, the control unit 111 checks which mode (e.g., image capture mode or reproduction mode) is indicated by the mode switching switch in the operation unit 112 based on an instruction signal from the operation unit 112.

[0036] In the moving image recording mode, the image data acquired by the imaging unit 101 and the sound data acquired by the sound processing unit 104 are saved as a single file. In the moving image recording mode, first, the control unit 111 sends a control signal for transitioning to the image capturing standby state to each block in the imaging apparatus 100 so that the blocks perform the following operations.

[0037] The imaging unit 101 converts the optical image of the subject captured by the optical lens 300 into an image signal using an imaging device, and performs analog-to-digital conversion and image adjustment processing on the image signal to generate image data. The imaging unit 101 sends the acquired image data to the display control unit 106 to display the image data on the display unit 107. The user prepares for image capture while viewing the screen displayed as above. Subsequently, when an instruction signal for starting image capture is sent to the control unit 111 in response to the user's operation of the record button of the operation unit 112, the control unit 111 sends the instruction signal for starting image capture to each block in the imaging device 100 so that the blocks perform the following operations.

[0038] The imaging unit 101 converts the optical image of the subject captured by the optical lens 300 into an image signal using an imaging device, and performs analog-to-digital conversion and image adjustment processing on the image signal to generate image data. The imaging unit 101 sends the acquired image data to the display control unit 106 to display the image data on the display unit 107. The imaging unit 101 also sends the acquired image data to the memory 105.

[0039] The sound processing unit 104 converts the sound signal acquired by the microphone 113 into a digital signal to generate sound data. When a plurality of microphones 113 are provided, the sound processing unit 104 processes the acquired plurality of digital sound signals to generate multi-channel sound data. The sound processing unit 104 sends the acquired sound data to the memory 105.

[0040] The encoding processing unit 108 reads out the image data and the sound data temporarily stored in the memory 105, and performs predetermined encoding on the image data and the sound data to generate compressed image data and compressed sound data. The control unit 111 combines the compressed image data and the compressed sound data to form a data stream, and outputs the data stream to the recording / reproducing unit 109. In the case where the sound data is not compressed, the control unit 111 combines the sound data and the compressed image data stored in the memory 105 to form a data stream, and outputs the data stream to the recording / reproducing unit 109. The recording / reproducing unit 109 writes the data stream as a single moving image file to the recording medium 110 under the management of a file system such as a universal disk format (UDF) or a file allocation table (FAT). The above operation continues during image shooting.

[0041] When an instruction signal for ending image capture is sent to the control unit 111 in response to the user's operation of the record button of the operation unit 112, the control unit 111 sends an instruction signal for ending image capture to each block in the image capture apparatus 100 so that the blocks perform the following operations. The image capture unit 101 stops generating image data, and the sound processing unit 104 stops generating sound data.

[0042] The encoding processing unit 108 reads out the remaining image data and the remaining sound data stored in the memory 105, and performs predetermined encoding on the image data and the sound data to generate compressed image data and compressed sound data. After that, the encoding processing unit 108 stops the operation. In the case where the sound data is not compressed, the operation stops after the generation of the compressed image data is completed. The control unit 111 combines the final compressed image data with the final compressed sound data or the final sound data to form a data stream, and outputs the data stream to the recording / reproducing unit 109. The recording / reproducing unit 109 writes the data stream as a single moving image file to the recording medium 110 under the management of a file system such as UDF or FAT. When the supply of the data stream stops, the moving image file is completed and the recording operation stops.

[0043] When the recording operation stops, the control unit 111 sends a control signal for transitioning to an image capture standby state to each block in the camera device 100, and returns to the image capture standby state. The image file recorded in the camera device 100 is reproduced as a stereoscopic image by a reproduction device (not shown) such as a head-mounted display. The image to be reproduced includes recorded sound superimposed thereon. For example, sound is output according to the rotation direction of the head of a user wearing, for example, a head-mounted display as a reproduction device, thereby providing a realistic image. The above has described an outline of the block configuration of the camera device 100 and the operation when recording a moving image.

[0044] The following will refer to Figures 2 to 8 A process for making a voice-over sound omnidirectional according to an embodiment is described. Figure 2 is a schematic diagram of the image pickup apparatus 100 to which the microphone 113 is attached. By connecting the microphone 113 capable of recording stereo sound to the sound input terminal of the image pickup apparatus 100, stereo sound data can be recorded while capturing a moving image.

[0045] Figure 3A and 3B is a schematic diagram of a microphone 113 capable of acquiring stereo sound using an ambient stereo method. Figure 3A As shown, microphone 113 includes four microphone elements A to D.

[0046] The four microphone elements A to D are arranged at respective predetermined angles. In one embodiment, the microphone elements A to D are provided facing Figure 3A The four vertices of the cube are shown. When the space in the figure is composed of Figure 3BWhen the direction is defined as shown, microphone element A faces the front upper left vertex, microphone element B faces the front lower right vertex, microphone element C faces the rear lower left vertex, and microphone element D faces the rear upper right vertex. Since each of the four microphone elements A to D is a unidirectional microphone, the microphone acquires sound components in the direction that the microphone faces. The four sound data acquired by the respective microphone elements are referred to as the A format of the ambient stereo format.

[0047] Figure 4 is a flowchart according to the first embodiment. In step S101, when an instruction to start image capture is issued by, for example, the operation unit 112, recording of moving images and sounds is started. Although only recording of sounds will be described below, recording of moving images may also be performed. Figure 4 The processing of the illustrated flowchart is performed by the control unit 111 that controls each unit.

[0048] When image capture is started in step S101, the microphone 113 starts acquiring sound in step S102. At this time, it is desirable to acquire sound at any regular intervals and perform sound processing calculations to be described below. Sound acquisition is continued until an instruction to end image capture is issued via, for example, the operation unit 112.

[0049] Figure 5 1 is a block diagram including a detailed configuration of the sound processing unit 104 for acquiring stereo sound data. In the sound processing unit 104, the amplifier 21 amplifies the sound signal acquired by each microphone element in the microphone 113 with an analog gain. The amplified sound signal is converted into a digital signal by analog-to-digital conversion processing in the A / D converter 22.

[0050] In step S103, the ambisonic A-format data acquired in step S102 is recorded in the recording medium 110. When the user does not set the recording of the A-format data, this step is unnecessary.

[0051] In step S104, the ambient stereo A format data acquired in step S102 is converted into B format data. In the stereo sound technology using ambient stereo, the A format cannot be used as it is, and needs to be converted into the B format. The B format is sound data including omnidirectional data and bidirectional data in the stereo sound technology using ambient stereo. The B format is a data format composed of an omnidirectional signal W in all directions, a front and rear direction signal X, a left and right direction signal Y, and an up and down direction signal Z. The B format encoder 23 performs conversion from the A format to the B format using expressions (1) to (4).

[0052] W=FLU+FRD+BLD+BRU···(1)

[0053] X = FLU + FRD - BLD - BRU ··· (2)

[0054] Y = FLU - FRD + BLD + BRU ··· (3)

[0055] Z = FLU + FRD + BLD + BRU ··· (4)

[0056] In these expressions,

[0057] W: omnidirectional signal (B format),

[0058] X: bidirectional signal in the front and rear directions (B format),

[0059] Y: Bidirectional signal in left and right directions (B format),

[0060] Z: Bidirectional signal in up and down directions (B format),

[0061] FLU: Front upper left signal acquired by microphone element A (A format),

[0062] FRD: Front lower right signal picked up by microphone element B (A format),

[0063] BLD: rear lower left signal picked up by microphone element C (A format), and

[0064] BRU: Rear upper right signal picked up by microphone element D (A format).

[0065] Each B-format signal is assigned a weight according to the direction the viewer faces at the time of reproduction, and is combined. As a result, sound based on the orientation of the viewer can be reproduced, and stereo sound can be realized.

[0066] In step S105, the B-format data converted in step S104 is recorded in the recording medium 110. When the user does not set recording of unedited B-format data, this step is unnecessary.

[0067] In step S106, it is checked whether a mode for making the narration sound omnidirectional (e.g., called narration mode) is set. The user can set the on or off state of the narration mode by operating the operation unit 112. When the narration mode is set, the process proceeds to step S107 to make the narration omnidirectional.

[0068] When the narration mode is not set, the process for making the narration omnidirectional is not performed (stops being performed), and the process proceeds to step S111.

[0069] In step S107, the position of the narration sound source is set to be omnidirectional. For example, the user sets the position of the narration sound source by operating the operation unit 112 before starting image capture. Figure 6 As shown, when the photographer wants to capture an image while recording a narration, it is assumed that the narration sound comes from the back. Therefore, the position of the narration sound source is set to the back of the camera 100. The camera 100 may include a facial recognition unit, the face of the person who makes the narration voice may be registered in advance, and the position where the facial recognition has been performed may be set as the position of the narration sound source.

[0070] In step S108, the narration sound is extracted using the position of the narration sound source set in step S107 and the ambient stereo B format data acquired in step S104. In post-production using the ambient stereo B format, the entire sound field can be easily rotated. For example, when it is necessary to reproduce a sound source such as Figure 7 In the case of the sound field that a viewer hears when the orientation of the viewer's face is changed by Φ on the horizontal plane shown, the reproduction can be achieved using Expression (5).

[0071]

[0072] Using the B-format data calculated from expression (5), the narration sound N is calculated from expression (6).

[0073] N=W'+X'···(6)

[0074] The method of acquiring the voice-over sound is not limited to the above expression. As will be described below, the voice-over sound may be acquired by other microphones than the microphone 113. The surrounding sound may be removed by, for example, filtering the extracted sound signal specific to human speech.

[0075] In step S109, the narration sound acquired in step S108 is combined with the ambisonic B-format omnidirectional signal W acquired in step S104. For example, calculation is performed using expression (7).

[0076] Wn = W + N ··· (7)

[0077] Where Wn is the omnidirectional signal including the narration sound.

[0078] The method of acquiring the omnidirectional signal including the narration sound is not limited to the above expression.

[0079] An example of a method of acquiring a narration sound will be described. Figure 8As shown, one of the four microphone elements included in the microphone 113 is arranged to face the direction in which the photographer is facing. Therefore, the sound of the microphone facing the direction in which the photographer is facing can be set as the narration sound. This eliminates the calculation in step S108 and can reduce the load on the sound processing unit 104.

[0080] In step S110, the data in the ambient stereo B format acquired by the processing to step S109 is recorded in the recording medium 110, in which the narration sound has been made omnidirectional. When an image capture stop instruction is issued by, for example, the operation unit 112 in step S111, the processing proceeds to step S112, and the recording of the moving image and sound is stopped. When the image capture stop instruction is not issued, the processing returns to step S102, and the sound acquisition continues.

[0081] As described above, when recording a stereo image and stereo sound while recording a narration sound as a specific sound source, by making the characteristics of the microphone at the narration sound source position omnidirectional, the narration sound is recorded as a sound having no directivity and can be provided without causing a sense of discomfort.

[0082] Second embodiment

[0083] In the first embodiment, by making the narration sound omnidirectional, the narration sound collected by the unidirectional microphone 113 can be clearly heard regardless of which direction the viewer is facing.

[0084] When viewing a reproduced image of a stereoscopic image including a voice-over sound, the voice-over sound may be annoying to some viewers. At this point, the viewer can turn off the sound output of the reproduction device. However, when the sound output is turned off, the surrounding sounds of the stereoscopic image cannot be heard. Thus, this loses the sense of reality. In a second embodiment, a camera device is provided that is configured to achieve the reproduction of a realistic image by removing only the annoying voice-over sound from the sound output of the reproduction device and retaining the surrounding sounds of the stereoscopic image.

[0085] In the present embodiment, in addition to the microphone 113 as an ambient stereo microphone, a narration microphone for collecting narration sound is also provided. The sound data collected by the narration microphone includes surrounding sounds in addition to the narration sound. Only the narration sound is extracted from the sound data acquired by the narration microphone. By subtracting the extracted sound data from each sound data collected by the plurality of microphone elements in the microphone 113, the sound data obtained by removing the narration sound from the sound data collected by the microphone 113 can be acquired.

[0086] Fig. 9 is a block diagram of an image pickup apparatus 100 according to the second embodiment. Figure 1and its description will be omitted. Figure 1 The first embodiment shown in the block diagram of FIG. 1 is different in that a narration microphone 114 for collecting narration sound is added. The narration microphone 114 is one or more unidirectional microphones incorporated in the camera 100, and the collected sound data is output to the sound processing unit 104.

[0087] In the following description of the second embodiment, for simplicity, the narration microphone 114 is a single unidirectional microphone. Although the narration microphone 114 is described as a built-in microphone in the camera apparatus 100 in the second embodiment, the narration microphone 114 may be externally connected to the camera apparatus 100 like the microphone 113, provided that the narration microphone 114 can accurately collect the narration sound. When the narration microphone 114 is externally connected to the camera apparatus 100, it may be connected to the camera apparatus 100 in a wired or wireless manner.

[0088] Next, an arrangement example of the narration microphone 114 will be described. Fig.10 Shown is a diagram of an image pickup device 100 to which a microphone 113 is attached. The angle of image pickup by the image pickup device 100 is 180 degrees. During image pickup, the photographer generally performs narration while checking the screen. Therefore, the narration sound source is the back side of the image pickup device 100 outside the angle of image pickup by the image pickup device 100. Therefore, it is preferable to arrange the narration microphone 114 on the back side of the image pickup device 100.

[0089] Next, we will refer to Fig.11 The sound processing unit 104 in the camera apparatus 100 including the narration microphone 114 is described. Fig.11 In the sound processing unit 104 shown in FIG. Figure 2 The same parts in FIG. 1 and description thereof will be omitted. Since the narration microphone 114 has been described above, description thereof will also be omitted.

[0090] The sound data collected by the narration microphone 114 is amplified by the amplifier 21 and converted into a digital signal by the A / D converter 22. The sound data that has been subjected to the analog-to-digital conversion includes not only the narration sound but also the surrounding sound. The purpose of collecting sound with the narration microphone 114 is to remove the narration sound component from each sound data collected by the microphone 113. Therefore, the sound extraction unit 24 extracts only the human voice component, that is, the narration sound, from the sound data collected by the narration microphone 114 using a known sound source enhancement technology.

[0091] The sound collected by the narration microphone 114 includes human voice components and other components superimposed on each other. In order to enhance only the human voice component in the collected sound data, for example, a method of estimating a real time-frequency mask in a discrete Fourier transform region of the collected sound data using a deep neural network is known.

[0092] The sound data extracted by the sound extraction unit 24 is recorded in the recording / reproducing unit 109 as the narration sound, and is also output to the removal unit 26. The removal unit 26 performs processing for reducing the narration sound included in each sound data by subtracting the narration sound component extracted by the sound extraction unit 24 from each sound data output from the A / D converter 22.

[0093] In the subtraction processing in the removal unit 26, the narration sound component extracted by the sound extraction unit 24 is subtracted in the time domain from each sound data output from the A / D converter 22. As another method, for example, the sound data of each of the microphone 113 and the narration microphone 114 may be converted into a frequency spectrum, subtraction may be performed in the frequency domain, and then the original time domain sound data may be restored. Any method capable of removing the narration sound component from the sound data of the microphone 113 may be adopted.

[0094] In the removal unit 26, a subtractor is connected to each A / D converted signal output of the microphone 113. The plurality of unidirectional microphones included in the microphone 113 have different sound collecting directions. The amount of subtraction in the subtraction process depends on the sound collecting direction of the microphone.

[0095] For example, in the case where the positional relationship between the microphone 113 and the narration microphone 114 is determined, weights based on the positional relationship of the respective microphones set in advance for the microphones are assigned to the sound data output from the sound extraction unit 24. The weighted narration sound is subtracted from the corresponding sound signal. Sound collection may be performed before sound recording to measure the relationship between the sound data of the microphone 113 and the narration microphone 114, and the result of the measurement may be reflected in the subtraction process.

[0096] The sound data from the microphone 113 is converted from the A format, which is a format of sound data from which the narration sound component has been removed in the removal unit 26, to the B format by the B format encoder 23. The respective sound data converted to the B format are recorded as sound information in the recording / reproducing unit 109 together with the sound data corresponding only to the narration sound from the sound extraction unit 24. The sound data corresponding only to the narration sound and the sound data from which the narration sound has been removed and converted to the B format are recorded in a state of being synchronized in time with the recorded stereoscopic image.

[0097] Next, we will refer to Fig.12 The flowchart shown describes the operation according to the second embodiment. In step S201, when an instruction to start image capture is issued in response to the photographer's operation of the operation unit 112, recording of moving images and sounds is started. Although only the recording of sounds will be described below, in fact, moving images are recorded simultaneously with sounds. Fig.12 The illustrated processing is performed by the control unit 111 that controls the respective units in the imaging apparatus 100 .

[0098] When the recording operation starts, the microphone 113 and the narration microphone 114 collect sounds for a certain period (steps S202 and S302). The collected sounds as the output of the microphone 113 are sound data in the A format. In steps S203 and S303, the sound data output from the microphone 113 and the narration microphone 114 are temporarily recorded. They may be recorded in the memory 105, or the A format sound data as the output of the microphone 113 may be recorded and stored in the recording medium 110.

[0099] In step S304, the sound extraction unit 24 extracts the frequency components of human voice from the sound data output by the narration microphone 114, and outputs the extracted frequency components as narration sound. The removal unit 26 subtracts the narration sound extracted by the sound extraction unit 24 from the respective sound data from the microphone 113. As a result, the narration sound component is removed from each sound data from the microphone 113 (step S204). A plurality of sound data from the microphone 113 are output to the B format encoder 23, wherein the narration sound component has been removed from the plurality of sound data. The B format encoder 23 converts each sound data from the A format into the B format, wherein the B format is the basic data format in the ambient stereo (step S205). The sound in the B format conforms to the basic data format.

[0100] In step S206, the recording / reproducing unit 109 records the plurality of sound data converted into the B format from the microphone 113 in a predetermined sound file format in the recording medium 110. In step S306, the narration sound data from the narration microphone 114 is similarly recorded in the recording medium 110. The respective recorded sound data are synchronized with each other on the time axis. The respective sound data are separate sound files, are embedded in the moving image file and are recorded and stored.

[0101] When the viewer performs an operation to turn off the narration sound, in the case where the reproduction device (not shown) reproduces the respective recorded sound data together with the stereoscopic image, only the sound corresponding to the stereoscopic image from the microphone 113 is reproduced. On the other hand, when the narration sound is turned on, the narration sound is superimposed on the above-mentioned sound corresponding to the stereoscopic image and reproduced. The narration sound is recorded and stored in the recording medium 110 so that the narration sound is omnidirectional when reproduced.

[0102] Return to reference Fig.12 , it is determined in step S207 whether the photographer has issued an instruction to stop sound recording. When a stop instruction has been issued (yes), the sound recording operation is stopped in step S208. When a stop instruction has not been issued (no), the process returns to steps S202 and S303 to continue the sound recording operation.

[0103] As described above, in the second embodiment, the sound recorded with the stereoscopic image is separated into the sound for the stereoscopic image and the sound for the narration, and these sounds are recorded in respective independent sound files to allow only the narration sound to be turned off when reproducing. As a result, even when the viewer finds the narration sound annoying when reproducing the stereoscopic image, only the narration sound can be turned off. Since the other sound is reproduced, the sense of reality of the stereoscopic image is not lost, and the viewer can enjoy the image viewing.

[0104] Modification of the Second Embodiment

[0105] In the second embodiment, if the viewer wants to know the explanation and various information of the stereoscopic image with narration when reproducing the stereoscopic image, the viewer reproduces the stereoscopic image without muting the narration sound. However, some viewers want to know the above explanation and the above various information, but find the voice quality of the speaker or the way the speaker speaks uncomfortable. These viewers need to endure it without muting the narration sound, and therefore cannot enjoy the image viewing.

[0106] In the modification of the second embodiment, the sound data of the narration sound component extracted by the sound extraction unit 24 is converted into text information. The converted text information is recorded in the recording medium 110 as a text file.

[0107] Fig.13 1 is a block diagram of the sound processing unit 104 according to the present modification. This block diagram is similar to the block diagram of the sound processing unit 104 according to the second embodiment ( Fig.11 ) are almost the same. The same reference numerals are added to Fig.11 The same blocks in the figure are described below and their description will be omitted. Fig.13 and Fig.11 The difference between them is that a text conversion unit 27 is added.

[0108] The text conversion unit 27 converts the narration sound component output from the sound extraction unit 24 into text data. The text conversion unit 27 converts the narration sound into text data using a known speech recognition technology. For example, the text conversion unit 27 extracts sound features from the narration sound output from the sound extraction unit 24. A language model and a pronunciation dictionary for extracting phonemes from data learned by AI are pre-recorded in the camera 100. The extracted sound features are compared with them to pick up a language combination. The picked up language combination is output as text data.

[0109] The text data output from the text conversion unit 27 is processed in a predetermined file format in the recording / reproducing unit 109 and then recorded in the recording medium 110. For example, when reproducing a stereoscopic image, an icon is displayed on the Fig.14 The current narration state is notified to the viewer on the reproduction screen shown in (A) to (C) (the lower left part of the screens 141, 142 and 143). The narration state can be switched cyclically by the viewer's operation.

[0110] Fig.14 The balloon portion shown in (A) indicates a state in which a reproducing device (not shown) outputs a narration sound from a sound output, Fig.14 The balloon parts shown in (B) and (C) indicate a state where the narration sound is not output. Fig.14 (A) shows a state in which the narration sound is enabled and the surrounding sound and the narration sound are output from the sound output of the reproducing device when the image is recorded. Fig.14 (B) shows a state in which the narration sound is disabled and only the surrounding sound during image recording is output from the sound output of the reproducing device.

[0111] Fig.14 (C) shows the following state, in which the narration sound is disabled, but only the surrounding sound during image recording is output from the sound output of the reproduction device, and the text into which the narration sound is converted is superimposed and displayed on the reproduction screen of the reproduction device.

[0112] The text conversion unit 27 may add the pitch variation of the input narration sound to the text data.

[0113] When reproducing a stereoscopic image, the above added information can be used to Fig.14 In (C), a text display expression method is adopted, for example, increasing the font size, making the font bold, or changing the font itself based on the pitch change state of the narration sound. The pitch change state of the narration sound can be determined based on, for example, the intensity and loudness of the sound detected at the time of sound feature extraction performed by the text conversion unit 27. The amount of change over time in the pitch change state of the narration sound.

[0114] The narration microphone 114 according to the present embodiment is a single unidirectional microphone as described above, but may include a plurality of microphones arranged in different directions. The direction of the narration sound may be estimated using these microphones, and the sound source direction may be added to the text signal. When reproducing a stereoscopic image, the text display frame may be made into a balloon shape to face the direction of the narration sound using the above added information. Fig.14 The direction of the narration source in (C).

[0115] As described above, in the modification of the second embodiment, the text data of the narration sound is also stored during the sound recording. Therefore, even when the viewer turns off the output of the narration sound when reproducing the stereoscopic image, the viewer can know the narration details without hearing the narration sound through his operation.

[0116] The present disclosure can provide an imaging apparatus configured to record stereo sound and capable of making specific sound data such as narration or unexpected noise sound omnidirectional or removing the specific sound data.

[0117] Other embodiments

[0118] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0119] Although the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A camera device, comprising: Camera unit; a conversion unit configured to convert a sound signal acquired by a microphone connected to the image pickup device and including a plurality of microphone elements arranged at respective predetermined angles into stereo sound data; an extraction unit configured to extract a sound signal of a specific sound source from the sound signal acquired by the microphone, and convert the sound signal of the specific sound source into omnidirectional data of the stereo sound data; a combining processing unit configured to combine the stereo sound data acquired by the converting unit and the omnidirectional data acquired by the extracting unit, and output the combined sound data; as well as A recording unit configured to record the sound data output from the combining processing unit and the image data acquired by the imaging unit into a memory.

2. The imaging device according to claim 1, further comprising: A setting unit configured to set whether to record the sound signal of the specific sound source; and a control unit configured to control so that a process of combining the omnidirectional data with the stereo sound data acquired by the conversion unit is stopped when a setting is made not to record the sound signal of the specific sound source, in, In the case where the setting is made not to record the sound signal of the specific sound source, the stereo sound data not combined with the omnidirectional data is recorded by the recording unit.

3. The imaging device according to claim 1 , further comprising a setting unit configured to set a position of the specific sound source, in, The extraction unit performs a process for extracting the sound signal of the specific sound source from the sound signal acquired from the microphone according to the set position.

4. The imaging device according to claim 1, wherein: The extraction unit extracts a sound signal of a photographer as the specific sound source.

5. The imaging device according to claim 4, wherein: One of the plurality of microphone elements is arranged to face the direction in which the photographer is facing.

6. The imaging device according to claim 1, wherein: The conversion unit converts the sound signal acquired by the microphone into ambient stereo B format sound data, Wherein, the extraction unit outputs ambient stereo B format omnidirectional data, and The combination processing unit performs a combination process for combining the ambient stereo B-format sound data and the ambient stereo B-format omnidirectional data.

7. The imaging device according to claim 1, wherein: The microphone is an external microphone of the camera device.

8. The imaging device according to claim 1, wherein: The recording unit records the image data acquired by the imaging unit as a stereoscopic image.

9. A method comprising: converting a sound signal acquired by a microphone connected to the imaging device and including a plurality of microphone elements arranged at respective predetermined angles into stereo sound data; extracting a sound signal of a specific sound source from the sound signal acquired by the microphone, and converting the sound signal of the specific sound source into omnidirectional data of the stereo sound data; combining the stereo sound data acquired by the converting and the omnidirectional data acquired by the extracting, and outputting combined sound data; as well as The combined sound data output from the combination and the image data acquired by the imaging circuit of the imaging device are recorded in a memory.

Images