Imaging device

Abstract

To provide an imaging device for recording stereophonic sound which makes it possible to make specific audio data such as narration or unintended noise non-directional or to remove the data.SOLUTION: An imaging device includes imaging means, a microphone 113 having a plurality of microphone elements arranged at a predetermined angle and acquiring a sound signal of a stereophonic sound, and an audio processing unit 104 which is audio processing means for processing the audio signal acquired by the microphone 113. The audio processing unit 104 converts the audio signal acquired by the microphone 113 into stereophonic data, extracts an audio signal of a specific sound source from the audio signal acquired by the microphone 113, converts the audio signal of the specific sound source into non-directional data of the stereophonic data, and combines it with the stereophonic data.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] As 3D imaging technology improves, 3D video has become commonplace as a form of content, and video sharing sites are now supporting 3D video. In this environment, there is a growing demand for the reproduction of a three-dimensional sound field in audio, in keeping with the advancement of 3D imaging technology.

[0003] Ambisonics is known as a technology that reproduces a sound field at a certain position in space. With Ambisonics, it is possible to reproduce the sound field at that point by reproducing sounds from all directions at the position where a dedicated microphone array is installed, and it is also possible to generate a signal with directivity in any direction at that position by processing the sound after collection.

[0004] As a microphone for acquiring stereophonic data using Ambisonics, a microphone as disclosed in Patent Document 1 has already been provided. By using a camera and an Ambisonics microphone in cooperation to capture images, it becomes possible to provide content with a stereoscopic image and sound field.

[0005] In recent years, content aimed at providing remote, real-life experiences, such as online travel and watching sports, has begun to be offered, so it is expected that this technology will be useful for recording such content. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2020-57987 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the conventional technology, since the sound from all directions is recorded three-dimensionally, the sound intentionally emitted by the cameraman, such as the narration, becomes directional, resulting in an unnatural content. It is also assumed that some content viewers will want to turn off the narration. Furthermore, it is also assumed that some viewers will want to eliminate sounds that were unintentionally recorded during the content shooting from the content.

[0008] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to provide an imaging device for recording stereophonic sound that makes it possible to make specific audio data, such as narration or unintended noise, non-directional or to remove such data. [Means for solving the problem]

[0009] The present invention comprises an imaging means, a microphone having a plurality of microphone elements arranged at a predetermined angle and acquiring stereophonic audio signals, and an audio processing means for processing the audio signals acquired by the microphone, wherein the audio processing means converts the audio signals acquired by the microphone into stereophonic data, extracts an audio signal of a specific sound source from the audio signal acquired by the microphone, converts the audio signal of the specific sound source into omnidirectional data of the stereophonic data, and synthesizes it with the stereophonic data. Effect of the Invention

[0010] According to the present invention, it is possible to provide an imaging device that records stereophonic sound, which makes it possible to make specific audio data, such as narration or unintended noise, non-directional or to remove the specific audio data. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram of an imaging apparatus according to a first embodiment. [Diagram 2] FIG. 2 is a schematic diagram of an imaging device illustrating the arrangement of microphones according to the first embodiment. [Diagram 3] FIG. 2 is a schematic diagram of a microphone for acquiring stereophonic sound according to the first embodiment. [Figure 4]3 is a flowchart according to the first embodiment. [Diagram 5] FIG. 2 is a block diagram of a voice processing unit according to the first embodiment. [Figure 6] 4 shows an example of the positional relationship between a user and a camera according to the first embodiment. [Figure 7] FIG. 2 is a diagram used to explain sound source extraction according to the first embodiment. [Figure 8] FIG. 11 is a schematic diagram showing a microphone arrangement according to a modified example of the first embodiment. [Figure 9] FIG. 11 is a block diagram of an imaging apparatus according to a second embodiment. [Figure 10] FIG. 11 is a schematic diagram illustrating the arrangement of microphones according to the second embodiment. [Figure 11] FIG. 11 is a block diagram of a voice processing unit according to a second embodiment. [Figure 12] 10 is a flowchart illustrating an operation according to a second embodiment. [Figure 13] FIG. 11 is a block diagram of a voice processing unit according to a modified example of the second embodiment. [Figure 14] 13A to 13C are diagrams illustrating a narration output state during stereoscopic video playback according to a modified example of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

[0013] (First embodiment) Fig. 1 is a block diagram showing the configuration of an imaging device 100 of the first embodiment. The optical lens 300 is a removable optical lens. An optical lens and a motor for driving the lens are mounted, and by controlling the optical lens, it is possible to perform various operations such as focusing on a moving subject, zooming, and image stabilization during imaging. There are multiple types of optical lens 300, which the user can freely replace according to the purpose.

[0014] Since the image captured by the imaging device 100 is a stereoscopic image, the optical lens 300 has a two-eye configuration with two optical axes. Stereoscopic images utilize binocular parallax, which is the positional shift of the images on the retinas of the left and right eyes. This positional shift allows the visual system to recognize the depth of the image, achieving stereoscopic vision. The range of the imaging angle formed by the optical lens 300 can cover all directions at maximum, but in the embodiment, it is described as 180 degrees.

[0015] The imaging unit 101 converts an optical image of a subject captured by the optical lens 300 into an image signal using an imaging element, and performs analog-to-digital conversion, image adjustment processing, etc. to generate image data. The imaging element may be a photoelectric conversion element such as a CCD or CMOS sensor. The lens control unit 102 controls the optical lens 300 as necessary based on information obtained from the imaging unit 101 and information from a control unit 111 (described later).

[0016] The audio processing unit 104 is connected to the microphone 113 via an audio input terminal. The microphone 113 is an Ambisonics microphone 113 (hereinafter, microphone) having at least four unidirectional microphone elements, and collects audio in all directions of the imaging device 100. An audio signal obtained by the microphone 113 is sent to the audio processing unit 104. The audio processing unit 104 performs amplification, analog-to-digital conversion, conversion from Ambisonics format to B format, audio signal processing, and the like on the audio signal from the microphone 113 to generate multiple pieces of audio data. In the embodiment, the microphone 113 is configured to be connected to an audio input terminal from outside the imaging device 100, but may be built into the imaging device 100. Details of the audio processing unit 104 will be described later.

[0017] The memory 105 temporarily stores image data obtained by the imaging unit 101 and audio data obtained by the audio processing unit 104. The display control unit 106 displays images related to the image data obtained by the imaging unit 101, an operation screen, a menu screen, and the like of the imaging device 100 on the display unit 107 or an external display via a video terminal of an external output unit 115 described later.

[0018] The encoding processor 108 reads out image data and audio data temporarily stored in the memory 105, performs predetermined encoding, and generates compressed image data, compressed audio data, etc. Also, audio data may not be compressed. The compressed image data may be compressed using any compression method, such as MPEG2 or H.264 / MPEG4-AVC. The compressed audio data may also be compressed using a compression method such as AC3AAC, ATRAC, ADPCM, etc.

[0019] The recording and reproducing unit 109 records the compressed image data, compressed audio data or audio data, and various other data generated by the encoding processing unit 108 onto the recording medium 110. Here, the recording medium 110 includes any type of recording medium, such as a magnetic disk, an optical disk, or a semiconductor memory, as long as it is capable of recording image data, audio data, and the like.

[0020] The control unit 111 can control each block of the imaging device 100 by transmitting a control signal to each block of the imaging device 100, and is composed of a CPU, memory, etc. for executing various controls. The memory used by the control unit 111 is a ROM for storing various control programs, a RAM for arithmetic processing, etc., and also includes a memory external to the control unit 111.

[0021] The operation unit 112 is made up of buttons, dials, etc., and transmits instruction signals to the control unit 111 in response to user operations. It also includes a shooting button for instructing the start and end of video recording, a manual volume for adjusting the volume during audio playback, etc. The data bus 118 transmits various data such as audio data and image data, and various control signals to each block of the imaging device 100.

[0022] Here, normal operation of the imaging device 100 of this embodiment will be described. In the imaging device 100 of this embodiment, in response to a user operating the operation unit 112 to issue an instruction to turn on the power, power is supplied to each block of the imaging device from a power supply unit (not shown). When power is supplied, the control unit 111 checks which mode the mode switch of the operation unit 112 is in, for example, shooting mode, playback mode, etc., based on an instruction signal from the operation unit 112. In the moving image recording mode, image data obtained by the imaging unit 101 and audio data obtained by the audio processing unit 104 are saved as one file. In the moving image recording mode, the control unit 111 first transmits a control signal to each block of the imaging device 100 to transition to a shooting standby state, and causes the blocks to operate as follows.

[0023] The imaging unit 101 converts an optical image of a subject captured by the optical lens 300 into an image signal using an imaging element, and performs analog-to-digital conversion, image adjustment processing, and the like to generate image data. The obtained image data is then sent to the display control unit 106, which causes it to be displayed on the display unit 107. The user prepares for shooting while looking at the screen displayed in this manner. Next, when the user operates the record button of the operation unit 112 to send an instruction signal to start shooting to the control unit 111, the control unit 111 sends an instruction signal to start shooting to each block of the imaging device 100, causing it to perform the following operations.

[0024] The imaging unit 101 converts an optical image of a subject captured by the optical lens 300 into an image signal using an imaging element, and performs analog-to-digital conversion, image adjustment processing, etc. to generate image data. The obtained image data is then transmitted to the display control unit 106, which then causes the display unit 107 to display the image. The obtained image data is also transmitted to the memory 105.

[0025] The audio processor 104 digitally converts the audio signal obtained by the microphone 113 to generate audio data. If the microphone 113 is configured with multiple microphones, the audio processor 104 processes the multiple obtained digital audio signals to generate multi-channel audio data. The obtained audio data is then transmitted to the memory 105.

[0026] The encoding processing unit 108 reads out the image data and audio data temporarily stored in the memory 105 and performs a predetermined encoding process to generate compressed image data, compressed audio data, etc. Then, the control unit 111 synthesizes the compressed image data and compressed audio data to form a data stream, which is output to the recording and reproducing unit 109. If the audio data is not compressed, the control unit 111 synthesizes the audio data and compressed image data stored in the memory 105 to form a data stream, which is output to the recording and reproducing unit 109. The recording and reproducing unit 109 writes the data stream as one moving image file to the recording medium 110 under the management of a file system such as UDF or FAT. The above operations are continued during shooting.

[0027] When the user operates the record button on the operation unit 112 to send an instruction signal to end shooting to the control unit 111, the control unit 111 sends an instruction signal to end shooting to each block of the imaging device 100, and causes them to perform the following operations: The imaging unit 101 and the audio processing unit 104 stop generating image data and audio data, respectively.

[0028] The encoding processor 108 reads the remaining image data and audio data stored in the memory, performs a predetermined encoding, and stops operation when compressed image data, compressed audio data, etc. are generated. If the audio data is not compressed, the operation stops when the generation of compressed image data is completed. Then, the control unit 111 synthesizes the last compressed image data and the compressed audio data or audio data to form a data stream, and outputs it to the recording and reproducing unit 109. The recording and reproducing unit 109 writes the data stream as one moving image file on the recording medium 110 under the management of a file system such as UDF or FAT. Then, when the supply of the data stream stops, the moving image file is completed and the recording operation is stopped.

[0029] When the recording operation stops, the control unit 111 transmits a control signal to each block of the imaging device 100 to transition to a shooting standby state, and returns to the shooting standby state. The image file recorded by the imaging device 100 is reproduced as a 3D image by a playback device (not shown) such as a head mounted display. In addition, the reproduced image has a recorded sound superimposed thereon, and a sound corresponding to the direction of rotation of the head of a user wearing a head mounted display, which is the playback device, is output, thereby providing a realistic image. The block configuration of the imaging device 100 and an overview of the operation during video recording have been described above.

[0030] The operation of making a narration sound non-directional according to an embodiment will be described below with reference to Fig. 2 to Fig. 8. Fig. 2 is a schematic diagram of an imaging device 100 equipped with a microphone 113. By connecting the microphone 113 capable of recording stereophonic sound to the audio input terminal of the camera, it becomes possible to record stereophonic sound data while shooting a video.

[0031] FIG. 3 is a schematic diagram of a microphone 113 capable of acquiring stereophonic sound using the Ambisonics method. As shown in FIG. 3(a), the microphone 113 has four microphone elements A to D. The four microphone elements A to D are arranged at a predetermined angle. In the embodiment, the microphone elements A to D are provided facing the four vertices of a cube shown in FIG. 3. If the space in the figure is defined by the directions shown in FIG. 3(b), the microphone element A faces the upper left front, the microphone element B faces the lower right front, the microphone element C faces the lower left rear, and the microphone element D faces the lower right rear. Since the four microphone elements A to D are unidirectional microphones, they acquire audio components in the direction in which they are each facing. The four pieces of audio data acquired by the microphone elements are called the A format of the Ambisonics format.

[0032] 4 shows a flowchart according to the first embodiment. In step S101, recording of moving images and audio is started when an instruction to start shooting is given from the operation unit 112 or the like. Although only audio recording will be described below, moving images may also be recorded at the same time.

[0033] In step S102, when shooting is started in step S101, voice acquisition is started by the microphone 113. At this time, it is preferable to acquire voice at any fixed interval and perform voice processing calculations described later. Voice acquisition is performed continuously until an instruction to end shooting is received from the operation unit 112 or the like.

[0034] 5 is a block diagram including a detailed configuration of the audio processing unit 104 for acquiring stereophonic data. In the audio processing unit 104, the audio signals acquired by each microphone element are subjected to analog gain by the amplifier 21, and are then subjected to analog-to-digital conversion by the A / D converter 22, converting them into digital signals.

[0035] In step S103, the Ambisonics A-format data acquired in step S102 is recorded on the recording medium 110. This step is unnecessary if the camera settings do not require recording of A-format data.

[0036] In step S104, the Ambisonics A format data acquired in step S102 is converted to B format data. In the 3D sound technology using Ambisonics, the A format cannot be used as is and must be converted to B format. The B format is audio data including omnidirectional data and bidirectional data in the 3D sound technology using Ambisonics. The B format is a data format consisting of an omnidirectional signal W, a front / rear directional signal X, a left / right directional signal Y, and an up / down directional signal Z. The B format encoder 23 converts the A format to the B format using equations (1) to (4). W = FLU + FRD + BLD + BRU (1) X=FLU+FRD―BLD―BRU ···(2) Y = FLU - FRD + BLD + BRU (3) Z = FLU + FRD + BLD + BRU (4)

[0037] Where: W: Omnidirectional signal (B format) X: Front-rear bidirectional signal (B format) Y: Bidirectional signal in left and right directions (B format) Z: Bidirectional signal in up and down directions (B format) FLU: Upper left front signal acquired by microphone element A (A format) FRD: Front right lower signal acquired by microphone element B (A format) BLD: Lower left rear signal acquired by microphone element C (A format) BRU: Signal in the upper right corner acquired by microphone element D (A format) It is.

[0038] By combining each B format signal and weighting it according to the direction the viewer is facing at the time of playback, it is possible to play back audio according to the direction the viewer is facing, achieving three-dimensional sound.

[0039] In step S105, the B format data converted in step S104 is recorded on the recording medium 110. This step is unnecessary if the camera settings do not allow recording of unedited B format data.

[0040] In step S106, it is confirmed whether or not a mode that makes the narration voice non-directional (for example, called narration mode) is set. If the narration mode is set, the process proceeds to step S107 and subsequent steps for making the narration non-directional. If the narration mode is not set, the process of making the narration non-directional is not performed, and the process proceeds to step S111.

[0041] In step S107, the position of the narration sound source to be made non-directional is set. For example, the user sets the position of the camera operator in advance using the camera settings, etc. As shown in FIG. 6, when the camera operator wants to record a narration while shooting, it is expected that the narration sound will come from the rear. Therefore, the position of the narration sound source is set to be behind the camera. Also, the camera may be provided with face recognition means, and the face of the person who will emit the narration sound may be registered in advance, and the position of the recognized face may be set as the position of the narration sound source.

[0042] In step S108, the narration sound is extracted using the narration sound source position set in step S107 and the Ambisonics B format data acquired in step S104. In post-production using the Ambisonics B format, the entire sound field can be easily rotated. For example, if you want to reproduce the sound field heard when the viewer changes the direction of his / her face by Φ on the horizontal plane as shown in Figure 7, you can reproduce it using equation (5).

[0043]

number

[0044] For the B format data calculated from equation (5), the narration sound source N is calculated from equation (6). N = W' + X' ... (6)

[0045] Here, the method of acquiring the narration voice is not limited to the above-mentioned formula. As will be described later, the narration voice may be acquired by a microphone other than the microphone 113. Furthermore, ambient sounds may be removed by applying a filter specialized for human voices to the extracted audio signal.

[0046] In step S109, the narration voice acquired in step S108 is synthesized with the Ambisonics B-format non-directional signal W acquired in step S104. For example, calculation is performed using equation (7). Wn = W + N (7) Wn: Omnidirectional signal including narration audio

[0047] Here, the method of acquiring the non-directional signal including the narration voice is not limited to the above formula.

[0048] Next, an example of a method for easily capturing narration audio will be described. As shown in Fig. 8, one of the four microphone elements of microphone 113 is arranged to face the direction of the cameraman. This allows the narration audio to be the audio from the microphone facing the direction of the cameraman, eliminating the need for the calculation in step S108 and reducing the load on audio processor 104.

[0049] In step S110, Ambisonics B format data in which the narration voice has been made non-directional, obtained by the processing up to step S109, is recorded on recording medium 110. In step S111, if an instruction to stop shooting is given from operation unit 112 or the like, the process proceeds to step S112, where recording of video and audio is stopped. If an instruction to stop shooting is not given, the process returns to step S102, and audio acquisition continues.

[0050] As described above, when recording stereoscopic video and stereoscopic sound while recording the audio of narration as a specific sound source, the narration sound source can be added without any sense of incongruity by making the characteristics of the microphone at the position of the narration sound source omnidirectional.

[0051] (Second Example) In the first embodiment, the narration voice was collected by the unidirectional microphone 113 and then made omnidirectional, so that the narration voice could be heard clearly no matter which direction the viewer was facing.

[0052] However, when viewing a reproduced image of a 3D image including a narration sound, some viewers may find the narration sound annoying. In such a case, the viewer turns off the audio output of the reproduction device, but turning off the audio output also makes it impossible to hear the surrounding sounds of the 3D image, which reduces the sense of realism. Therefore, in the second embodiment, an imaging device is provided that can realize realistic image reproduction by removing only the annoying narration sound from the audio output of the reproduction device and leaving the surrounding sounds of the 3D image.

[0053] In this embodiment, a narration microphone is provided for collecting narration voice in addition to the microphone 113 which is an Ambisonic microphone. The voice data collected by the narration microphone includes ambient sounds in addition to the narration voice. Therefore, only the narration voice is extracted from the voice data acquired by the narration microphone. This extracted voice data is subtracted from each of the voice data collected by the multiple microphones provided in the microphone 113, so that the voice data collected by the microphone 113 becomes voice data from which the narration voice has been removed.

[0054] 9 is a block diagram of an image capture device 100 according to a second embodiment, where blocks that are the same as those in FIG. 1 are given the same reference numerals and will not be described. The difference from FIG. 1, which shows the block configuration of the first embodiment, is that a narration microphone 114 for collecting narration audio is added. The narration microphone 114 is one or more unidirectional microphones built into the image capture device 100, and collected audio data is output to the audio processing unit 104.

[0055] In the following description of the second embodiment, for simplicity, the narration microphone 114 is a single unidirectional microphone. Also, the second embodiment will be described assuming that the narration microphone 114 is a built-in microphone of the imaging device 100, but as long as it can accurately collect the narration voice, it may be an external connection like the microphone 113. In the case of an external connection, the connection means with the imaging device 100 may be a wired connection or a wireless connection.

[0056] Next, an example of the placement of the narration microphone 114 will be described. Fig. 10 shows the camera that is the imaging device 100 when the microphone 113 is attached. The imaging angle of view of the imaging device 100 in this embodiment of the present invention is 180 degrees. Generally, when a photographer narrates while shooting, the action is performed while checking the screen being shot, so the narration sound source is from behind the imaging device 100, outside the imaging angle of view. Therefore, it is preferable to place the narration microphone 114 on the back of the imaging device 100.

[0057] Next, the audio processing unit 104 of the imaging device 100 equipped with a narration microphone 114 will be described with reference to Fig. 11. Note that in the audio processing unit 104 in Fig. 11, the same components as those in Fig. 2 are given the same reference numerals and their description will be omitted. The narration microphone 114 has also been described above, so its description will be omitted.

[0058] The audio data collected by the narration microphone 114 is analog-amplified by the amplifier 21 and converted to a digital signal by the AD converter 22. The AD-converted audio data contains not only the narration sound but also ambient sounds. The purpose of collecting audio with the narration microphone 114 is to remove the narration sound components from each piece of audio data collected by the microphone 113. For this reason, the audio extraction unit 24 uses a known sound source emphasis technique to extract human voice components, that is, only the narration sound, from the audio data collected by the narration microphone 114.

[0059] The sound collected by the microphone 114 is composed of a mixture of human voice components and other components. To emphasize only the human voice components from the collected sound data, for example, a method is known in which a real-number time-frequency mask in the discrete Fourier transform domain of the collected sound data is estimated by using a deep neural network.

[0060] The audio data from which only the narration audio components have been extracted by the audio extraction unit 24 is recorded by the recording unit 109 as the recorded audio of the narration, and is also output to the removal unit 26. The removal unit 26 performs subtraction processing using the audio data from which only the narration audio components have been extracted by the audio extraction unit 24 in order to remove the narration audio components contained in the audio data collected, amplified, and AD converted by the microphone 113, amplifier 21, and AD converter 22, respectively.

[0061] Here, the subtraction process in this embodiment has been described on the assumption that the output level of the audio data is subtracted with respect to the time axis. Alternatively, for example, the audio data from the microphones 113 and 114 may be frequency-converted and the results may be subtracted. However, this is not the only option as long as the narration audio component can be removed from the audio data from the microphone 113.

[0062] In the removal unit 26, a subtractor is connected to each of the AD converted signal outputs of the microphone 113. The sound collection directions of the multiple unidirectional microphones constituting the microphone 113 are different from each other. Therefore, the amount of subtraction during the subtraction process differs depending on the sound collection direction of the microphone.

[0063] For example, if the positional relationship between the microphones 113 and 114 has been determined, a subtraction process is performed on the audio data extracted by the audio extraction unit 24 from which only the narration audio components have been extracted, using a signal that has been weighted according to the positional relationship of each microphone that has been set in advance for each microphone. Also, sound may be collected before recording, and the relationship between the audio data of the microphones 113 and 114 may be measured, and the measurement result may be reflected in the subtraction process.

[0064] The audio data from microphone 113, from which the narration voice component has been removed by removal unit 26, is converted by B format encoder 23 from A format to B format. Each piece of audio data converted to B format is recorded by recording unit 109 as audio information containing only the narration voice. The recorded audio data containing only the narration voice and the audio data converted to B format from which the narration voice has been removed are recorded in time synchronization with the 3D video to be recorded.

[0065] Next, the operation of the second embodiment will be described with reference to the flowchart in Fig. 12. In step S201, when the photographer issues an instruction to start imaging by operating the operation unit 112, recording of moving images and audio is started. Although only audio recording will be described below, it is assumed that moving images and audio recording are performed simultaneously.

[0066] When the recording operation starts, audio is collected for a certain period of time by microphone 113 and microphone 114 (steps S202 and S302). The collected audio data output from microphone 113 is A-format audio data. Steps S203 and S303 temporarily record the audio data output from microphones 113 and 114, respectively. The recording destination may be memory 105, or the recording medium 110 may be used as the recording destination if the audio data output from microphone 113 is to be recorded and stored in A-format.

[0067] In step S304, the voice extraction unit 24 extracts frequency components of human voice from the voice data output from the microphone 114, and outputs the extracted voice as narration voice. The removal unit 26 subtracts the narration voice extracted by the voice extraction unit 26 from each piece of voice data from the microphone 113. This removes the narration voice components from each piece of voice data from the microphone 113 (step S204). The multiple pieces of voice data from the microphone 113 from which the narration voice components have been removed are output to the B-format encoder 25. The B-format encoder 25 converts each piece of voice data from the A-format to the B-format, which is the basic data format of the Ambisonics system (step S205). Voice in the B-format format is the basic data format.

[0068] In step S206, the recording unit 109 records the multiple pieces of audio data from the microphone 113, converted into the B format, in a predetermined audio file format on the recording medium 110. Similarly, in step S306, the narration audio data from the microphone 114 is recorded on the recording unit 110. Each piece of recorded audio data is synchronized in terms of the time axis. Each piece of audio data is an individual audio file, and is recorded and saved by being embedded in the video file.

[0069] When each recorded audio data is played back together with the stereoscopic video on a playback device (not shown), if the narration audio is turned off by the viewer's operation, only the audio corresponding to the stereoscopic video from the multiple audio data from microphone 113 is played back. On the other hand, if it is turned on, the narration audio is superimposed on the audio corresponding to the stereoscopic video and played back. The narration audio is recorded and saved in storage medium 110 so that the audio during playback is omnidirectional.

[0070] Returning to the flowchart of Fig. 12, in step S207, it is determined whether or not an instruction to stop recording has been issued by the user. If a stop command has been issued (Yes), the recording operation is stopped in step S208, and if a stop command has not been issued (No), the process returns to steps S202 and S303 to continue the recording operation.

[0071] As described above, according to the second embodiment, the audio recorded during stereoscopic video is separated into audio for stereoscopic video and audio for narration, and each is recorded in an independent audio file so that only the narration audio can be turned off during playback. As a result, even if the viewer feels that the narration audio is bothersome during stereoscopic video playback, only the narration audio can be turned off while other audio is played back, so the viewer can enjoy watching the video without losing the sense of realism of the stereoscopic video.

[0072] (Modification of the second embodiment) According to the second embodiment, when a viewer wants to know the explanation or various information of the video obtained by the narration during stereoscopic video playback, the narration voice is played without being muted. However, some people want to know the explanation or various information, but depending on the narration voice, some people may find the speaker's voice quality or speaking style unpleasant, so the viewer must endure watching the video without muting the narration voice, which makes it difficult to enjoy watching the video.

[0073] Therefore, as a modified example of the second embodiment, the audio data of the narration audio component extracted by the audio extraction unit 24 is converted into text information, and the text data converted into text information is recorded on the recording medium 110 as a text file.

[0074] Fig. 13 is a diagram for explaining the block configuration of the speech processing unit 104 according to this modified example, and the block configuration is almost the same as the block configuration diagram (Fig. 11) of the speech processing unit 104 according to the second embodiment, so the same blocks are given the same reference numerals and the description of the blocks is omitted. The difference between Fig. 13 and Fig. 11 is that a text conversion unit 27 is added.

[0075] The text conversion unit 27 converts the voice data obtained by extracting the narration voice components by the voice extraction unit 24 into text. A known voice recognition technology is used for the text conversion. For example, the text conversion unit 27 extracts voice features from the narration voice extracted by the voice extraction unit 24. A language model and a pronunciation dictionary for extracting phonemes from data learned by AI are stored in advance in the imaging device 100. The extracted voice features are compared with these to pick up language combinations, which are output as text data.

[0076] The text signal converted into text by text conversion unit 27 is processed in a predetermined text file format by recording unit 109, and then recorded on recording medium 110. When playing back stereoscopic video, icons are displayed on the playback screen as shown in Fig. 14 (for example, at the bottom left of 141, 142, 143) to notify the viewer of the current narration state. The viewer can cyclically switch and set the narration state by operating the icon.

[0077] Here, in the speech bubbles in Fig. 14, 14(a) indicates a state in which narration sound is being output from the audio output of the playback device value (not shown), while 14(b) and 14(c) indicate states in which it is not being output. In 14(a), the narration sound is enabled, and the ambient sound during recording and the narration sound are output from the audio output of the playback device. In 14(b), the narration sound is disabled, and only the ambient sound during recording is output from the audio output of the playback device. In 14(c), the narration sound is disabled, but a text version of the narration sound is superimposed on the playback screen of the playback device, and only the ambient sound during recording is output from the audio output of the playback device.

[0078] Furthermore, the text conversion unit 27 may add the intonation of the input narration voice to the text signal and output it as a text signal.

[0079] During stereoscopic video playback, the added information may be used to express the text display in Fig. 14(c), for example by making the font size larger or thicker, or by changing the font itself, depending on the intonation of the narration voice. Note that the intonation of the narration voice may be determined, for example, by detecting the strength and volume of the voice when extracting the voice characteristics in the text conversion unit 27, and judging the amount of change in the detection result over time.

[0080] In addition, although the microphone 114 in this embodiment has been described as a single unidirectional microphone, multiple microphones may be arranged in different directions to estimate the direction of the narration voice and add the sound source direction to the text signal. When playing back 3D video, the added information may be used to make the frame of the text display in Fig. 14(c) into a speech bubble shape aligned with the direction of the narration voice source.

[0081] As described above, according to the modified example of the second embodiment, even if the viewer turns off the output of the narration voice during stereoscopic video playback, the text data of the narration voice is also saved during recording. Therefore, the viewer can know the contents of the narration without listening to the narration voice by operating the narration voice.

Claims

1. An imaging means; A microphone having a plurality of microphone elements arranged at a predetermined angle and adapted to acquire a stereophonic sound signal; a voice processing unit for processing a voice signal acquired by the microphone, The imaging device is characterized in that the audio processing means converts the audio signal acquired by the microphone into stereophonic data, extracts an audio signal of a specific sound source from the audio signal acquired by the microphone, converts the audio signal of the specific sound source into omnidirectional data of the stereophonic data, and synthesizes it with the stereophonic data.

2. 2. The imaging apparatus according to claim 1, further comprising a control unit for controlling whether or not the audio signal of the specific sound source is converted into non-directional data of the stereophonic sound data.

3. 2. The imaging apparatus according to claim 1, further comprising a control means for setting the position of the specific sound source.

4. 4. The imaging apparatus according to claim 3, wherein the position of the specific sound source is the position of a photographer.

5. 5. The image pickup apparatus according to claim 4, wherein one of the plurality of microphone elements is arranged facing a photographer.

6. An imaging means; a first microphone having a plurality of microphone elements arranged at a predetermined angle and configured to acquire a stereophonic sound signal; a second microphone for acquiring sound from a predetermined direction; a voice processing unit for processing a voice signal acquired by the first microphone and the second microphone; The imaging device is characterized in that the audio processing means extracts an audio signal of a specific sound source from the audio signal from the second microphone, removes the extracted audio signal of the specific sound source from each audio signal acquired by each of the first microphone elements, and performs processing to convert each of the removed audio signals into stereophonic data.

7. 7. The imaging apparatus according to claim 6, wherein the predetermined direction is a direction of a photographer.

8. 7. The imaging apparatus according to claim 6, wherein the specific sound source is a sound having a frequency component of a human voice.

9. 7. The imaging apparatus according to claim 6, further comprising a recording means for recording the extracted audio signal and the stereophonic data.

10. 7. The imaging apparatus according to claim 6, wherein the audio processing means converts the extracted audio data into text and outputs the text data.

Images