Imaging apparatus and imaging system

Abstract

To provide an imaging apparatus or an imaging system capable of generating a video file with synchronized audio and video even when starting recording in the middle of a song.SOLUTION: When receiving an operation input to execute audio playback, an imaging apparatus 1 causes an audio decoder 141 to execute the operation to output audio data to a speaker 12. At this time, the playback position in the audio encoding data is calculated, and playback position data is sent to a multiplexing unit 144. When receiving the operation input to start video recording, a video encoding unit 143 and the multiplexing unit 144 are caused to execute the operation. The multiplexing unit 144 generates a video file by multiplexing the playback position data onto the video encoding data and the audio encoding data.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an imaging device and an imaging system. [Background technology]

[0002] Patent Document 1 discloses a technique for an imaging device that enables easy retrieval of a part of a video that is shot in sync with audio, the part corresponding to the timing of switching the audio. This imaging device includes a video shooting means for shooting a video in synchronization with a predetermined audio, a timing information means for acquiring timing information output from an external device related to a predetermined part of the predetermined audio, and a timing information storage means for storing the acquired timing information in association with the video being shot. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2015-32846 Summary of the Invention [Problem to be solved by the invention]

[0004] In a conventional imaging device or imaging system, when a video is shot while playing a specific sound, the recorded sound data includes sounds made during shooting other than the specific sound, so that after shooting is completed, the audio data in the video data must be replaced with the specific sound data. The present disclosure is directed to eliminating the need for such an editing process.

[0005] The present disclosure provides a technique that enables an imaging device or imaging system to generate a video file in which audio and video are synchronized, even if video shooting is started in the middle of a song. [Means for solving the problem]

[0006] An imaging device according to the present disclosure includes an imaging sensor, a speaker, an operation unit that receives operation inputs related to video shooting and audio playback, a memory unit, a video encoding unit that encodes video data captured by the imaging sensor to generate video encoded data, an audio decoding unit that decodes first audio encoded data stored in the memory unit and outputs it to the speaker as audio data, a multiplexing unit that multiplexes the video encoded data and the first audio encoded data or second audio encoded data obtained by re-encoding the audio data to generate a video file, and a control unit that controls the operation of the imaging device, wherein when an operation input for performing audio playback is received via the operation unit, the control unit causes the audio decoding unit to perform an operation, and performs a process of calculating a playback position in the first audio encoded data and sends the calculated playback position data to the multiplexing unit, and when an operation input for performing video shooting is received via the operation unit, the control unit causes the video encoding unit and the multiplexing unit to perform operations, and the multiplexing unit multiplexes the playback position data onto the video encoded data and the first or second audio encoded data to generate a video file. Effect of the Invention

[0007] According to the imaging device according to the present disclosure, even if video shooting is started in the middle of a song, a video file in which the audio and video are synchronized can be generated. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1A is a flowchart showing a situation according to the first embodiment, and FIG. 1B is a flowchart showing an operation in the situation shown in FIG. [Diagram 2] Schematic configuration of an imaging device according to a first embodiment [Diagram 3] Main configuration example of an imaging device according to a first embodiment [Figure 4] Image of the recorded video file [Diagram 5] FIG. 10A is a flowchart showing a situation according to a second embodiment, and FIG. 10B is a flowchart showing an operation in the situation shown in FIG. [Figure 6]Schematic configuration of an imaging system according to a second embodiment [Figure 7] Main configuration example of an imaging system according to a second embodiment [Figure 8] Main configuration example of an imaging system according to a modified example of the second embodiment [Figure 9] Main configuration example of an imaging system according to a modified example of the second embodiment [Figure 10] Configuration example of an imaging device according to another embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or duplicate explanation of substantially the same configuration may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the inventor(s) provide the attached drawings and the following explanation so that those skilled in the art can fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims.

[0010] (First embodiment) Fig. 1(a) shows a situation according to the first embodiment, and Fig. 1(b) shows an operation in the situation of Fig. 1(a). In Fig. 1, person A is using an imaging device 1 to shoot a video (e.g., a dance video) of person B.

[0011] As shown in FIG. 2, the imaging device 1 includes an imaging sensor 11, a speaker 12, an operation unit 13, a video / audio processing unit 14, a control unit 15, and a storage unit 16. The imaging device 1 can capture a video using the imaging sensor 11, encode the captured video using the video / audio processing unit 14 to generate a video file, and store the video file in the storage unit 16. The imaging device 1 can also decode audio encoded data stored in the storage unit 16 using the video / audio processing unit 14, and play audio data from the speaker 12. The operation unit 13 is composed of, for example, a button, a touch panel, etc., and receives operation input related to video capture and audio playback by a user. The control unit 15 controls the operation of the imaging device 1 according to the operation input via the operation unit 13.

[0012] When video shooting starts, Person A plays back the audio data using the imaging device 1. Person B performs an action, for example, a dance while listening to the played back sound, and Person A continues shooting while listening to the played back sound. The imaging device 1 multiplexes the audio data onto the shot video data in real time. When video shooting ends, a video file in which the audio data is synchronized with the shot video is recorded.

[0013] 3 shows an example of the main configuration of the imaging device 1 according to this embodiment. The MP3 decoding unit 141, the AAC-LC encoding unit 142, the H.264 / H.265 encoding unit 143, and the MP4 multiplexing unit 144 are included in the video / audio processing unit 14. The playback position calculation processing unit 151 is included in the control unit 15.

[0014] MP3 decoding unit 141, which is an example of an audio decoding unit, decodes MP3 data, which is an example of audio encoded data stored in storage unit 16, and outputs the decoded data as audio data to speaker 12. AAC-LC encoding unit 142 re-encodes the audio data output to speaker 12 into AAC-LC data.

[0015] An H.264 / H.265 encoding unit 143, which is an example of a video encoding unit, encodes video data captured by the imaging sensor 11 to generate video encoded data. An MP4 multiplexing unit 144, which is an example of a multiplexing unit, multiplexes the video encoded data and AAC-LC data to generate a video file of MP4 data, and stores it in the storage unit 16.

[0016] When the control unit 15 receives an operation input for executing audio playback via the operation unit 13, it causes the MP3 decoding unit 141 to execute the operation. At this time, the playback position calculation processing unit 151 executes a process for calculating the playback position in the MP3 data. Here, for example, the time from the beginning of the MP3 data is converted into a time code. For example, the time code at the beginning of the MP3 data is set to 00:00:00:00, and anything less than a second is converted into a video frame. The time code, which is an example of playback position data, is sent to the MP4 multiplexing unit 144.

[0017] Furthermore, when control unit 15 receives an operation input for executing video shooting via operation unit 13, it causes AAC-LC encoding unit 142, H.264 / H.265 encoding unit 143, and MP4 multiplexing unit 144 to execute operations. MP4 multiplexing unit 144 multiplexes a time code onto the video encoded data and the AAC-LC data to generate a video file.

[0018] FIG. 4 is an image diagram of a generated video file. Person A shoots a video of Person B while playing MP3 audio data, which is an original sound source, using the imaging device 1. For example, in take 1, a file P1000001.MP4 is generated. In this file P1000001.MP4, video encoding data, audio encoding data (AAC-LC), and a time code are multiplexed. The beginning of the time code is "00:00:40:09" corresponding to the start of video shooting, and the end of the time code is "00:03:20:00" corresponding to the end of video shooting. Similarly, in take 2, a file P1000002.MP4 is generated. In this file P1000002.MP4, video encoding data, audio encoding data (AAC-LC), and a time code are multiplexed. The beginning of the time code is "00:01:10:10", which corresponds to the start of video recording, and the end is "00:01:47:26", which corresponds to the end of video recording.

[0019] As described above, according to the present embodiment, when the imaging device 1 receives an operation input for playing back audio, the imaging device 1 causes the MP3 decoding unit 141 to perform an operation, decodes the MP3 data stored in the storage unit 16, and outputs the audio to the speaker 12. At this time, a time code in the MP3 data is calculated and sent to the MP4 multiplexing unit 144. When the imaging device 1 receives an operation input for shooting a moving image, the imaging device 1 causes the AAC-LC encoding unit 142, the H.264 / H.265 encoding unit 143, and the MP4 multiplexing unit 144 to perform an operation. The AAC-LC encoding unit 142 re-encodes the audio data output to the speaker 12 into AAC-LC data. The H.264 / H.265 encoding unit 143 encodes the moving image data captured by the imaging sensor 11 to generate encoded moving image data. The MP4 multiplexing unit 144 multiplexes the time code onto the encoded moving image data and the AAC-LC data to generate a moving image file. As a result, even if video shooting is started in the middle of a song, a video file in which audio and video are synchronized can be generated. With the imaging device 1 according to this embodiment, even if video shooting is started while playing a specific audio, the audio of the recorded video data becomes the specific audio data, eliminating the need for an editing process after shooting. During editing, a video can be easily created by combining multiple video files in which audio and video are synchronized.

[0020] Second embodiment Fig. 5(a) shows a situation according to the second embodiment, and Fig. 5(b) shows the operation in the situation of Fig. 5(a). In Fig. 5, Person A is using an imaging device 2 to shoot a video (e.g., a dance video) of Person B while playing back sound with an audio device 3 as an audio playback device. The imaging system according to this embodiment includes the imaging device 2 and the audio device 3.

[0021] As shown in Fig. 6, the imaging device 2 includes an imaging sensor 21, a microphone 22, an operation unit 23, a video / audio processing unit 24, a control unit 25, a storage unit 26, and a communication unit 27. The imaging device 2 can capture a video using the imaging sensor 21, encode the captured video using the video / audio processing unit 24 to generate a video file, and store the video file in the storage unit 26. The imaging device 2 can also encode audio data collected by the microphone 22 using the video / audio processing unit 24, and store the encoded audio data in the storage unit 26. The operation unit 23 is composed of, for example, a button, a touch panel, etc., and receives an operation input related to video capture by a user. The control unit 25 controls the operation of the imaging device 2 according to the operation input via the operation unit 23.

[0022] The audio device 3 includes a speaker 31, an operation unit 32, an audio processing unit 33, a control unit 34, a storage unit 35, and a communication unit 36. The audio device 3 can decode audio encoded data stored in the storage unit 35 using the audio processing unit 33 and output the decoded data from the speaker 31 as audio data. The operation unit 32 is composed of, for example, a button or a touch panel, and receives operation input related to audio playback from a user. The control unit 34 controls the operation of the audio device 3 in accordance with the operation input via the operation unit 32.

[0023] Before starting video shooting, Person A connects the imaging device 2 and the audio device 3 so that they can communicate with each other. This connection may be a wireless connection such as Bluetooth (registered trademark) or a wired connection. When video shooting starts, Person A plays back audio data on the audio device 3. Person B performs an action, such as dancing, while listening to the played back sound, and Person A performs shooting with the imaging device 2 while listening to the played back sound. The imaging device 2 multiplexes the audio data onto the shot video data in real time. When video shooting ends, the audio data is synchronized with the shot video and a video file is recorded.

[0024] 7 shows an example of the main configuration of the imaging device 2 and audio device 3 according to this embodiment. In the imaging device 2, the AAC-LC encoding unit 241, the H.264 / H.265 encoding unit 242, and the MP4 multiplexing unit 243 are included in the video / audio processing unit 24. In the audio device 3, the MP3 decoding unit 331 is included in the audio processing unit 33. Moreover, the playback position calculation processing unit 341 is included in the control unit 34.

[0025] In the audio device 3, an MP3 decoding section 331, which is an example of an audio decoding section, decodes MP3 data, which is an example of audio encoded data stored in the storage section 35, and outputs the decoded data to the speaker 31 as audio data.

[0026] In the imaging device 2, the AAC-LC encoding unit 241 encodes audio data collected by the microphone 22 into AAC-LC data. The H.264 / H.265 encoding unit 242, which is an example of a moving image encoding unit, encodes moving image data captured by the imaging sensor 21 to generate encoded moving image data. The MP4 multiplexing unit 243, which is an example of a multiplexing unit, multiplexes the encoded moving image data and the AAC-LC data to generate a moving image file of MP4 data, and stores it in the storage unit 26.

[0027] In the audio device 3, when the control unit 34 receives an operation input for executing audio playback via the operation unit 32, it causes the MP3 decoding unit 331 to execute the operation. At this time, the playback position calculation processing unit 341 executes a process for calculating the playback position in the MP3 data. Here, for example, the time from the beginning of the MP3 data is converted into a time code. For example, the time code at the beginning of the MP3 data is set to 00:00:00:00, and values ​​below a second are converted into video frames. The time code, which is an example of playback position data, is sent to the MP4 multiplexing unit 243 of the imaging device 2.

[0028] In the imaging device 2, when the control unit 25 receives an operation input for executing video shooting via the operation unit 23, it causes the AAC-LC encoding unit 241, the H.264 / H.265 encoding unit 242, and the MP4 multiplexing unit 243 to execute operations. The MP4 multiplexing unit 243 multiplexes a time code onto the video encoding data and the AAC-LC data to generate a video file. As a result, a video file such as that shown in FIG. 4 is generated, similar to the first embodiment.

[0029] According to this embodiment, when the audio device 3 receives an operation input for playing back audio, the audio device 3 causes the MP3 decoding unit 331 to perform an operation, decodes the MP3 data stored in the storage unit 35, and outputs the audio to the speaker 32. At this time, a time code in the MP3 data is calculated and sent to the MP4 multiplexing unit 243 of the imaging device 2. When the imaging device 2 receives an operation input for shooting a moving image, the imaging device 2 causes the AAC-LC encoding unit 241, the H.264 / H.265 encoding unit 242, and the MP4 multiplexing unit 243 to perform an operation. The AAC-LC encoding unit 241 re-encodes the audio data input via the microphone 22 into AAC-LC data. The H.264 / H.265 encoding unit 242 encodes the moving image data captured by the imaging sensor 21 to generate encoded moving image data. The MP4 multiplexing unit 243 multiplexes the time code onto the encoded moving image data and the AAC-LC data to generate a moving image file. This makes it possible to generate a video file in which the audio and video are synchronized, even if you start recording a video in the middle of a song.

[0030] 8 and 9 show an example of the main configuration of an imaging device 2 and an audio device 3 according to a modified example of this embodiment. In the configuration of Fig. 7, the imaging device 2 receives audio via a microphone 22, and the audio data is input in analog form, resulting in poor sound quality. The modified examples shown in Figs. 8 and 9 improve on this point.

[0031] 8, the imaging device 2 stores in the storage unit 26 the same MP3 data as the MP3 data stored in the storage unit 35 by the audio device 3. The imaging device 2 reads out the MP3 data stored in the storage unit 26 in accordance with the received time code. The MP4 multiplexing unit 243 multiplexes the video encoding data and the MP3 data with the time code to generate a video file.

[0032] In the configuration example of Fig. 9, the MP3 data stored in the storage unit 35 of the audio device 3 is transmitted to the imaging device 2 by, for example, Wi-Fi or Bluetooth. The imaging device 2 receives the MP3 data. The MP4 multiplexing unit 243 multiplexes the video encoding data and the MP3 data with a time code to generate a video file. These modifications can improve the sound quality of the audio data included in the video file.

[0033] (Other embodiments) As described above, the above-mentioned embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are appropriately made. In addition, it is also possible to combine the components described in the above-mentioned embodiments to create new embodiments.

[0034] In the above-described embodiment, the reproduced audio data is re-encoded into AAC-LC data and multiplexed onto the video file, but instead, for example, MP3 data may be multiplexed onto the video file.

[0035] Fig. 10 shows another example of the configuration of the main parts of the imaging device 1. In Fig. 10, the AAC-LC encoding unit 142 in the configuration of Fig. 3 is omitted, and the MP3 data stored in the storage unit 16 is decoded by an MP3 decoding unit 141 and sent to an MP4 multiplexing unit 144. The MP4 multiplexing unit 144 multiplexes the video encoded data and the MP3 data with a time code to generate a video file.

[0036] As described above, the embodiments have been described as examples of the technology in the present disclosure, and the accompanying drawings and detailed description have been provided for this purpose.

[0037] Therefore, among the components described in the attached drawings and the detailed description, not only are there components essential for solving the problem, but there may also be components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that such non-essential components are described in the attached drawings or the detailed description should not be interpreted as immediately indicating that such non-essential components are essential.

[0038] Furthermore, since the above-described embodiments are intended to illustrate the technology in the present disclosure, various modifications, substitutions, additions, omissions, and the like can be made within the scope of the claims or their equivalents. [Industrial Applicability]

[0039] The present disclosure is applicable to an imaging device that captures images in sync with audio. [Explanation of symbols]

[0040] 1. Imaging device 2. Imaging device 3 Audio equipment (audio playback equipment) 11 Image sensor 12 Speaker 13 Control section 15 Control section 16 Memory section 21 Image sensor 22. Mike 23 Control section 25 Control Unit 26 Memory section 31 Speaker 32 Operation section 34 Control section 35 Storage section 141 MP3 Decoding Unit (Audio Decoding Unit) 143 H.264 / H.265 encoding unit (video encoding unit) 144 MP4 multiplexing unit (multiplexing unit) 242 H.264 / H.265 Symbolization Department (Animation Symbolization Department) 243 MP4 multiplexing unit (multiplexing unit) 331 MP3 multiplexing department (audio multiplexing department)

Claims

1. 1. An imaging device, comprising: An imaging sensor; A speaker and an operation unit for receiving operation inputs related to video recording and audio playback; A storage unit; a moving image encoding unit that encodes moving image data captured by the imaging sensor and generates encoded moving image data; an audio decoding unit that decodes the first audio encoded data stored in the storage unit and outputs the decoded data as audio data to the speaker; a multiplexing unit that multiplexes the video encoded data and the first audio encoded data or second audio encoded data obtained by re-encoding the audio data to generate a video file; A control unit that controls an operation of the imaging device, The control unit is when receiving an operation input for executing audio playback via the operation unit, making the audio decoding unit execute an operation and performing a process of calculating a playback position in the first audio encoded data, and sending the calculated playback position data to the multiplexing unit; when an operation input for executing video shooting is received via the operation unit, the video encoding unit and the multiplexing unit are caused to execute operations; The multiplexing unit multiplexes the playback position data onto the video encoded data and the first or second audio encoded data to generate a video file. Imaging device.

2. An imaging system including an audio playback device and an imaging device, The audio reproduction device includes: A speaker and a first operation unit that receives an operation input related to audio reproduction; A first storage unit; an audio decoding unit that decodes the first audio encoded data stored in the first storage unit and outputs the decoded data as audio data to the speaker; a first control unit for controlling an operation of the audio playback device; The imaging device includes: An imaging sensor; a second operation unit that receives an operation input related to video shooting; A second storage unit; a moving image encoding unit that encodes moving image data captured by the imaging sensor and generates encoded moving image data; a multiplexing unit that multiplexes the video encoded data and second audio encoded data corresponding to the first audio encoded data to generate a video file and stores the video file in the second storage unit; a second control unit that controls an operation of the imaging device, The first control unit in the audio reproducing device when receiving an operation input for playing back audio via the first operation unit, making the audio decoding unit execute an operation and calculating a playback position in the first audio encoded data, and sending the calculated playback position data to the imaging device; The second control unit in the imaging device is when an operation input for executing moving image shooting is received via the second operation unit, the moving image encoding unit and the multiplexing unit are caused to execute operations; The multiplexing unit multiplexes the playback position data sent from the audio playback device onto the video encoded data and the second audio encoded data to generate a video file. Imaging system.

3. 3. The imaging system according to claim 2, The imaging device includes: With a microphone a voice encoding unit that encodes the voice data collected by the microphone and generates the second voice encoded data. Imaging system.

4. 3. The imaging system according to claim 2, The imaging system according to the present invention, wherein the imaging device stores, in the second storage unit, as the second audio encoded data, data identical to the first audio encoded data.

5. 3. The imaging system according to claim 2, The first audio encoded data is transmitted as the second audio encoded data from the audio reproduction device to the imaging device. Imaging system.

Images