Imaging device

Abstract

To provide an imaging device configured to add and record additional information matched with audio information of a recorded video to the record video using a method according to the additional information.SOLUTION: Included are: imaging means; a microphone; recording means for recording video data from the imaging means and audio data from the microphone on a recording medium; main scene determination means for determining main scenes in the video data recorded on the recording medium; audio analysis means for analyzing the audio data corresponding to the main scenes and extracting a keyword; obtaining means for obtaining a still image or video related to the keyword as a related image; and editing means for editing the recorded video data so as to edit the scene from which the keyword is extracted from among the main scenes. The editing means adds the related image obtained by the obtaining means to the scene from which the keyword is extracted in the video data, using a method according to the related image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0006] , ,

[0007] ,

[0001] The present invention relates to an imaging device.

Background Art

[0002] In recent years, cameras that perform regular and continuous shooting without the user giving a shooting instruction have been developed and put into practical use. For example, by installing such a camera at an arbitrary location indoors and having the camera automatically shoot, ordinary daily scenes can be recorded.

[0003] When playing back a video or audio recorded by such a camera, simply playing back the recorded video as it is may result in a video with little movement, which may lack interest for viewers. Therefore, it is desirable to edit the recorded video to make it a more suitable video for viewers and then play it back.

[0004] On the other hand, Patent Document 1 discloses a technique of extracting keywords from the input audio, searching for an image corresponding to the keywords from a plurality of predetermined images, and displaying the image.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In the conventional technology disclosed in Patent Document 1, related images corresponding to the audio can be displayed. However, when applying the same technology to the audio accompanying the recorded video, how to add and play back the related images during the playback of the video becomes a problem.

[0007] Therefore, the present invention aims to record additional information that corresponds to audio recorded together with the video, by adding it to the video in a manner appropriate to the additional information. [Means for solving the problem]

[0008] The present invention comprises an imaging means, a microphone, a recording means for recording video data from the imaging means and audio data from the microphone onto a recording medium, a main scene determination means for determining the main scenes within the video data recorded on the recording medium, an audio analysis means for analyzing audio data corresponding to the main scenes and extracting keywords, an acquisition means for acquiring still images or videos related to the keywords as related images, and an editing means for editing the recorded video data to edit the scenes from which the keywords have been extracted, wherein the editing means adds the related images acquired by the acquisition means to the scenes within the video data from which the keywords have been extracted in a manner corresponding to the related images. [Effects of the Invention]

[0009] According to the present invention, additional information corresponding to audio recorded together with the video is added to the video in a manner appropriate to the additional information and recorded. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram of an imaging device. [Figure 2] This is a flowchart for video recording processing. [Figure 3] This is a flowchart for the video editing process. [Figure 4] This figure shows an example of a video after filming and editing. [Modes for carrying out the invention]

[0011] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0012] The imaging device 100 in an embodiment of the present invention will be described below with reference to Figure 1. The lens 1001 is a lens group including a zoom lens and a focus lens. The lens control unit 1002 has the function of lens control processing that controls the focal length and aperture state of the lens 1001 based on the calculation results of the evaluation unit 1010 (described later) and subject information extracted by the recognition unit 1011. The CPU 1003 controls the entire imaging device 100. The CPU bus 1004 communicates between the CPU 1003 and each functional block.

[0013] The RAM control unit 1007 controls access to RAM 1006 based on RAM access requests from each functional block. The RAM bus 1005 communicates between the RAM control unit 1007 and each functional block. The RAM bus 1005 also has the function of mediating access to RAM 1006 from each functional block.

[0014] The imaging unit 1008 is an imaging means that converts the optical signal captured by the lens 1001 into an electrical signal using an image sensor (not shown), and has functions to correct lens aberrations in the obtained image data and to interpolate defective pixels of the image sensor. The developing unit 1009 applies debayer processing to the image data generated by the imaging unit 1008 to convert it into a signal consisting of a luminance signal and a chromatic difference signal, and has functions to perform developing processing such as noise reduction, correction of optical distortion, and optimization of the image contained in each signal. The evaluation unit 1010 has a function to calculate evaluation values ​​such as focus state and exposure state based on the image data generated by the imaging unit 1008.

[0015] The recognition unit 1011 has the function of detecting, recognizing, and generating subject information within the image data processed by the development unit 1009. For example, it can detect faces in the image data, output information indicating the position of the face, and further perform authentication of a specific person based on the characteristic information such as the face.

[0016] The display control unit 1012 has a function of display control processing that performs predetermined display processing on the image data developed by the developing unit 1009 and then outputs it to the display unit 1015. The display unit 1015 is a reproducing means for reproducing image data and is constituted by, for example, a liquid crystal display.

[0017] The video encoding unit 1013 has a function of video encoding processing that compresses and encodes the image data developed by the developing unit 1009 using a predetermined video compression method such as MEPG and converts it into a video with compressed information amount. The recording control unit 1014 has a function of recording control processing that records the image data developed by the developing unit 1009 on the recording medium 1016. The recording medium 1016 is, for example, a nonvolatile memory card, a hard disk, or the like.

[0018] The microphone 1017 converts sound into a sound signal. The microphone control unit 1018 is connected to the microphone 1017 and has a function of controlling the microphone 1017, starting and stopping sound collection, and acquiring sound from the microphone 1017. The control of the microphone 1017 is, for example, gain adjustment and status acquisition. The audio codec unit 1019 encodes or decodes the sound signal from the microphone 1017 using a predetermined encoding method such as AAC. The speaker 1020 outputs the sound signal decoded by the audio codec unit 1019.

[0019] The audio analysis unit 1021 is an audio analysis means for analyzing the sound acquired by the microphone 1017, and performs analysis such as detecting a specific keyword in the audio data and detecting a change in volume in the audio data.

[0020] The communication unit 1022 is a communication interface that connects the imaging device 100 to other devices by wire or wirelessly and transmits and receives image data, audio data, etc., and can also be connected to a network such as a wireless LAN or the Internet. The communication unit 1022 can transmit the image data acquired by the imaging device 100 and the image data recorded in the recording unit 1016 to the outside, and can receive image data and various information from external devices.

[0021] The operation unit 1023 receives various operations from the user for performing various settings of the imaging device 100. The related image acquisition unit 1024 acquires related images related to the keyword detected by the voice analysis unit 1021 from the Internet connected via the recording unit 1016 or the communication unit 1022.

[0022] The main scene determination unit 1025 is a main scene determination means for determining whether the scene during shooting is a main scene based on the image recognition result by the recognition unit 1011 or the voice analysis result by the voice analysis unit 1021. The video editing unit 1026 is a video editing means for editing the video recorded on the recording medium 1016 and adding the related images acquired by the related image acquisition unit 1024.

[0023] Next, referring to FIG. 2, the video shooting process in the imaging device 100 will be described. The flowchart of FIG. 2 is executed when the power switch (not shown) of the imaging device 100 is ON and the video shooting mode is set.

[0024] In step S201, video shooting is started by a shooting start instruction operation of the user on the operation unit 1023. In step S202, the CPU 1003 starts video recording processing and voice acquisition processing. In the video recording processing, continuous shooting is performed by the imaging unit 1008 at the set frame rate, and the acquired image signal is encoded by the video encoding unit 1013 and recorded in the RAM 1006 via the RAM bus 1005 and the RAM control unit 1007 as video data.

[0025] Also, in parallel with the video recording, voice is acquired from the microphone 1017 by the microphone control unit 1018, and the acquired voice data is encoded by the voice codec unit 1019 and recorded in the RAM 1006 as voice data accompanying the video data. The video data and voice data recorded in the RAM 1006 are recorded in the recording medium 1016 via the recording control unit 1014 as one video file. This series of video recording processing and voice acquisition processing is repeatedly executed until a shooting end instruction operation by the user.

[0026] In step S203, it is determined whether the user has issued an instruction to the control unit 1023 to end shooting. If an instruction to end shooting has been issued, the process proceeds to step S208. If no instruction to end shooting has been issued, the process proceeds to step S204.

[0027] In step S204, the main scene determination unit 1025 determines whether the scene being filmed is a main scene. A main scene is defined, for example, when the volume of the scene being filmed increases. If the scene being filmed is determined to be a main scene, the process proceeds to step S205. If it is determined not to be a main scene, the process returns to step S203.

[0028] In step S205, the voice analysis unit 1021 analyzes the voice data from the microphone 1017. In step S206, the voice analysis unit 1021 determines whether there are keywords in the voice data. The keywords are assumed to be set in advance by the user and recorded in the recording unit 1016. The keywords are assumed to be, for example, place names indicating a specific location or personal names indicating a specific person. If there are keywords in the voice data, the process proceeds to step S207. If there are no keywords, the process returns to step S203.

[0029] In step S207, the video encoding unit 1013 adds internal markers to scenes in the acquired video that are determined to contain keywords, in conjunction with the video data encoding process. The information added as internal markers includes information indicating the keywords acquired within that scene. The added internal markers serve as indicators for edit points in the video editing process described later.

[0030] In step S208, the video and audio recorded through continuous video recording from the start to the end of video recording are recorded as a single video file on the recording medium 1016, and the video file is closed. Then the series of processes ends.

[0031] Thus, during video recording and processing, if the recorded audio contains keywords, an internal marker is added to the corresponding scene in the video data to indicate an edit point.

[0032] Next, referring to Figure 3, the editing process for the recorded video file will be explained. The flowchart in Figure 3 is automatically executed by the CPU 1003 after the video recording process in Figure 2 is completed.

[0033] In step S301, the CPU 1003 determines whether there are any internal markers in the video file recorded on the recording medium 1016 that have not yet been edited. If there are any unprocessed internal markers, the process proceeds to step S302. If there are no unprocessed internal markers, the process ends.

[0034] In step S302, keywords are obtained from unprocessed internal markers. In step S303, the related image acquisition unit 1024 acquires related images related to the keywords obtained in step S302. Related images are, for example, if the keyword is a place name indicating a specific location, then images taken at the location indicated by the keyword. If the keyword indicates a specific person, then images that include the person indicated by the keyword as a subject are used. The related image acquisition unit 1024 acquires related images from other video files recorded on the recording medium 1016. The related image acquisition unit 1024 also acquires related images from servers on a network (not shown) connected via the communication unit 1022.

[0035] In step S304, it is determined whether the related image acquired in step S303 is a video. If it is a video, proceed to step 305. If it is a still image and not a video, proceed to step S307.

[0036] In step S305, it is determined whether the video acquired in step S303 is longer than a predetermined time. If it is longer than the predetermined time, the process proceeds to step S306. If it is shorter than the predetermined time, the process proceeds to step S307 with the video treated as a related video.

[0037] In step S306, a predetermined time segment is extracted from the video acquired in step S303 to create a related video for editing. The video can be extracted, for example, by cutting out a predetermined time segment from the beginning of the video, or by cutting out a predetermined time segment after a certain period of time has elapsed from the beginning.

[0038] In step S307, the video editing unit 1026 edits the related images acquired in step S303 or the related videos created in step S306 to add them to the scenes in the video files recorded by the recording unit 1016 that have internal markers.

[0039] If the related image acquired in step S303 is a still image, the video file is edited so that the still image is displayed for a predetermined time at the timing of the scene in the video file to which the internal marker is attached. If the related image acquired in step S303 is a video, the video or the related video extracted in step S306 is edited so that it is played at the timing of the scene in the video file to which the internal marker is attached.

[0040] The editing method here is determined by the content of the still images or videos that are added as related images to the video file. For example, if the related image contains the main subject, it is likely to be a related image of high importance to the viewer, so the video file is edited by inserting the related image into the scene for a predetermined amount of time. If the related image does not contain the main subject, it is unlikely to be a related image of high importance to the viewer, so the video file is edited by displaying it as a small superimposed image in the lower right corner of the scene.

[0041] After step S307, the process returns to step S301 and repeats the series of operations until video editing is completed for all internal markers attached to the video file.

[0042] In this way, by acquiring images related to the keywords detected by the audio analysis unit 1021 and creating a video with those related images attached, a more suitable video for the viewer can be obtained.

[0043] Figure 4 shows an example of a video captured by the imaging device 100 and a video after related images have been added to it. Figure 4(a) is a video captured by the imaging device 100. The video shown in Figure 4(a) has scenes 401 to 403 in chronological order. If the audio analysis unit 1021 detects a keyword in the audio data associated with scene 401, a related image is added after scene 401, creating a video like the one shown in Figure 4(b). In Figure 4(b), related image 404 is inserted between scene 401 and scene 402.

[0044] As described above, the objective of this embodiment is to provide an imaging device that adds additional information to the recorded video in a manner appropriate to the additional information, corresponding to the audio information of the recorded video.

[0045] In the embodiment described above, the case where the related image acquisition unit 1024 acquires only one related image was described. However, if multiple images are acquired, the images may be configured to prioritize the addition of the image with the most recent shooting date and time.

[0046] Furthermore, while the main scene in step S204 of Figure 2 is defined as a time when the volume of the audio data in the scene being filmed increases, it is not limited to this. For example, it could be when multiple subjects in the scene are speaking, or when there is a significant change in emotion indicated by the audio data.

[0047] Furthermore, to improve the viewing quality of the edited video files, it is acceptable to limit the number of edits made to the recorded video files to one or less per predetermined time interval.

[0048] To further enhance the viewing experience, it would be beneficial to prioritize playing video files with more edits over those with fewer edits, and to recommend video files with more edits to users as recommended videos.

[0049] Furthermore, although the video editing process is initiated immediately after the video recording is completed, the video editing process may also be initiated in response to a user's start command. The user's start command may be, for example, an operation on the control unit 1023, or a voice command via the microphone 1017. The display unit 1015 is, for example, a liquid crystal display, but is not limited to this. For example, it may have a projector function and project outside the imaging device 100.

[0050] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of Symbols]

[0051] 100 Imaging device

Claims

1. An imaging means; With a microphone a recording means for recording the moving image data from the imaging means and the audio data from the microphone on a recording medium; a key scene determination means for determining key scenes in the video data recorded on the recording medium; a voice analysis means for analyzing the voice data corresponding to the main scenes and extracting keywords; An acquisition means for acquiring still images or videos related to the keyword as related images; an editing means for editing the recorded video data so as to edit a scene from which the keyword is extracted among the main scenes; The imaging device according to claim 1, wherein the editing means adds the related image acquired by the acquisition means to a scene in the video data from which the keyword was extracted in a manner corresponding to the related image.

2. 2. The imaging apparatus according to claim 1, wherein the acquisition means acquires the related image from other image data recorded on the recording medium or from a network.

3. The imaging device according to claim 1, characterized in that, when the related image acquired by the acquisition means is a video, the editing means creates an excised video by extracting a predetermined period of time from the related image, and adds the excised video to the video data.

4. 2. The imaging device according to claim 1, wherein, when the related image contains a main subject, the editing means inserts the related image into a scene from which the keyword was extracted in the video data.

5. The imaging device according to claim 1, characterized in that if the related image does not contain a main subject, the editing means superimposes the related image in a small size on a scene from which the keyword is extracted in the video data.

6. The imaging device according to claim 1, characterized in that when there are multiple related images acquired by the acquisition means, the editing means adds the keyword in the video data to the scene from which the keyword was extracted, preferentially from the image with the most recent shooting date and time among the multiple related images.

7. 2. The imaging apparatus according to claim 1, wherein the main scene determination means determines the main scene based on a volume of sound corresponding to the video data.

8. 2. The imaging apparatus according to claim 1, wherein the main scene determination means determines a scene in which a plurality of subjects are talking in the video data as the main scene.

9. 2. The imaging device according to claim 1, wherein the key scene determination means determines a scene in the video data in which there is a large change in emotional tone of the sound as the key scene.

10. 2. The imaging device according to claim 1, wherein, when the keyword indicates a place name, the acquisition means acquires still images and videos captured at a place indicated by the keyword.

11. 2. The imaging apparatus according to claim 1, wherein, when the keyword represents a person's name, the acquisition means acquires still images and moving images that include the person represented by the keyword as a subject.

12. 2. The image pickup apparatus according to claim 1, wherein the number of times that the editing means edits the moving image data is set to once or less for each predetermined time period of the moving image data.

13. 2. The imaging apparatus according to claim 1, further comprising a playback unit that plays back, as a recommended video, the video data that has been edited by the editing unit more than any other video data recorded on the recording medium.

Images