Imaging device

Abstract

To automatically select BGM when projecting an image on the basis of surrounding audio, audio volume, and a captured image at the time of capturing the image.SOLUTION: An imaging device includes: imaging means; sound obtaining means; recording means for recording still images captured by the imaging means and audio obtained by the sound obtaining means on a recording medium in association with each other; video generation means for selecting a plurality of recommended images from the still images recorded by the recording means and generating video; BGM selection means for selecting BGM of the recommended image on the basis of the audio corresponding to the recommended image or the recommended image; BGM determination means for determining BGM of the video generated by the video generation means from the BGM selected by the BGM selection means; projection means for projecting the video generated by the video generation means; and BGM output means for outputting the BGM determined by the BGM determination means as BGM of the video projected by the projection means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device.

Background Art

[0002] In recent years, devices such as cameras with projection devices that can immediately project and display images taken have emerged. There are also devices with functions such as automatically selecting recommended images, and devices that create movies with music such as BGM (Background music) from captured images.

[0003] In Patent Document 1, a method is disclosed in which, when playing a digest movie of a moving image, the audio recorded in the moving image is not played, and the user selects and plays BGM.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the prior art disclosed in the above patent documents, since the user selects and plays BGM, it takes time to select BGM. Also, when displaying or projecting the captured image, the flowing BGM is not necessarily the BGM suitable for the image, and there may be cases where it is not suitable for the user.

[0006] Therefore, an object of the present invention is to provide a viewing opportunity suitable for the user by selecting and playing BGM suitable for a plurality of captured images.

Means for Solving the Problems

[0007] The system includes an imaging means, an audio acquisition means, a recording means for recording the captured still images and the audio acquired by the audio acquisition means onto a recording medium in association with each other, an image generation means for selecting a plurality of recommended images from the still images recorded by the recording means and generating an image, a background music selection means for selecting background music suitable for the recommended images based on the audio corresponding to the recommended images or information about the recommended images, a background music determination means for determining a background music suitable for the image generated by the image generation means from among the background music selected by the background music selection means, a projection means for projecting the image generated by the image generation means, and a background music output means for outputting the background music determined by the background music determination means as background music for the image projected by the projection means. [Effects of the Invention]

[0008] According to the present invention, by selecting and playing background music suitable for multiple captured images, it is possible to provide users with a suitable viewing experience. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram illustrating the configuration of the imaging device. [Figure 2] This is a flowchart showing the operation of the imaging device. [Figure 3] This is a block diagram illustrating the configuration of the imaging device. [Figure 4] This is a flowchart showing the operation of the imaging device. [Modes for carrying out the invention]

[0010] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0011] Figure 1 is a block diagram illustrating the configuration of the imaging device in the first embodiment.

[0012] The imaging device 100 includes an imaging unit 101, an audio acquisition unit 102, a recording unit 103, an image generation unit 104, a BGM selection unit 105, a projection unit 106, a BGM determination unit 107, and a BGM output unit 108. The imaging unit 101 captures images and can be performed using an image sensor such as a CMOS sensor or a CCD sensor. The audio acquisition unit 102 has a microphone and acquires ambient sound when capturing images. The recording unit 103 records the acquired images and sound onto a recording medium. Examples of recording media include memory cards and hard disks. External recording media may also be used using a communication unit such as wireless or wired, but this is not particularly limited here.

[0013] The video generation unit 104 selects multiple recommended images from the captured images and automatically generates a video. The BGM selection unit 105 selects the background music for the recommended images based on the audio corresponding to the recommended images or the information of the recommended images. The BGM determination unit 107 determines the background music for the video from the selected background music. The BGM output unit 108 outputs the background music. The projection unit 106 projects the generated video.

[0014] In this embodiment, the imaging unit 101 captures and records still images. The still images captured by the imaging unit 101 and the audio acquired by the audio acquisition unit 102 are sent to the recording unit 103 for recording. The audio acquisition unit 102 acquires audio during the shutter release period when capturing still images, or for a predetermined time after the shutter release, or for a predetermined time before and after the shutter release.

[0015] A longer period for acquiring sound allows for the acquisition of more sound, but this also increases the amount of information and requires more storage capacity. For example, if the sound is only 1-2 seconds long, it is usually possible to hear the sound at the word level. The sound can be the voices of people or music in the vicinity of the imaging device 100.

[0016] The recording unit 103 records multiple still images captured in this manner, along with their corresponding audio. The video generation unit 104 receives multiple still images and the audio associated with these still images, read from the recording medium, from the recording unit 103. The video generation unit 104 selects the still images sent from the recording unit 103.

[0017] For example, if there are multiple images of a family (hereinafter, a family of three consisting of a father, mother, and child will be used as an example), a certain number of images in which all family members are smiling will be selected as recommended images. The image selection method involves extracting features using various techniques. For image selection, thousands to tens of thousands of image data are stored in a machine learning database beforehand, and a selector is generated using this image data to enable the selection of target objects through machine learning.

[0018] Various methods have been proposed for feature calculation and selector generation, such as HOG features and SVM classifiers, but we will not limit ourselves to any particular method here. The number of recommended images to select can be set arbitrarily by the user. For example, 10 images. Alternatively, it can be set by the length of the automatically generated video. The automatically generated video is created as a slideshow that displays multiple recommended images in sequence. For example, if the automatically generated video is 1 minute long and the display time per image is 5 seconds, a total of 12 images will be selected as recommended images. These can also be set arbitrarily by the user.

[0019] Here, we have described the video generation unit 104 as selecting the image sent from the recording unit 103, but this is not the only option. The recording unit 103 may make the selection, or the user may make a selection at will.

[0020] To the BGM selection unit 105, a recommended image and the audio recorded in association with it are sent from the video generation unit 104. In the BGM selection unit 105, a piece of music is estimated from the music or voice included in the audio, and a BGM is selected from the estimation result. For example, when the theme music of a theme park is recorded in the audio, the theme music of the theme park is selected as the BGM. When a child's voice is used as the audio and the name of a character in the theme park is recorded, the theme music of that character is selected as the BGM. If there is no character-specific theme music, the theme music of the theme park may be used as the BGM.

[0021] In selecting BGM from music, it is possible by using the same method as in image selection, but it is not particularly limited here. The same applies to selecting BGM from voice. Also, instead of from the audio associated with the image, a suitable piece of music may be estimated from the subject or character string included in the image, and a BGM may be selected from the estimation result. For example, when a character of a theme park is included in the image, the BGM may be selected from that character information. When the name of the theme park is included as a character string in the image, the BGM may be selected from the name of that theme park.

[0022] Also, the scene of the recommended image may be identified, and a BGM suitable for the scene identification result may be selected. For example, in the case of an image of a children's sports meet, music suitable for the sports meet may be selected as the BGM. Also, a piece of music suitable for a specific individual, age, or scene photographed in the recommended image may be associated in advance, and the BGM may be selected.

[0023] At this time, if the specific individual is a child, a song preferred by the child, if the specific individual is a father, a popular song from when he was young, etc. are associated in advance. If it is by age, popular songs corresponding to the age are associated in advance. If it is by scene, for example, songs corresponding to scenes such as a sports meet or a festival are associated in advance. Thereby, the selection of the BGM can be facilitated.

[0024] Furthermore, the BGM selection unit 105 retrieves background music (BGM) from a server on the network or from an internal storage device. The BGM determination unit 107 determines the background music for the video based on the number of BGM tracks selected by the BGM selection unit 105. Alternatively, if there are multiple BGM tracks with the most selections, the BGM determination unit 107 randomly selects a BGM from among the BGM tracks with the most selections.

[0025] For example, if there are 10 recommended images and the BGM selection unit 105 selects 4 tracks of BGM1, 3 tracks of BGM2, and 3 tracks of BGM3, then BGM1 will be selected as the background music. Also, if there are 4 tracks of BGM1, 4 tracks of BGM2, and 2 tracks of BGM3, then since there are the same number of BGM1 and BGM2 tracks, either BGM1 or BGM2 will be randomly selected as the background music.

[0026] The projection unit 106 performs projection using optical functions such as a projector. This allows the image generated by the image generation unit 104 to be projected and viewed immediately. The BGM output unit 108 outputs the background music determined by the BGM determination unit 107 via a speaker. The BGM is output along with the image projected by the projection unit 106. In this way, when projecting a captured still image as a video, background music suitable for the captured image is selected. By outputting the selected BGM as the background music for the projected video, a suitable viewing experience can be provided to the user.

[0027] Figure 2 is a flowchart showing the operation of the imaging device 100. In step S101, the imaging unit 101 captures a still image. In step S102, the sound acquisition unit 102 acquires sounds such as ambient voices and music. In step S103, the captured still image and the acquired sound are recorded in association. In step S104, the user selects whether or not to perform automatic video generation. If the user selects not to perform automatic video generation, the process returns to image capture S101. If the user selects to perform automatic video generation, the process proceeds to step S105.

[0028] In step S105, a predetermined number of recommended images (here, n images as an example, with an initial value of 1) are selected, and video generation is performed. In step S106, the audio corresponding to the nth recommended image is determined. In step S107, a specific background music (BGM) is estimated from the audio corresponding to the nth recommended image. In step S108, it is determined whether the specific BGM can be estimated. If the BGM can be estimated, the process proceeds to step S109, and the identified BGM is set as the appropriate BGM for the nth recommended image.

[0029] If the background music (BGM) cannot be estimated, proceed to step S113. In step S113, a specific BGM is estimated from the subject or text in the recommended image. In step S114, it is determined whether a specific BGM can be estimated from the subject or text in the recommended image. If the BGM can be estimated, proceed to step S115 and set the identified BGM as the BGM for the nth recommended image. If the BGM cannot be estimated, proceed to step S116.

[0030] In step S116, the scene of the recommended image is identified. In step S116, the background music (BGM) suitable for the identified scene is set as the BGM for the nth recommended image. In step S110, it is determined whether suitable BGM has been selected for all of the predetermined number of recommended images. If suitable BGM has been selected for all of the n recommended images, the process proceeds to step S111.

[0031] If no suitable background music (BGM) has been selected for all recommended images, the process proceeds to step S118, where the selection of suitable BGM for the next recommended image is initiated. Specifically, 1 is added to n, the process returns to step S106, and the execution of steps S106 and beyond is restarted. In step S111, the number of selected BGMs is counted, and the BGM with the most selections is determined as the suitable BGM for the video. If there are multiple BGMs with the most selections, a BGM is randomly selected. In step S112, the suitable BGM for the video is projected along with the video.

[0032] The above describes the embodiments, but the present invention is not limited to these embodiments, within the scope of the technical concept of the present invention.

[0033] Next, a second embodiment will be described with reference to the drawings. Note that the same reference numerals are used in the drawings for the same configurations, operations, and processes as in the first embodiment, and their descriptions will be omitted.

[0034] Figure 3 is a block diagram illustrating the configuration of the imaging device 100 in the second embodiment. The BGM selection unit 201 assigns weighting information to the selected BGM. When the BGM selection unit 201 selects BGM from audio, it assigns weighting information to the selected BGM according to the volume. For example, under normal circumstances, the weighting information may be 1.5, with 1 representing no weighting.

[0035] Furthermore, when selecting background music (BGM) from a still image, the BGM selection unit 201 assigns weighting information to the selected BGM according to the area in which the subject or text is visible. For example, this weighting is 0.8. The BGM selection unit 201 sends the information to the BGM determination unit 202 along with the BGM suitable for the recommended image, along with the weighting information assigned to that BGM.

[0036] The BGM determination unit 202 determines the appropriate background music for the video based on the weighting information of the background music. For example, suppose there are 5 recommended images and 3 selected background music tracks from the BGM selection unit 201. BGM1 is selected for recommended image 1, with a weighting information of 0.8. BGM2 is selected for recommended image 2, with a weighting information of 1.5. BGM1 is selected for recommended image 3, with a weighting information of 0.7. BGM2 is selected for recommended image 4, with a weighting information of 1.2. BGM3 is selected for recommended image 5, with a weighting information of 1.5.

[0037] In this case, the BGM selected for the recommended image is the same number, with two tracks each for BGM1 and BGM2. However, due to the weighting information, BGM1 has a weighted value of 1.5, while BGM2 has a weighted value of 2.7. In such a case, BGM2, with its higher weighted value, will be selected. The numerical value of the weighting information can be interpreted as a BGM that is more suitable for the recommended image if it is higher, or vice versa. Furthermore, weighting information can also be assigned according to the degree of fit between the estimated BGM and the machine learning database.

[0038] In this way, when projecting captured images as video, selecting and playing background music suitable for the captured images provides users with a more favorable viewing experience. Figure 4 is a flowchart showing the operation of the imaging system in Example 2.

[0039] In step S201, weighting information is assigned to the BGM estimated from the audio corresponding to the recommended image. In step S202, weighting information is assigned to the BGM estimated from the subjects or text in the recommended image. In step S203, weighting information is assigned to the BGM that is appropriate for the scene in the recommended image.

[0040] In step S204, the number of selected background music tracks is counted, and the most frequently selected track is determined as the most suitable background music for the video. If multiple tracks are selected for the most frequently selected track, the best track is determined based on weighting information. [Explanation of Symbols]

[0041] 100 Imaging device

Claims

1. An imaging means; A voice acquisition means; a recording means for recording the still image captured by the imaging means and the sound acquired by the sound acquisition means in a recording medium in association with each other; an image generating means for selecting a plurality of recommended images from the still images recorded by the recording means and automatically generating an image; a BGM selection means for selecting BGM for the recommended image based on the audio corresponding to the recommended image or based on the recommended image; a BGM determining means for determining BGM for the image generated by the image generating means from the BGM selected by the BGM selecting means; a projection means for projecting the image generated by the image generation means; and a BGM output means for outputting the BGM determined by the BGM determination means as the BGM for the image projected by the projection means.

2. 2. The imaging device according to claim 1, wherein the sound acquisition means acquires the sound during a period during which the still image is released by the imaging means, a predetermined time after the release, or a predetermined time before or after the release.

3. 2. The imaging device according to claim 1, wherein the BGM selection means estimates a song from music or a voice included in the sound recorded by the recording means, and selects the BGM from a result of the estimation.

4. 2. The imaging device according to claim 1, wherein the BGM selection means estimates a song from a subject or a character string included in the recommended image, and selects the BGM from a result of the estimation.

5. 2. The imaging device according to claim 1, wherein the BGM selection means identifies a scene of the recommended image and selects the BGM based on the result of the scene identification.

6. 2. The imaging device according to claim 1, wherein the BGM selection means selects the BGM from a server on a network or from an internal storage device.

7. 2. The imaging device according to claim 1, wherein the BGM selection means selects the BGM by associating in advance music suitable for a specific individual, age, and scene.

8. The image generating means generates the image as a slide show that successively displays a plurality of the recommended images, 2. The imaging device according to claim 1, wherein the image generating means determines the number of the recommended images to be selected depending on the length of the automatically generated image.

9. 2. The imaging device according to claim 1, wherein the BGM determining means determines the BGM based on the number of the BGMs selected by the BGM selecting means.

10. The BGM selection means assigns weighting information to the selected BGM, 2. The imaging apparatus according to claim 1, wherein the BGM determining means determines the BGM based on weighting information of the BGM.

Images