Information processing device, information processing method, and program

Abstract

This information processing device comprises a processor and a memory. The processor: causes a moving image reproduction device to reproduce a plurality of pieces of moving image data; generates, on the basis of selection information obtained during reproduction, storage-use sound data from a plurality of pieces of sound data associated with the plurality of pieces of moving image data; and stores, in the memory, the plurality of pieces of moving image data and the storage-use sound data as a file in which the foregoing are combined.

Description

Information processing device, information processing method, and program

[0001] The technology disclosed herein relates to an information processing device, an information processing method, and a program.

[0002] The image processing device disclosed in Japanese Patent Laid-Open No. 2009-171498 includes an image conversion information supply unit, an image conversion unit, an audio conversion information calculation unit, and an audio conversion processing unit. The image conversion information supply unit calculates affine transformation parameters for converting images for each frame based on motion information of the video. The image conversion unit affinely transforms images constituting the video for each frame using the calculated affine transformation parameters, with a reference image as a reference. The audio conversion information calculation unit calculates audio conversion information for converting audio corresponding to the image based on information indicating the center position, angle, or magnification of the converted image from the image conversion unit. The audio conversion processing unit adjusts the volume of each channel constituting the audio based on the audio conversion information, adds the adjusted audio of each channel, and outputs the result to a speaker as output audio.

[0003] The technology disclosed herein provides an information processing device, an information processing method, and a program that enable easy creation of a file that combines multiple pieces of video data and audio data for storage.

[0004] In order to achieve the above-mentioned objective, the information processing device of the present disclosure includes a processor and a memory, and the processor causes a video playback device to play multiple video data, and based on selection information obtained during playback, generates sound data for storage from multiple sound data associated with the multiple video data, and stores the combined multiple video data and sound data for storage in the memory as a file.

[0005] The processor preferably combines the plurality of video data and stores the combined data in the memory in a format different from that of each of the plurality of video data.

[0006] The selection information is information representing a selection history in which one piece of video data is selected alternatively from multiple pieces of video data, and it is preferable that the processor generates sound data for storage by combining multiple pieces of sound data based on the selection history.

[0007] It is preferable that an operation device is provided that allows a plurality of video data to be alternatively selected during playback, and the selection history corresponds to the operation history of the operation device.

[0008] The selection information is information representing a selection history of the ratio of the output volume of multiple sound data, and it is preferable that the processor generates the sound data to be saved by changing the ratio of the output volume of the multiple sound data based on the selection history.

[0009] It is preferable that an operation device be provided that can change the ratio of the output volumes of the plurality of sound data during playback, and the selection history corresponds to the operation history of the operation device.

[0010] It is preferable that an operating device is provided that can change the ratio of the image sizes of the multiple video data during playback, the ratio of the output volume of the multiple sound data corresponds to the ratio of the image sizes of the multiple video data, and the selection history corresponds to the operation history of the operating device.

[0011] The selection information is information representing the audio, environmental sounds, and selection history contained in each of the multiple sound data, and it is preferable that the processor generates the sound data to be saved by combining the audio and environmental sounds based on the selection history.

[0012] It is preferable that an operation device is provided that allows selection of the voice and environmental sound contained in each of the plurality of sound data during playback, and the selection history corresponds to the operation history of the operation device.

[0013] The processor preferably displays, on the video playback device, an object related to the audio and environmental sound included in each of the plurality of sound data.

[0014] The selection information is information representing the selection history of subjects included in multiple video data, and it is preferable that the processor generates the sound data for storage by extracting and combining sounds corresponding to the selected subjects from multiple sound data based on the selection history.

[0015] It is preferable that an operation device is provided that allows a subject included in a plurality of video data to be selected during playback, and the selection history corresponds to the operation history of the operation device.

[0016] The selection information is information representing a history of changes in the size of the subject included in the multiple video data, and it is preferable that the processor generates the sound data to be saved by changing the ratio of the output volume of the multiple sound data based on the history of changes.

[0017] It is preferable that the processor accepts operation information and changes the ratio of image sizes of the plurality of video data included in the file based on the operation information.

[0018] The processor preferably sets the lengths of the multiple video data contained in the file.

[0019] The video playback device preferably includes a display and a speaker.

[0020] The information processing method disclosed herein is a method in which a processor executes processes including causing a video playback device to play multiple video data, generating sound data for storage from multiple sound data associated with the multiple video data based on selection information obtained during playback, and storing the multiple video data and the sound data for storage as a combined file in memory.

[0021] The program disclosed herein causes a processor to perform processes including causing a video playback device to play multiple video data, generating sound data for storage from multiple sound data associated with the multiple video data based on selection information obtained during playback, and storing the multiple video data and the sound data for storage as a combined file in memory.

[0022] 1 is a schematic perspective view showing an example of the front side of a camera. FIG. 2 is a schematic perspective view showing an example of the rear side of a camera. FIG. 3 is a block diagram showing an example of the internal configuration of a camera. FIG. 4 is a block diagram showing an example of the functional configuration of a processor in multiple video playback mode. FIG. 5 is a diagram showing an example of the configuration of a video file read from memory. FIG. 6 is a diagram showing an example of the configuration of a video file saved in memory. FIG. 7 is a diagram showing an example of a process for combining two video data. FIG. 8 is a flowchart showing an example of the processing flow of the processor in multiple video playback mode. FIG. 9 is a diagram showing an example of a selection operation of two video data. FIG. 10 is a diagram showing an example of a process for generating sound data for storage. FIG. 11 is a diagram showing an example of an operation for changing the image size ratio. FIG. 12 is a diagram showing an example of a time change in the ratio of output volume. FIG. 13 is a diagram showing an example of a volume bar display. FIG. 14 is a block diagram showing an example of the functional configuration of a processor in multiple video playback mode according to a third embodiment. FIG. 15 is a diagram showing an example of an audio identification process. FIG. 16 is a flowchart showing an example of the processing flow of the processor in multiple video playback mode according to the third embodiment. FIG. 17 is a diagram showing an example of an object display. FIG. 18 is a diagram showing an example of a process for generating sound data for storage. FIG. 19 is a diagram showing an example of a subject area display. FIG. 19 is a block diagram showing an example of the functional configuration of a processor in multiple video playback mode according to a fourth embodiment. FIG. 19 is a flowchart showing an example of the processing flow of the processor in multiple video playback mode according to the fourth embodiment. FIG. 19 is a diagram showing an example of a subject area display. FIG. 19 is a diagram showing an example of a process for generating sound data for storage. FIG. 10 is a block diagram showing an example of the functional configuration of a processor in a multiple moving image playback mode according to a fifth embodiment. FIG. 11 is a flowchart showing an example of the flow of processing by a processor in a multiple moving image playback mode according to the fifth embodiment. FIG. 12 is a diagram showing an example of a display of a subject area. FIG. 13 is a diagram showing an example of a generation process of sound data for storage. FIG. 14 is a block diagram showing an example of the functional configuration of a processor in a multiple moving image playback mode according to a sixth embodiment. FIG. 15 is a flowchart showing an example of the flow of processing by a processor in a multiple moving image playback mode according to the sixth embodiment. FIG. 16 is a diagram showing an example of a process of changing an image size ratio. FIG. 17 is a diagram showing an example of an editing process. FIG. 18 is a diagram showing an example of an editing process.

[0023] An example of an embodiment of the technology of the present disclosure will be described with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following description, "IC" is an abbreviation for "Integrated Circuit." "CPU" is an abbreviation for "Central Processing Unit." "CMOS" is an abbreviation for "Complementary Metal Oxide Semiconductor." "FPGA" is an abbreviation for "Field-Programmable Gate Array." "PLD" is an abbreviation for "Programmable Logic Device." "ASIC" is an abbreviation for "Application Specific Integrated Circuit."

[0026] First Embodiment The technology of the present disclosure will be described using an interchangeable lens digital camera as an example of a camera. Note that the camera according to the technology of the present disclosure is not limited to an interchangeable lens digital camera, and may also be a digital camera with an integrated lens.

[0027] Fig. 1 shows an example of the front side of a camera 10. As shown in Fig. 1, the camera 10 is a digital camera with an interchangeable lens. The camera 10 is composed of an imaging device 11 and an imaging lens 12 that is interchangeably attached to the imaging device 11. The imaging lens 12 is attached to the front of the imaging device 11. The imaging device 11 is an example of an "information processing device" according to the technology of the present disclosure.

[0028] A dial 13 and a release button 14 are provided on the top surface of the imaging device 11. The dial 13 is operated when setting the operation mode, etc. The operation modes of the camera 10 include, for example, a still image capture mode, a video capture mode, a still image playback mode, a video playback mode, and a multiple video playback mode. The release button 14 is operated by the user when starting to capture a still image or a video.

[0029] Fig. 2 shows an example of the rear side of the camera 10. As shown in Fig. 2, the rear of the imaging device 11 is provided with a display 15, command keys 16, and a finder eyepiece 18 (not shown). The finder can be an optical viewfinder or an electronic viewfinder. The display 15 displays images based on image signals obtained by imaging, various menu screens, and the like. The command keys 16 accept various commands.

[0030] 3 shows an example of the internal configuration of the camera 10. The imaging lens 12 is attached to the front side of the imaging device 11 via a mount 11A. The imaging device 11 and the imaging lens 12 are electrically connected via electrical contacts 11B provided on the mount 11A.

[0031] The imaging lens 12 includes a focus lens 31, an aperture 32, and a lens drive control unit 33. The lens drive control unit 33 is electrically connected to the processor 25 housed in the imaging device 11 via an electrical contact 11B.

[0032] The lens drive control unit 33 drives the focus lens 31 and the diaphragm 32 based on a control signal transmitted from the processor 25. The lens drive control unit 33 controls the drive of the focus lens 31 based on a control signal for focus control transmitted from the processor 25 in order to adjust the position of the focus lens 31.

[0033] The diaphragm 32 has an aperture with a variable diameter. The lens drive control unit 33 controls the drive of the diaphragm 32 based on an diaphragm adjustment control signal sent from the processor 25 in order to adjust the amount of light incident on the image sensor 20.

[0034] The imaging device 11 also includes an imaging sensor 20, an image processing circuit 21, a microphone 22, a speaker 23, a processor 25, and a memory 26. The operations of the imaging sensor 20, the image processing circuit 21, the microphone 22, the speaker 23, the memory 26, and the display 15 are controlled by the processor 25.

[0035] The image sensor 20 is, for example, a CMOS image sensor. Light that has passed through the imaging lens 12 is incident on a light-receiving surface 20A of the image sensor 20. A plurality of pixels that generate image signals by performing photoelectric conversion are formed on the light-receiving surface 20A. The image sensor 20 photoelectrically converts the light incident on each pixel to generate and output image data.

[0036] The image processing circuit 21 performs image processing including white balance correction, gamma correction, etc. on the image data output from the image sensor 20 .

[0037] The microphone 22 collects external sounds and generates sound signals. The microphone 22 may be a built-in microphone built into the imaging device 11, or may be an external microphone that is detachably attached to the imaging device 11. The microphone 22 may be a monaural microphone or a stereo microphone.

[0038] The speaker 23 converts the sound signal into sound and outputs the sound. The speaker 23 may be a monaural speaker or a stereo speaker.

[0039] The processor 25 is configured, for example, by a CPU. The memory 26 is, for example, a non-volatile memory such as a flash memory. The processor 25 executes various processes based on a program 27 stored in the memory 26. The processor 25 may be configured by a collection of multiple IC chips. Also, for example, the memory 26 stores a video file generated as a result of the camera 10 performing an imaging operation in video imaging mode.

[0040] The operation unit 28 includes the above-mentioned dial 13, release button 14, and instruction keys 16. The operation unit 28 also includes a touch panel (not shown) provided on the surface of the display 15.

[0041] 4 shows an example of the functional configuration of the processor 25 in the multiple video playback mode. The multiple video playback mode allows the user to select sounds while simultaneously playing multiple videos, and creates a single video file based on the selected information.

[0042] The processor 25 executes processes in accordance with the program 27 stored in the memory 26 to realize various functional units. The various functional units shown in Fig. 4 are realized in the multiple video playback mode. As shown in Fig. 4, for example, the processor 25 realizes a video playback unit 40, a selection information acquisition unit 41, a storage sound data generation unit 42, and a file creation unit 43.

[0043] Memory 26 stores a plurality of moving image files 50 generated by camera 10 performing an imaging operation in moving image imaging mode. As shown in Fig. 5, each moving image file 50 includes moving image data 51 and sound data 52 associated with moving image data 51. Moving image data 51 is data composed of image data of a plurality of frames. Sound data 52 is data based on sound signals collected by microphone 22 during moving image capture.

[0044] The video playback unit 40 performs processing to play two video files 50 selected by the user from among a plurality of video files 50 stored in the memory 26. The user can select the two video files 50 to play by operating the operation unit 28. The video playback unit 40 displays two moving images based on two pieces of video data 51 included in the two selected video files 50 on the display 15, and outputs sounds based on two pieces of audio data 52 included in the two selected video files 50 from the speaker 23. The display 15 and the speaker 23 are an example of a "video playback device" according to the technology of the present disclosure.

[0045] The selection information acquisition unit 41 acquires selection information selected by the user operating the operation unit 28 while multiple videos are being played back by the video playback unit 40. In this embodiment, the selection information is a selection history in which one of two pieces of sound data 52 is alternatively selected. The selection history corresponds to the operation history of the operation unit 28.

[0046] In this embodiment, one of two sound data 52 is alternatively selected by alternatively selecting one of two moving images displayed on the display 15. That is, in this embodiment, the selection information is information representing history information that has been alternatively selected from two moving image data 51. For example, the user can perform a selection operation by touching a touch panel provided on the surface of the display 15 with a finger while multiple moving images are being played back. The operation unit 28 is an example of an operation device that allows the user to selectively operate multiple moving image data.

[0047] The storage sound data generating unit 42 generates storage sound data 62 from the two pieces of sound data 52 based on the selection information acquired by the selection information acquiring unit 41. Specifically, the storage sound data generating unit 42 generates storage sound data 62 by combining a plurality of pieces of selection data alternatively selected from the two pieces of sound data 52.

[0048] The file creation unit 43 creates one moving image file 60 by combining the two pieces of played moving image data 51 with the storage sound data 62 generated by the storage sound data generation unit 42. For example, as shown in Fig. 6, the file creation unit 43 combines the two pieces of played moving image data 51 into one piece of moving image data 61, and creates a moving image file 60 including the moving image data 61 and the storage sound data 62. The file creation unit 43 stores the created moving image file 60 in the memory 26.

[0049] The moving image file 60 may include data other than the moving image data 61 and the storage sound data 62. For example, the moving image file 60 may include the played moving image data 51 or sound data 52. The moving image file 60 is an example of a "file" according to the technology of the present disclosure.

[0050] Furthermore, the video data 61 has a different format from each of the two video data 51 that have been played back. For example, the format is the aspect ratio of the image. As shown in FIG. 7 , the video data 61 is created by combining two pieces of video data 51, and therefore the aspect ratio of the video data 61 is different from the aspect ratio of each of the two pieces of video data 51. In FIG. 7 , "A" and "B" are written to distinguish between the two pieces of video data 51. Hereinafter, a video based on one piece of video data 51 may be referred to as "video A," and a video based on the other piece of video data 51 may be referred to as "video B."

[0051] 8 shows an example of the processing flow of the processor 25 in the multiple video playback mode. As shown in Fig. 8, first, the video playback unit 40 acquires from the memory 26 two video files 50 selected by the user through the operation unit 28 (step S10).

[0052] Next, the video playback unit 40 plays the two video files 50 in response to a start instruction issued by the user operating the operation unit 28 (step S11). As a result, two pieces of video data 51 contained in the two video files 50 are simultaneously played on the display 15, as shown in Fig. 9. Furthermore, sounds based on two pieces of audio data 52 contained in the two video files 50 are output from the speaker 23. For example, if the speaker 23 is stereo, sounds based on one piece of audio data 52 are output in monaural from the left channel, and sounds based on the other piece of audio data 52 are output in monaural from the right channel.

[0053] 9, while multiple videos are being played back, the user can alternatively select between video A and video B by touching the touch panel on the surface of display 15 with finger F. For example, the user can select from video A and video B the video in which sound is being output from a desired subject.

[0054] The selection information acquisition unit 41 acquires selection information representing a selection history of the video A and the video B during playback of multiple videos (step S12). The hatching in Fig. 10 indicates an example of the selection history of the video A and the video B.

[0055] The video playback unit 40 ends the playback of the two video files 50 in response to an end instruction given by the user operating the operation unit 28 (step S13).

[0056] The storage sound data generator 42 generates storage sound data 62 from the sound data 52 corresponding to the video A and the sound data 52 corresponding to the video B based on the selection information (step S14). Specifically, as shown in Fig. 10, for the period when video A is selected, the selection data SD is extracted from the sound data 52 corresponding to video A, and for the period when video B is selected, the selection data SD is extracted from the sound data 52 corresponding to video B, and the storage sound data 62 is generated by combining the extracted multiple selection data SD.

[0057] The file creation unit 43 creates a moving image file 60 including moving image data 61 generated by combining the two pieces of played moving image data 51 and the storage sound data 62 generated by the storage sound data creation unit 42 (step S15).The file creation unit 43 then stores the created moving image file 60 in the memory 26 (step S16).

[0058] In this embodiment, two moving image files 50 are played in the multiple moving image playback mode, but three or more moving image files 50 may be played. That is, three or more moving image data 51 may be alternatively selected, and the sound data for storage 62 may be generated from three or more sound data 52. Furthermore, a moving image file may be a file that combines three or more moving image data 51 and the sound data for storage 62.

[0059] As described above, in this embodiment, the video playback device plays multiple video data 51, and based on the selection information obtained during playback, generates storage sound data 62 from multiple sound data 52 associated with the multiple video data 51, and stores the combined multiple video data 51 and storage sound data 62 in memory 26 as a video file 60. Therefore, the user can easily create a video file 60 that combines multiple video data 51 and storage sound data 62.

[0060] [Second Embodiment] Next, a second embodiment of the present disclosure will be described. In the above embodiment, in the multiple video playback mode, multiple video data 51 are selected, thereby allowing multiple audio data 52 to be alternatively selected. In this embodiment, however, the ratio of the output volumes of the multiple audio data 52 can be changed.

[0061] This embodiment is similar to the above embodiment except that the functions of the video playback unit 40, selection information acquisition unit 41, and storage sound data generation unit 42 are different.

[0062] In this embodiment, the video playback unit 40 performs processing to play two video files 50 selected by the user from among a plurality of video files 50 stored in the memory 26. While the video playback unit 40 is playing back a plurality of videos, the user can change the ratio of the output volumes of the two pieces of sound data 52 included in the two video files 50.

[0063] In this embodiment, the operation unit 28 is capable of changing the ratio of the image sizes of the two video data 51, and the video playback unit 40 changes the ratio of the output volume of the two sound data 52 output to the speaker 23 according to the ratio of the image sizes.

[0064] In this embodiment, the selection information acquisition unit 41 acquires, as selection information, information representing a selection history of the ratio of the output volumes of the two sound data 52. In this embodiment, the storage sound data generation unit 42 synthesizes the two sound data 52 and generates storage sound data 62 by changing the ratio of the output volumes of the two sound data 52 based on the selection history.

[0065] 11 , when video A and video B are being played on display 15, the user can change the ratio of the image sizes of video A and video B by touching boundary line 53 between video A and video B with finger F and moving boundary line 53. Video playback unit 40 changes the image sizes of video A and video B in response to the user's operation. For example, the user can increase the output volume from a desired subject by increasing the image size of either video A or video B, whichever contains the subject.

[0066] The selection information acquisition unit 41 acquires the history of changes to the image size ratio between the moving image A and the moving image B as selection information on the ratio of the output volumes of the two sound data 52. The output volume of the sound data 52 corresponding to the moving image A is set to V Ａ and the output volume of the sound data 52 corresponding to the moving image B is V Ｂ In this case, the ratio V Ａ / V Ｂ takes a value between 0 and 1. Output volume V Ａ is proportional to the image size of the moving image A, and the output volume V Ｂ is proportional to the image size of the moving image B. For example, as shown in FIG. Ａ / V Ｂ changes over time depending on the change in image size ratio.

[0067] According to this embodiment, the user can intuitively and easily create a moving image file 60 that includes sound data for storage 62 in which the ratio between two output volumes is changed over time.

[0068] In this embodiment, when combining two pieces of played video data 51, the file creation unit 43 may change the image size ratio based on the change history of the image size ratio of the two pieces of video data 51.

[0069] 13, the video playback unit 40 displays the ratio V of the two output volumes on the display 15. Ａ / V Ｂ In addition, the video playback unit 40 may display a volume bar 54 that indicates the ratio V Ａ / V Ｂ may be changed.

[0070] In this embodiment, two moving image files 50 are played in the multiple moving image playback mode, but three or more moving image files 50 may be played. In other words, the volume ratio of three or more pieces of sound data 52 included in three or more moving image files 50 may be changeable.

[0071] Third Embodiment Next, a third embodiment of the present disclosure will be described. In this embodiment, it is possible to select between audio and environmental sound included in each of the plurality of sound data 52.

[0072] 14 shows an example of the functional configuration of the processor 25 in the multiple video playback mode according to the third embodiment. In this embodiment, the processor 25 includes a video playback unit 40, a selection information acquisition unit 41, a storage sound data generation unit 42, and a file creation unit 43, as well as an identification unit 44.

[0073] The identification unit 44 distinguishes between audio and environmental sounds from each of the two pieces of audio data 52 contained in the two video files 50 selected by the user. For example, audio is sound including human voices. Environmental sounds are sounds other than audio. For example, environmental sounds include ambient noise, music, etc.

[0074] 15 , the identification unit 44 performs frequency decomposition on the sound data 52, identifies sound components included in a predetermined wavelength band W as speech, and identifies sound components included in bands other than the wavelength band W as environmental sound. Note that the identification unit 44 may use a machine-learned model to identify speech and environmental sound.

[0075] 16 shows an example of the processing flow of the processor 25 in the multiple video playback mode according to the third embodiment. The flowchart shown in FIG. 16 differs from the flowchart shown in FIG. 8 only in that step S20 is added between step S10 and step S11. In this embodiment, the identification unit 44 identifies audio and environmental sound from each of the two pieces of sound data 52 in step S20. Note that the identification unit 44 may also identify audio and environmental sound while two video files 50 are being played.

[0076] In this embodiment, the video playback unit 40 displays objects related to audio and environmental sound on the display 15, thereby enabling the user to select audio and environmental sound using the operation unit 28. For example, as shown in Fig. 17 , the video playback unit 40 displays an object 55 related to audio and an object 56 related to environmental sound for each of video images A and B. The objects 55 and 56 may be icons with a picture representing a subject or the like.

[0077] In this embodiment, while multiple videos are being played, the user can touch either of the objects 55, 56 with his / her finger to select audio and environmental sound for each of the video images A and B. In addition to alternatively selecting audio and environmental sound for each of the video images A and B, it is also possible to deselect both audio and environmental sound, or to select both audio and environmental sound. For example, if audio is being generated from a desired subject in video A, the user can select audio and deselect environmental sound by touching the object 55 for video A.

[0078] In this embodiment, the selection information acquisition unit 41 acquires, as selection information, information representing the audio and environmental sound included in each of the output volumes of the two sound data 52, and the selection history. In this embodiment, as shown in Fig. 18 , the storage sound data generation unit 42 synthesizes the two sound data 52 and generates storage sound data 62 by combining the audio and environmental sound based on the selection history.

[0079] According to this embodiment, the user can intuitively and easily create a moving image file 60 that includes sound data for storage 62 that combines a desired voice and environmental sounds.

[0080] In this embodiment, two moving image files 50 are played in the multiple moving image playback mode, but three or more moving image files 50 may be played. That is, it may be possible to select the audio and environmental sounds contained in each of three or more pieces of sound data 52 contained in three or more moving image files 50.

[0081] Furthermore, instead of or in addition to the objects 55 and 56, the video playback unit 40 may display on the display 15 an area including a subject corresponding to the sound identified by the identification unit 44. For example, as shown in FIG. 19 , the video playback unit 40 may display on the display 15 a subject area 57 including a human face as the subject. In this case, the user may be able to select between the sound and the environmental sound depending on whether the user selects the subject area 57 or an area other than the subject area 57 using the operation unit 28. For example, when the user touches the subject area 57 with a finger, the sound is selected, and when the user touches an area other than the subject area 57 with a finger, the environmental sound is selected.

[0082] Fourth Embodiment Next, a fourth embodiment of the present disclosure will be described. In this embodiment, audio corresponding to a subject included in a plurality of video data 51 can be selected.

[0083] 20 shows an example of the functional configuration of the processor 25 in the multiple moving image playback mode according to the fourth embodiment. In this embodiment, the processor 25 includes a moving image playback unit 40, a selection information acquisition unit 41, a storage sound data generation unit 42, and a file creation unit 43, as well as a subject detection unit 45 and an audio extraction unit 46.

[0084] The subject detection unit 45 detects a subject from each of the two pieces of video data 51 contained in the two video files 50 selected by the user. For example, the subject is a human face.

[0085] The audio extraction unit 46 extracts audio corresponding to the subject detected by the subject detection unit 45 from each of the two pieces of audio data 52 contained in the two video files 50 selected by the user. In order to identify the relationship between the subject and the audio, it is preferable that the microphone 22 is a directional microphone and that the audio data 52 contains directional information. The audio extraction unit 46 can identify the audio emitted by the subject from the relationship between the position of the subject and the position where the audio is generated.

[0086] FIG. 21 shows an example of the processing flow of the processor 25 in the multiple moving image playback mode according to the fourth embodiment. The flowchart shown in FIG. 21 differs from the flowchart shown in FIG. 8 only in that steps S30 and S31 are added between steps S10 and S11. In this embodiment, the subject detection unit 45 detects a subject from each of the two moving image data 51 in step S30. The audio extraction unit 46 extracts audio corresponding to the subject from each of the two sound data 52 in step S31. Note that the subject detection unit 45 and the audio extraction unit 46 may detect the subject and extract the audio during moving image capture. That is, the moving image file 50 may include subject detection information and audio extraction information.

[0087] In this embodiment, the video playback unit 40 displays an area including a detected subject on the display 15, thereby allowing the user to select the subject using the operation unit 28. For example, as shown in Fig. 22, the video playback unit 40 displays subject areas including the detected subjects from each of video images A and B. Subject areas SA1 and SA2 are displayed in video image A. Subject areas SB1 and SB2 are displayed in video image B.

[0088] In this embodiment, while multiple videos are being played back, the user can select a subject by touching one or more of subject areas SA1, SA2, SB1, and SB2 with a finger. Note that it is possible to select one subject from each of video A and video B, as well as multiple subjects from each of video A and video B.

[0089] Furthermore, the video playback unit 40 may output only the sound corresponding to the subject selected by the user from the speaker 23 while playing back multiple videos.

[0090] In this embodiment, the selection information acquisition unit 41 acquires, as selection information, information representing the selection history of the subjects included in the two moving image data 51. In this embodiment, as shown in Fig. 23 , the storage sound data generation unit 42 synthesizes the two sound data 52 and generates storage sound data 62 by extracting and combining sounds corresponding to the subjects selected based on the selection history from the two sound data 52.

[0091] According to this embodiment, the user can intuitively and easily create a moving image file 60 that includes sound data for storage 62 that combines sounds corresponding to a desired subject.

[0092] In this embodiment, two moving image files 50 are played in the multiple moving image playback mode, but three or more moving image files 50 may be played. That is, it may be possible to select a subject included in each of three or more pieces of moving image data 51 included in three or more moving image files 50.

[0093] Fifth Embodiment Next, a fifth embodiment of the present disclosure will be described. In the above-described embodiments, information representing an operation history of a user operating the operation unit 28 is acquired as selection information. In this embodiment, information representing a change history of the size of a subject included in a plurality of video data 51 is acquired as selection information.

[0094] 24 shows an example of the functional configuration of the processor 25 in the multiple moving image playback mode according to the fifth embodiment. In this embodiment, the processor 25 includes a moving image playback unit 40, a selection information acquisition unit 41, a sound data for storage generation unit 42, and a file creation unit 43, as well as a subject detection unit 45.

[0095] As in the fourth embodiment, the subject detection unit 45 detects a subject from each of the two pieces of video data 51 included in the two video files 50 selected by the user. For example, the subject is a human face.

[0096] In this embodiment, the selection information acquisition unit 41 acquires, as selection information, information representing a change history of the size of the subject detected by the subject detection unit 45. Specifically, the selection information acquisition unit 41 selects the video data 51 that includes a larger subject (for example, a larger area) from two video data 51. This is the same as the first embodiment, except that the selection information is acquired based on the size of the subject instead of the operation of the operation unit 28.

[0097] Figure 25 shows an example of the processing flow of the processor 25 in multiple video playback mode according to the fifth embodiment. The flowchart shown in Figure 25 differs from the flowchart shown in Figure 8 only in that steps S40 and S41 are added between steps S10 and S13 instead of step S12. In this embodiment, the subject detection unit 45 detects a subject from each of the two video data 51 during multiple video playback in step S40. The selection information acquisition unit 41 records the size of the detected subject.

[0098] In this embodiment, the video playback unit 40 displays an area including the detected subject on the display 15. For example, as shown in Fig. 26, the video playback unit 40 displays subject areas including the detected subject from each of video images A and B. Subject area SA is displayed in video image A. Subject area SB is displayed in video image B. The subject corresponding to the larger of subject area SA and subject area SB is selected.

[0099] The video playback unit 40 may output from the speaker 23 only the sound corresponding to the subject selected based on the size while playing back multiple videos.

[0100] In this embodiment, the selection information acquisition unit 41 acquires selection information representing a change history of the size of the subject included in the two moving image data 51. In this embodiment, as shown in Fig. 27 , the storage sound data generation unit 42 synthesizes the two sound data 52 and generates storage sound data 62 by combining multiple pieces of selection data alternatively selected from the two sound data 52 based on the change history.

[0101] According to this embodiment, it is possible to easily create a moving image file 60 that includes sound data for storage 62 that is a combination of sound data 52 selected according to the size of the subject.

[0102] The storage sound data generator 42 may generate the storage sound data 62 by changing the ratio of the output volumes of the two sound data 52 based on the history of changes in the size of the subject included in the two moving image data 51. For example, the ratio of the output volumes of the two sound data 52 is changed according to the size ratio of the two subjects. Specifically, the output volume of the sound data 52 with the larger subject size is increased.

[0103] In this embodiment, two moving image files 50 are played in the multiple moving image playback mode, but three or more moving image files 50 may be played. That is, information representing a change history of the size of a subject included in three or more moving image data 51 included in three or more moving image files 50 may be acquired as selection information.

[0104] Sixth Embodiment Next, a sixth embodiment of the present disclosure will be described. In this embodiment, after a moving image file 60 including storage sound data 62 is saved in the memory 26, the user can perform editing operations on the moving image file 60.

[0105] 28 shows an example of the functional configuration of the processor 25 in the multiple video playback mode according to the sixth embodiment. In this embodiment, the processor 25 includes an editing processing unit 47 in addition to a video playback unit 40, a selection information acquisition unit 41, a storage sound data generation unit 42, and a file creation unit 43.

[0106] The editing processing unit 47 edits the video data 61 included in the video file 60 based on a selection operation by the user on the operation unit 28. As described in the second embodiment, the video data 61 is created by combining two pieces of video data 51 whose image size ratios have been changed. In this embodiment, the editing processing unit 47 changes the image size ratio of the two pieces of video data 51 based on operation information received from the operation unit 28.

[0107] 29 shows an example of the processing flow of the processor 25 in the multiple video playback mode according to the sixth embodiment. The flowchart shown in Fig. 29 differs from the flowchart shown in Fig. 8 only in that step S50 is added after step S16. In this embodiment, the editing processing unit 47 receives operation information from the operation unit 28 in step S50, and changes the image size ratio of the two video data 51 that make up the video data 61 in the video file 60 based on the operation information.

[0108] For example, as shown in Fig. 30, video data 61 in which the ratio of video image A to video image B is 8:2 can be changed to video data 61 in which the ratio of video image A to video image B is 5:5. Furthermore, the image size ratio is not limited to a fixed value, and can be freely changed depending on the playback time of the video. For example, the image size ratio can be changed to "8:2" for the period from 0 to 10 seconds, and "5:5" for the period from 10 to 20 seconds.

[0109] Even if the image size ratio is fixed as in the first embodiment, the image size ratio can be changed after the moving image file 60 is saved.

[0110] In this embodiment, two moving image files 50 are played in the multiple moving image playback mode, but three or more moving image files 50 may be played. In other words, the image size ratio of the three or more moving image data 51 that make up the moving image data 61 may be changeable.

[0111] As a modification of this embodiment, the editing processing unit 47 may set the lengths of the multiple video data 51 that make up the video data 61. This is because the multiple video data 51 may have different lengths.

[0112] 31 to 33 show cases where video B finishes playing before video A. For example, as shown in Fig. 31, video B may be blacked out for a period of time after video B finishes playing until video A finishes playing. Alternatively, as shown in Fig. 32, after video B finishes playing, the display layout may be changed so that video A is displayed in the center of the display 15. Alternatively, as shown in Fig. 33, video A may be blacked out after video B finishes playing.

[0113] Furthermore, the editing processing unit 47 may change the playback speed so that the video A and the video B end simultaneously. For example, if the playback of video B ends earlier than the playback of video A, the playback speed of video B is slowed down so that the videos A and B end simultaneously.

[0114] The above-described embodiments can be combined as long as no contradiction occurs. The technology of the present disclosure is not limited to digital cameras, and can also be applied to information processing devices such as smartphones.

[0115] In each of the above embodiments, the hardware structure of the control unit, with the processor 25 being an example, can use the various processors listed below. The various processors listed above include a CPU, which is a general-purpose processor that functions by executing software (programs), as well as a processor such as an FPGA, whose circuit configuration can be changed after manufacture. FPGAs include dedicated electrical circuits, such as PLDs or ASICs, which are processors with a circuit configuration designed specifically to execute specific processes.

[0116] The control unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple control units may be configured with a single processor.

[0117] There are several possible examples of configuring multiple control units with a single processor. A first example is a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple control units. A second example is a form in which a processor is used to realize the functions of an entire system including multiple control units with a single IC chip, as typified by a system on chip (SOC). In this way, the control unit can be configured as a hardware structure using one or more of the various processors described above.

[0118] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0119] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0120] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0121] The above description allows the understanding of the following technologies. [Supplementary Item 1] An information processing device comprising a processor and a memory, wherein the processor causes a video playback device to play a plurality of video data, generates sound data for storage from a plurality of sound data associated with the plurality of video data based on selection information obtained during playback, and stores the plurality of video data and the sound data for storage as a combined file in the memory. [Supplementary Item 2] The information processing device according to Supplementary Item 1, wherein the processor combines the plurality of video data and stores the combined file in the memory in a format different from that of each of the plurality of video data. [Supplementary Item 3] The information processing device according to Supplementary Item 1 or Supplementary Item 2, wherein the selection information is information representing a selection history of alternative selections from the plurality of video data, and the processor generates the sound data for storage by combining the plurality of sound data based on the selection history. [Supplementary Item 4] The information processing device according to Supplementary Item 3, comprising an operation device that allows the plurality of video data to be alternatively selected during playback, and the selection history corresponds to an operation history of the operation device. [Supplementary Item 5] The information processing device according to Supplementary Item 1 or Supplementary Item 2, wherein the selection information is information representing a selection history of ratios of output volumes of the plurality of sound data, and the processor generates the sound data to be saved by changing the ratios of output volumes of the plurality of sound data based on the selection history. [Supplementary Item 6] The information processing device according to Supplementary Item 5, further comprising an operation device that enables the ratios of output volumes of the plurality of sound data to be changed during playback, and the selection history corresponds to an operation history of the operation device. [Supplementary Item 7] The information processing device according to Supplementary Item 5, further comprising an operation device that enables the ratios of image sizes of the plurality of video data to be changed during playback, and the ratios of output volumes of the plurality of sound data to be changed correspond to the ratios of image sizes of the plurality of video data, and the selection history corresponds to an operation history of the operation device.[Supplementary Item 8] The information processing device according to Supplementary Item 1 or Supplementary Item 2, wherein the selection information is information representing audio and environmental sound included in each of the plurality of sound data and a selection history, and the processor generates the sound data to be saved by combining the audio and the environmental sound based on the selection history. [Supplementary Item 9] The information processing device according to Supplementary Item 8, further comprising an operation device that allows the audio and the environmental sound included in each of the plurality of sound data to be selected during playback, and the selection history corresponds to an operation history of the operation device. [Supplementary Item 10] The information processing device according to Supplementary Item 9, wherein the processor displays objects related to the audio and the environmental sound included in each of the plurality of sound data on the video playback device. [Supplementary Item 11] The information processing device according to Supplementary Item 1 or Supplementary Item 2, wherein the selection information is information representing a selection history of subjects included in the plurality of video data, and the processor generates the sound data to be saved by extracting audio corresponding to the selected subjects from the plurality of sound data and combining it based on the selection history. [Supplementary Item 12] The information processing device according to Supplementary Item 11, comprising an operation device that allows selection of a subject included in the plurality of video data during playback, and the selection history corresponds to an operation history of the operation device. [Supplementary Item 13] The information processing device according to Supplementary Item 12, wherein the selection information is information representing a history of changes in size of the subject included in the plurality of video data, and the processor generates the sound data for storage by changing a ratio of output volumes of the plurality of sound data based on the change history. [Supplementary Item 14] The information processing device according to any one of Supplementary Items 1 to 13, wherein the processor receives operation information, and changes a ratio of image sizes of the plurality of video data included in the file based on the operation information. [Supplementary Item 15] The information processing device according to any one of Supplementary Items 1 to 14, wherein the processor sets lengths of the plurality of video data included in the file.

Claims

1. An information processing device comprising a processor and a memory, wherein the processor causes a video playback device to play multiple video data, generates sound data for storage from multiple sound data associated with the multiple video data based on selection information obtained during playback, and stores the multiple video data and the sound data for storage as a combined file in the memory.

2. The information processing device according to claim 1, wherein the processor combines the plurality of video data and stores the combined data in the memory in a format different from that of each of the plurality of video data.

3. The information processing device according to claim 1, wherein the selection information is information representing a selection history of alternatively selected video data from the plurality of video data, and the processor generates the sound data for storage by combining the plurality of sound data based on the selection history.

4. The information processing device according to claim 3, further comprising an operation device that allows the plurality of video data to be alternatively selected during playback, and the selection history corresponds to the operation history of the operation device.

5. The information processing device according to claim 1, wherein the selection information is information representing a selection history of the ratio of the output volume of the plurality of sound data, and the processor generates the sound data to be saved by changing the ratio of the output volume of the plurality of sound data based on the selection history.

6. The information processing device according to claim 5, further comprising an operation device that enables a ratio of output volumes of the plurality of sound data to be changed during playback, and the selection history corresponds to an operation history of the operation device.

7. An information processing device according to claim 5, further comprising an operating device that enables the ratio of image sizes of the plurality of video data to be changed during playback, wherein the ratio of output volumes of the plurality of sound data corresponds to the ratio of image sizes of the plurality of video data, and the selection history corresponds to the operation history of the operating device.

8. The information processing device described in claim 1, wherein the selection information is information representing the audio, environmental sound, and selection history contained in each of the plurality of sound data, and the processor generates the sound data to be saved by combining the audio and the environmental sound based on the selection history.

9. An information processing device according to claim 8, further comprising an operation device that enables selection of the audio and the environmental sound contained in each of the plurality of sound data during playback, and the selection history corresponds to the operation history of the operation device.

10. The information processing device according to claim 9, wherein the processor displays an object related to the audio and the environmental sound included in each of the plurality of sound data on the video playback device.

11. The information processing device of claim 1, wherein the selection information is information representing a selection history of subjects included in the plurality of video data, and the processor generates the sound data for storage by extracting and combining sounds corresponding to the selected subjects from the plurality of sound data based on the selection history.

12. The information processing device according to claim 11, further comprising an operation device that enables selection of the subject included in the plurality of video data during playback, and the selection history corresponds to the operation history of the operation device.

13. The information processing device described in claim 12, wherein the selection information is information representing a history of changes in the size of the subject included in the plurality of video data, and the processor generates the sound data for storage by changing the ratio of the output volume of the plurality of sound data based on the history of changes.

14. The information processing device according to claim 1, wherein the processor receives operation information and changes the image size ratio of the plurality of video data included in the file based on the operation information.

15. The information processing device according to claim 1, wherein the processor sets the lengths of the plurality of video data included in the file.

16. The information processing device according to claim 1, wherein the video playback device includes a display and a speaker.

17. An information processing method in which a processor executes processes including: causing a video playback device to play multiple video data; generating sound data for storage from multiple sound data associated with the multiple video data based on selection information obtained during playback; and storing the multiple video data and the sound data for storage as a combined file in memory.

18. A program that causes a processor to execute processes including: causing a video playback device to play multiple video data; generating sound data for storage from multiple sound data associated with the multiple video data based on selection information obtained during playback; and storing the multiple video data and the sound data for storage as a combined file in memory.

Images