Information processing device, video compositing method, and video compositing program

Abstract

To provide an information processing device, a moving image synthesis method, and a moving image synthesis program capable of easily creating a desired moving image for a user when the user records a moving image using a camera.SOLUTION: A portable terminal recording an image captured by a camera as a moving image determines a synthesis start position at which to start synthesizing a previously recorded first singing moving image and a second singing moving image to be recorded later, then records the second singing moving image by the camera, and finally synthesizes the first singing moving image and the second singing moving image at the synthesis start position. Therefore, when a singing user records a moving image by a camera, the portable terminal can easily create a desired moving image for the user.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to an information processing device, a moving image compositing method, and a moving image compositing program. [Background technology]

[0002] 2. Description of the Related Art In recent years, online services such as websites (web pages, web services), online games, and application software (hereinafter referred to as "apps") that are distributed to information processing devices such as mobile terminals via computer networks have become widespread.

[0003] One of the online services is a video posting site where users can post videos that they have taken themselves and other users can view the videos. Users also post performance videos, such as singing videos, on this video posting site. For this reason, users may edit the singing videos they post in order to post better singing videos.

[0004] Here, Patent Document 1 discloses a karaoke recording system that can handle the interruption of a performance, and aims to obtain the final recorded singing voice data without having to sing the karaoke song again from the beginning, even if the performance of the karaoke song is interrupted.

[0005] This karaoke recording system that can handle performance interruptions includes a re-play instruction means for instructing re-play of a karaoke song whose performance has been interrupted, a recording information recording means for, when performance of any karaoke song whose singing voice has been recorded by the recording means, linking and recording at least the music of the karaoke song, the recorded performance range data, and the mid-recording data, and, when re-play of the karaoke song is instructed, controlling so that the singing voice is not recorded for the recorded performance range based on the recorded performance range data. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2010-237389 A Summary of the Invention [Problem to be solved by the invention]

[0007] In the system disclosed in Patent Document 1, recording is performed from the point where recording was stopped midway, and the user cannot decide the timing of the karaoke song to perform the additional recording. Therefore, even if the user makes a mistake in singing, he or she cannot re-record the failed part. Furthermore, the karaoke recording system that supports performance interruption disclosed in Patent Document 1 records singing voices but does not shoot performance videos such as singing videos.

[0008] The present invention has been made in consideration of the above circumstances, and aims to provide an information processing device, a video compositing method, and a video compositing program that enable a user to easily create a video that the user finds desirable when recording a video using a camera. [Means for solving the problem]

[0009] In order to solve the above problems, an information processing device, a moving image compositing method, and a moving image compositing program according to the present invention employ the following means.

[0010] In order to solve the above problems, an "information processing device" which is one embodiment of the present invention is an information processing device which records images captured by a camera as videos, and which includes a determination means which determines a synthesis start position at which to start synthesizing a second video to be recorded later with a first video recorded earlier, a recording control means which records the second video with the camera, and a video synthesis means which synthesizes the first video and the second video at the synthesis start position of the first video.

[0011] In order to solve the above problems, a "video compositing method" according to one embodiment of the present invention includes a first step of recording a first video with a camera, a second step of determining a compositing start position for the first video at which to start compositing a second video to be recorded later, a third step of recording the second video with the camera, and a fourth step of compositing the first video and the second video at the compositing start position of the first video.

[0012] In order to solve the above problem, a "video compositing program" which is one aspect of the present invention causes a computer provided in an information processing device which records images captured by a camera as videos to function as: a determination means for determining a compositing start position at which to start compositing a second video to be recorded later with a first video recorded earlier; a recording control means for recording the second video with the camera; and a video compositing means for compositing the first video and the second video at the compositing start position of the first video.

[0013] The above-mentioned "information processing device" may be subject to various technical limitations, as exemplified below. In addition, technical limitations of the same purport may be applied to the processing steps executed by the "motion picture compositing method" or the functions of the "motion picture compositing program."

[0014] The video camera further comprises an image display control means for displaying, on a screen at a combination start position when recording of the second moving image starts, a combination start position image, which is an image of the first moving image at the combination start position.

[0015] The image display control means displays, on the screen, the synthesis start position image superimposed on an image captured by the camera to record the second moving image.

[0016] The moving image synthesizing means performs image processing to make a synthesized portion of the first moving image and the second moving image less noticeable.

[0017] The moving image synthesizing means superimposes the first moving image and the second moving image for a predetermined period from the synthesis start position, thereby synthesizing the first moving image and the second moving image.

[0018] The first video and the second video are videos that are recorded while a piece of music is being played back.

[0019] The recording control means starts playing the music before the combination start position, and starts recording the second moving image when the played music reaches a timing corresponding to the combination start position.

[0020] The recording control means starts playing the music before recording of the second moving image starts, and also performs a countdown to the start of recording of the second moving image.

[0021] The position that can be set as the synthesis start position is a predetermined position in the music piece. Effect of the Invention

[0022] According to the present invention, when a user uses a camera to record a video, it is possible to easily create a video that the user finds desirable. [Brief description of the drawings]

[0023] [Figure 1] 1 is a configuration diagram of a karaoke system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing an electrical configuration of a server according to an embodiment of the present invention. [Diagram 3] 2 is a block diagram showing an electrical configuration of a mobile terminal according to an embodiment of the present invention; FIG. [Figure 4] FIG. 11 is a diagram showing a screen display of a mobile terminal when shooting a singing video according to an embodiment of the present invention. [Diagram 5] FIG. 11 is a diagram showing a screen display of a mobile terminal when viewing a singing video according to an embodiment of the present invention. [Figure 6] FIG. 2 is a schematic diagram showing a video composition function according to an embodiment of the present invention. [Figure 7] 13A and 13B are diagrams showing screen displays of a mobile terminal when determining a synthesis start position according to an embodiment of the present invention. [Figure 8]FIG. 13 is a diagram showing a synthesis start position image according to the embodiment of the present invention. [Figure 9] FIG. 13 is a diagram showing a screen display of the mobile terminal when starting to shoot a second singing video according to an embodiment of the present invention. [Figure 10] FIG. 2 is a schematic diagram showing the display timing of a synthesis start position image and the superimposition synthesis of a first singing video and a second singing video according to the embodiment of the present invention. [Figure 11] FIG. 13 is a diagram illustrating a composition effect according to an embodiment of the present invention. [Figure 12] FIG. 2 is a functional block diagram relating to a moving image synthesis function according to an embodiment of the present invention. [Figure 13] 1 is a flowchart showing a flow of a moving image synthesis process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An information processing device, a moving image compositing method, and a moving image compositing program according to an embodiment of the present invention will be described below with reference to the drawings.

[0025] In this embodiment, a performer records his / her own performance as a video using a mobile terminal and transmits it to a server, thereby uploading it to a video sharing site. The video uploaded to the video sharing site (posted video) can be viewed via a mobile terminal. Note that in this embodiment, a performance is referred to as singing, a performer is referred to as a singing user, and a video uploaded to the video sharing site is referred to as a singing video. Also, a user who views a singing video uploaded to the video sharing site is referred to as a viewing user.

[0026] [1. Karaoke system configuration] 1 is a schematic diagram of a karaoke system 1 according to this embodiment. The karaoke system 1 includes a communication line 2, a plurality of mobile terminals 3 (mobile terminals 3A and 3B), and a server 4.

[0027] The communication line 2 forms a computer network, and is, for example, a wide area communication line provided by an electric utility company.

[0028] The mobile terminal 3 is, for example, an information processing terminal such as a smartphone, a tablet terminal, or a notebook computer, and is used by a user to utilize online services. The mobile terminal 3 includes a touch panel display 3a for displaying images, a speaker 3b for outputting sound, a microphone 3c for inputting sound, a camera 3d for photographing a subject, and an earphone terminal 3e for connecting earphones (not shown). Note that photographing here refers to a state in which the camera 3d is functioning and the subject is displayed on the touch panel display 3a regardless of whether recording is performed or not. The touch panel display 3a includes, for example, an LCD (Liquid Crystal Display) and a touch sensor. The LCD displays various images, and the touch sensor accepts various input operations performed using a pointing body such as a finger, a stylus, or a pen. In the following description, the touch panel display 3a is also referred to as a screen 3a.

[0029] The server 4 is an information processing device that provides online services to the mobile terminal 3 via the communication line 2. In the example of FIG. 1, a singing user uploads his / her singing video (singing video data) to a video posting site by transmitting the singing video from the mobile terminal 3A to the server 4. Then, a viewing user accesses the video posting site using the mobile terminal 3B and views the singing video. The singing user can also view the singing video that he / she uploaded by accessing the video posting site using the mobile terminal 3A. Also, the user of the mobile terminal 3B can become a singing user and upload a singing video to the video posting site.

[0030] [2. Server configuration] FIG. 2 is a block diagram showing the electrical configuration of the server 4 according to the present embodiment.

[0031] The server 4 according to this embodiment includes a CPU (Central Processing Unit) 20 which is a main control unit that controls the operation of the entire server 4, a ROM in which various programs and various data are stored in advance, The computer includes a Read Only Memory (RAM) 22, a Random Access Memory (RAM) 24 used as a work area when the CPU 20 executes various programs, and a Hard Disk Drive (HDD) 26 as storage means for storing various programs and various data.

[0032] The HDD 26 stores singing video data transmitted from the mobile terminal 3A, i.e., singing video data uploaded to a video posting site, song data indicating songs that the singing user can sing, etc. The storage means is not limited to the HDD 26, and may be, for example, another storage medium such as a semiconductor memory such as a flash memory.

[0033] Furthermore, the server 4 is equipped with an operation input unit 28 composed of a keyboard, a mouse, etc. and which accepts input of various operations, a monitor 30 such as a liquid crystal display device for displaying various images, and an external interface 32 which is connected to other information processing devices such as a mobile terminal 3 via the communication line 2 and which transmits and receives various data to and from the other information processing devices.

[0034] The CPU 20, ROM 22, RAM 24, HDD 26, operation input unit 28, monitor 30, and external interface 32 are electrically connected to one another via a system bus 34. Therefore, the CPU 20 can access the ROM 22, RAM 24, and HDD 26, grasp the operation state of the operation input unit 28, display images on the monitor 30, and transmit and receive various data to and from other information processing devices, etc. via the external interface 32.

[0035] [3. Electrical configuration of mobile terminal] FIG. 3 is a functional block diagram showing the electrical configuration of the mobile terminal 3. As shown in FIG.

[0036] The mobile terminal 3 includes a main control unit 40, a main memory unit 42, an auxiliary memory unit 44, a communication unit 46, and an operation button 48 in addition to the components shown in FIG.

[0037] The main control unit 40 is, for example, a CPU, a microprocessor, a DSP (Digital Signal Processor), or the like, and controls the overall operation of the mobile terminal 3.

[0038] The main storage unit 42 is configured with, for example, a RAM or a DRAM (Dynamic Random Access Memory), and is used as a work area when the main control unit 40 executes processes based on various programs.

[0039] The auxiliary storage unit 44 is, for example, a non-volatile memory such as a flash memory, and stores various data such as images, as well as programs used in the processing of the main control unit 40. The programs stored in the auxiliary storage unit 44 are, for example, an OS (Operating System) for implementing the basic functions of the mobile terminal 3, drivers for controlling various hardware, and programs for implementing e-mail, web browsing, and various other functions. The auxiliary storage unit 44 also stores applications for shooting and posting singing videos and for viewing video posting sites. (hereinafter referred to as the "video posting and viewing app") is pre-stored.

[0040] The communication unit 46 is, for example, a network interface controller (NIC), and has a function of connecting to the communication line 2. Note that the communication unit 46 may have, instead of or together with the NIC, a function of connecting to a wireless local area network (LAN), a function of connecting to a wireless wide area network (WAN), and a function of enabling short-distance wireless communication such as Bluetooth (registered trademark), infrared communication, and the like.

[0041] The operation buttons 48 are provided on the side surface of the mobile terminal 3, and include a power button for starting and stopping the mobile terminal 3, a volume adjustment button for the sound output by the speaker 3b, and the like.

[0042] The main control unit 40, the main memory unit 42, the auxiliary memory unit 44, the communication unit 46, the operation buttons 48, the touch panel display 3a, the speaker 3b, the microphone 3c, the camera 3d, and the earphone terminal 3e are electrically connected to one another via a system bus 49. Therefore, the main control unit 40 can access the main memory unit 42 and the auxiliary memory unit 44, display images on the touch panel display 3a, grasp the operation state of the touch panel display 3a and the operation buttons 48 by the user, input sound to the microphone 3c, output sound from the speaker 3b or an earphone connected to the earphone terminal 3e, control the camera 3d, and access various communication networks and other information processing devices via the communication unit 46.

[0043] [4. Filming of singing videos by singing users] A case where a singing user shoots a singing video using the mobile terminal 3A will be described.

[0044] When a singing user shoots a singing video, the singing user launches a video posting and viewing application on the mobile terminal 3A. When the video posting and viewing application is launched, the mobile terminal 3A accesses the server 4 that stores a plurality of pieces of music data. The singing user then arbitrarily selects a piece of music to sing from the video posting and viewing application, and downloads the music data from the server 4 to the mobile terminal 3A. The singing user then plays the music at a timing of his or her choice using the video posting and viewing application, and sings. The video posting and viewing application starts playing the music, and starts shooting the video with the camera 3d. That is, the singing video is a video shot by the mobile terminal 3A while the music is being played from the mobile terminal 3A.

[0045] Note that song data is also associated with lyric data, and when song data is downloaded from the server 4 to the mobile terminal 3A, the associated lyric data is also downloaded to the mobile terminal 3A. Note that in the following description, song data is considered to include lyric data.

[0046] FIG. 4 is an example of a display state (hereinafter referred to as "screen display") on the screen 3a of the mobile terminal 3A when a singing video is being filmed.

[0047] 4, the screen 3a is divided into a lyrics display area 50A and a captured image display area 50B. The lyrics display area 50A includes a lyrics image 52 showing the lyrics of the song sung by the user, a pitch image 54 showing the pitch of the song, and a progress bar 56 showing the progress of the shooting.

[0048] The lyric image 52 and the pitch image 54 are updated according to the progress of the music. In the present embodiment, as an example, the lyric image 52 and the pitch image 54 are updated every few phrases and displayed in the lyric display area 50A. Note that the update timing of the lyric image 52 and the pitch image 54 may be the same or different.

[0049] As an example, the lyrics image 52 displays lyrics in multiple rows (two rows in the example of FIG. 4), and the color of the lyrics in the upper row changes from the left end to the right end in accordance with the progress of the music so that the singing user can understand the lyrics that should currently be sung. When the color change of the lyrics in the upper row reaches the right end, the lyrics in the lower row rises and is displayed in the upper row, and new lyrics are displayed in the lower row, and the color of the lyrics in the upper row changes again from the left end to the right end in accordance with the progress of the music.

[0050] As an example, the pitch image 54 is displayed with multiple pitch bars 54A in a staircase pattern in the left-right direction according to the strength of the pitch. Then, in order to allow the singing user to grasp the pitch of the lyrics that should be sung at present, the color of the pitch bars 54A changes from the left end to the right end as the song progresses, and the pointer 54B moves from the left end to the right end. When the color of the pitch bars 54A changes and the pointer 54B reaches the right end, the pitch image 54 indicating the next pitch is updated and displayed.

[0051] As an example, the length of the progress bar 56 from the left end to the right end indicates the length of the entire song. When playback of the song starts, a pointer 56A indicating the playback position of the song moves from the left end to the right end, and when the pointer 56A reaches the right end, the song ends. Note that the progress bar 56 that the pointer 56A has passed through is displayed thicker than the position before the progress.

[0052] Recording of a singing video starts a predetermined time (e.g., 10 seconds) after the singing user selects a song and clicks a recording start button (not shown) displayed on the screen 3a. The start and end of video recording may coincide with the start and end of the song, but this is not limited thereto. Video recording may start a predetermined time (e.g., 5 seconds) before the start of the song, or end a predetermined time (e.g., 5 seconds) after the end of the song.

[0053] The singing user connects earphones to the earphone terminal 3e, listens to the music being played using the earphones, and sings along with the music. The mobile terminal 3A photographs the singing user with the camera 3d and records the singing user's singing with the microphone 3c. In other words, the microphone 3c does not capture the sound of the music being played. The mobile terminal 3A then records the singing user's singing voice captured by the microphone 3c, and creates singing data.

[0054] The singing data may be obtained by extracting the frequency band of the human voice through a filtering process, which removes noises caused by the surrounding environment of the singing user from the singing data, thereby making the recorded singing voice of the singing user clearer.

[0055] Then, the video posting and viewing application combines the music data and singing data with the recording data to generate singing video data that can be transmitted to the server 4. Note that, as an example, the user can select one of the following two types of timing for transmitting the singing video data to the server 4, i.e., the timing for uploading the singing video data to the video posting site.

[0056] One is live distribution, in which a singing user uploads singing video data to a video sharing site in real time while singing. In live distribution, a viewing user watches the singing user sing in real time. The other is non-live distribution, in which a singing user uploads singing video data to a video sharing site at a timing of their choice after singing of a song is completed.

[0057] When performing live distribution, the singing user sets up the live distribution before recording the singing video, and uploads the singing video data to the video posting site when the video recording starts. In the case of live distribution, the singing video data may be uploaded to the video posting site without being stored in the mobile terminal 3A.

[0058] In addition, as settings for live streaming, a singing user can set either a first live streaming setting that allows viewing users to view the singing video in live streaming, or a second live streaming setting that allows viewing users to view the singing video even after the live streaming. That is, in the first live streaming setting, when the live streaming ends, the singing video data is deleted from the server 4, and the viewing users cannot view the live-streamed singing video after the live streaming ends. On the other hand, in the second live streaming setting, the server 4 continues to store the singing video data even after the live streaming ends, so that the viewing users can view the singing video as non-live streaming even after the live streaming ends.

[0059] In addition, when performing non-live distribution, the singing video data is temporarily stored in the mobile terminal 3A, and the singing user operates the video posting and viewing application to upload the singing video to the video posting site at any time.

[0060] [5. Viewing of singing videos by viewers] A case where a viewing user views a singing video using the mobile terminal 3B will be described.

[0061] When the viewing user wishes to view a singing video, the viewing user launches a video posting and viewing application on the mobile terminal 3B. When the video posting and viewing application is launched, the mobile terminal 3B accesses a server 4 that stores multiple singing video data, i.e., a video posting site. The viewing user then selects the singing video that he or she wishes to view via the video posting and viewing application and displays it on the screen 3a. Note that the method by which the server 4 distributes the singing video to the mobile terminal 3B is, for example, streaming distribution.

[0062] FIG. 5 is an example of a screen display of the mobile terminal 3B when watching a singing video, showing a screen display when live distribution is being performed.

[0063] The screen 3a displays a singing video and also has a singing user display area 50C, a lyrics display area 50D, and a comment input display area 50E. The singing user display area 50C, the lyrics display area 50D, and the comment input display area 50E may be displayed superimposed on the singing video.

[0064] The singing user display area 50C displays the username of the singing user who posted the singing video, an indication of whether or not the video is being distributed live, and the name of the song being sung.

[0065] The lyrics of the singing video are displayed in the lyrics display area 50D. As an example, the lyrics displayed are multiple phrases at a time, and the color of the lyrics changes from the left end to the right end as the music progresses. As an example, the lyrics display area 50D may display lyrics in multiple rows. In this case, when the color change of the lyrics in the upper row reaches the right end, the lyrics in the lower row rises and is displayed in the upper row, and new lyrics are displayed in the lower row, and the color of the lyrics in the upper row changes again from the left end to the right end as the music progresses.

[0066] In the comment input display area 50E, a comment input field is displayed, and comments from viewing users who are viewing the singing video are displayed together with their user names. As an example, each time a comment is input from a viewing user, the comment is additionally displayed at the top of the comment input display area 50E, and comments that were previously displayed are moved down. When the comments cannot be displayed in the comment input display area 50E, a scroll bar (not shown) is displayed on the right side of the comment display area, and comments that were not displayed on the screen 3a until then are displayed by the viewing user operating the scroll bar.

[0067] Furthermore, operation icons 58A to 58D are displayed on the screen 3a to allow the viewing user to perform various operations.

[0068] The operation icon 58A is an icon that is clicked when the viewing user sympathizes with the singing video that the viewing user is viewing, and the total number of clicks of the operation icon 58A for the singing video is displayed above the operation icon 58A.

[0069] The operation icon 58B is an icon that is clicked when the viewing user challenges a singing user who is live streaming the singing video displayed on the screen 3a to a battle (hereinafter referred to as a "singing battle"). In a singing battle, multiple singing videos (first singing video, second singing video) by different singing users are simultaneously displayed on the screen 3a of the viewing user's mobile terminal 3B, and the singing videos take turns singing the same song. In other words, the viewing user who clicks the operation icon 58B becomes a singing user who will participate in a singing battle.

[0070] The operation icon 58C is an icon that is clicked when the viewing user performs various settings for the video posting / viewing application.

[0071] The operation icon 58D is an icon that is clicked by the viewing user when a decorative image is to be superimposed on the singing video displayed on the screen 3a. Note that the decorative images according to this embodiment are priced according to their type, and can be purchased by the viewing user for a fee. The viewing user then superimposes the decorative image on the singing video that he or she is viewing by clicking the operation icon 58D. The singing user of the singing video on which the decorative image is superimposed receives money according to the superimposed decorative image from the operator of the video posting site. In other words, the superimposition (display instruction) of a decorative image on a singing video by the viewing user corresponds to a so-called tipping to the singing user.

[0072] [6. Video composition function] The video posting and viewing application according to the present embodiment has a video synthesis function for synthesizing a plurality of singing videos. The video synthesis function is a function for synthesizing a second singing video recorded later with a first singing video recorded earlier. Note that the songs sung in the first singing video and the second singing video are the same.

[0073] [6-1.Composition start position] Fig. 6 is a schematic diagram showing the contents of the video composition function according to this embodiment. The video composition function according to this embodiment determines a composition start position at which composition of a second singing video to be recorded later starts with a first singing video recorded earlier (Fig. 6(A)), records the second singing video with the camera 3d after determining the composition start position (Fig. 6(B)), and composes the first singing video and the second singing video at the composition start position of the first singing video (Fig. 6(C)). Note that the position used in the composition start position, playback position, etc. is expressed by, for example, the elapsed time from the beginning of the singing video or the song.

[0074] With such a video synthesis function, if a singing user records a singing video (first singing video) but feels that the result is undesirable, the singing user sets a position before the undesirable video part as the synthesis start position and re-shoots the singing video (second singing video) from the synthesis start position. Then, by synthesizing the first singing video and the second singing video at the synthesis start position of the first singing video, the singing user can easily create a singing video that he or she feels is desirable.

[0075] As described above, the video composition function according to the present embodiment starts recording the second singing video after determining the composition start position for the first singing video. Therefore, the video composition function starts recording the second singing video from a playback position of the song corresponding to the composition start position in the first singing video. Therefore, the singing user does not need to sing the song from the beginning when recording the second singing video, but can sing it from the composition start position, so that the singing videos can be composed more easily.

[0076] 7 shows an image displayed on the screen 3a of the mobile terminal 3A operated by the singing user when determining the synthesis start position, and the screen 3a is divided into a playback control area 50F and a playback video display area 50G. The playback video display area 50G displays the first singing video to be played. The playback control area 50F displays a lyric image 52 showing the lyrics of the song of the first singing video, a pitch image 54 showing the pitch of the song, and a slide bar 60 for selecting the playback position of the video.

[0077] The left end of the slide bar 60 indicates the start of the first singing video, and the right end indicates the end of the first singing video. When the singing user moves the pointer 60A left or right, the playback position of the first singing video displayed on the screen 3a changes in accordance with the movement, and the first singing video is displayed on the screen 3a in a paused state. In addition, the lyric image 52 and the pitch image 54 are also updated in accordance with the movement of the pointer 60A.

[0078] Then, when the singing user clicks the enter button 62, the playback position of the first singing video displayed in the playback video display area 50G is determined as the synthesis start position.

[0079] The position that can be set as the synthesis start position may be a predetermined position of the music piece. The predetermined position is, for example, a position where it is easy to start singing from the middle of the lyrics, such as a boundary between phrases in the lyrics, and a plurality of such positions are set. In other words, the singing user cannot select a position where it is difficult to start singing from the middle, such as the middle of a phrase, as the synthesis start position. The slide bar 60 or the lyrics image 52 may display a plurality of positions that can be set as the synthesis start position so that the singing user can recognize them.

[0080] Furthermore, without being limited to the example of Figure 7, screen 3a may display buttons for playing (unpausing) the first singing video, a button for stopping playback, a button for fast-forwarding or rewinding, a button for slow-motion playback, etc.

[0081] [6-2. Recording the second singing video] The video synthesis function records the second singing video when the synthesis start position is determined.

[0082] The video composition function according to the present embodiment has a guide function of displaying a composition start position image 64 (still image), which is an image at the composition start position of the first singing video, on the screen 3a when recording of the second singing video starts, as shown in Fig. 8. The composition start position image 64 is displayed dimmed, for example, so that the singing user can recognize that the image displayed on the screen 3a is the composition start position image 64. Then, the video composition function displays the composition start position image 64 on the screen 3a by superimposing it on an image (hereinafter referred to as a "currently captured image") captured by the camera 3d for recording the second singing video, as shown in Fig. 9.

[0083] In this way, the guide function makes the singing user inevitably check the image at the synthesis start position of the first singing video when recording of the second singing video starts by displaying the synthesis start position image 64 on the screen 3a. Therefore, the singing user can have the same image composition, pose, and expression as the synthesis start position in the first singing video when recording of the second singing video starts. As a result, the first singing video and the second singing video can be synthesized with less awkwardness. Furthermore, as shown in FIG. 9, by superimposing the synthesis start position image 64 and the currently shot image, the singing user can more easily make the image at the start of recording of the second singing video the same as the synthesis start position image 64.

[0084] The guide function may include a detection function for detecting a matching rate between an image of the singing user at a synthesis start position of the first singing video and an image of the singing user at the start of recording of the second singing video. The guide function may include a function for notifying the user when a matching rate between an image of the singing user at a synthesis start position of the first singing video and an image of the singing user at the start of recording of the second singing video is lower than a predetermined threshold.

[0085] 8 and 9, the combination start position image 64 is displayed in the entire playback moving image display area 50G on the screen 3a, but the combination start position image 64 may be displayed in a window and superimposed on a part of the playback moving image display area 50G. The combination start position image 64 displayed in a window and the currently captured image may be displayed superimposed on each other.

[0086] Next, recording of the second singing video will be described with reference to Fig. 10. Fig. 10 is a schematic diagram showing the timing of displaying a synthesis start position image 64 on the screen 3a, and the superimposition of the first singing video and the second singing video, the details of which will be described later, and the bars indicated by diagonal lines (hatching) indicate images displayed on the screen 3a when the second singing video is recorded.

[0087] As shown in FIG. 10, the synthesis start position image 64 is displayed as a still image on the screen 3a before the recording of the second singing video starts, and the display on the screen 3a is stopped when the recording of the second singing video starts.

[0088] Then, the video synthesis function starts playing the music before the synthesis start position, and starts recording the second singing video when the played music reaches a timing corresponding to the synthesis start position. As an example, the video synthesis function according to the present embodiment starts playing the music a predetermined period (e.g., 30 seconds) before the synthesis start position. This allows the singing user to record the second singing video by aligning the start of singing in the second singing video with the start of recording with plenty of time to spare. Furthermore, the video synthesis function starts playing the music before the start of recording the second singing video, and also displays a countdown to the start of recording the second singing video (see FIG. 9). This allows the singing user to clearly recognize the timing of the start of singing in the second singing video.

[0089] 10, after the display of the synthesis start position image 64 is stopped and the recording of the second singing video is started, the first singing video may be superimposed on the screen 3a for a predetermined period from the synthesis start position as a guide function. This allows the singing user to record the second singing video while making an image composition, pose, and expression that match the first singing video, so that the first singing video and the second singing video can be synthesized with less awkwardness. Note that this predetermined period is, as an example, a superimposition synthesis period, the details of which will be described later, and is, for example, 5 seconds, but the superimposition synthesis period may be set based on the number of frames rather than on the basis of time.

[0090] Here, the overall flow of recording the second singing video will be described with reference to FIG.

[0091] When the synthesis start position is determined by the operation of the singing user, a synthesis start position image 64 is displayed on the screen 3a, and shooting by the camera 3d is started, and the synthesis start position image 64 and the currently shot image are superimposed and displayed on the screen 3a. At this time, music data that is the same as the music of the first singing video is downloaded from the server 4 to the mobile terminal 3A. Then, when the singing user inputs an instruction to start recording the second singing video to the mobile terminal 3A, a countdown to recording starts along with playback of the music.

[0092] When the recording countdown ends, recording of the second singing video starts and display of the composition start position image 64 stops. Then, the first singing video from the composition start position during a predetermined period (superimposition period) is displayed on the screen 3a superimposed on the currently captured image. Note that the second singing video to be recorded does not include the first singing video superimposed. When the predetermined period (superimposition period) has elapsed, display of the superimposed first singing video stops, and recording of the second singing video continues until the song ends.

[0093] [6-3. Combining the first and second singing videos] When the recording of the second singing video is finished, the video composition function combines the first singing video with the recorded second singing video at a composition start position of the first singing video. In the following description, the video obtained by combining the first singing video and the second singing video is called a combined singing video.

[0094] The video synthesis function may perform image processing (hereinafter referred to as "effect processing") to make the synthesized portion of the first singing video and the second singing video less noticeable. The effect processing may be, for example, a predetermined period before and after the synthesized portion (for example, 5 seconds before and after, or 5 frames before and after). In the example of FIG. 11, a plurality of predetermined images (star images) are randomly scattered as the effect processing, but the effect processing is not limited to this, and other image processing such as mosaic processing and blurring processing may be performed.

[0095] This effect processing makes the boundary between the first singing video and the second singing video unclear, so that the first singing video and the second singing video can be combined with each other with less awkwardness.

[0096] As an effect process, the entire image of the composite part may be instantly replaced with a predetermined color. The predetermined color may be, for example, white or black, and by intentionally making the predetermined color white (or black), the border between the first singing video and the second singing video becomes unclear.

[0097] The video composition function may also superimpose the first singing video and the second singing video for a predetermined period (the above-mentioned superimposition period) from the composition start position to compose the first singing video and the second singing video (hereinafter referred to as "superimposition") as shown in Fig. 10. This superimposition is an image process called cross dissolve or overlap, and for example, the first singing video to be superimposed may be faded out over time while the second singing video may be faded in. This superimposition makes the boundary between the first singing video and the second singing video unclear, so that the first singing video and the second singing video can be composed with less awkwardness.

[0098] It should be noted that the moving image synthesis function according to the present embodiment does not perform superimposition synthesis on recorded singing voices, but is not limited to this, and may also perform superimposition synthesis on singing voices.

[0099] [7. Video composition function block] 12 is a functional block diagram of the video composition function according to this embodiment. The main control unit 40 of the mobile terminal 3 includes an image display control unit 70, a recording control unit 72, a composition start position determination unit 74, and a video composition unit 76. The processes executed by each function of the main control unit 40 are realized by a program stored in the auxiliary storage unit 44.

[0100] The image display control unit 70 controls the display of images on the screen 3a, and for example, causes a video distributed from the server 4 when a video posting and viewing application is started to be displayed on the screen 3a, or causes an image captured by the camera 3d to be displayed on the screen 3a. When the second singing video is captured, the image display control unit 70 according to the present embodiment causes a synthesis start position image 64 and the first singing video for a predetermined period from the synthesis start position to be superimposed on an image captured by the camera and displayed on the screen 3a.

[0101] The recording control unit 72 records the image captured by the camera 3d. The recording control unit 72 according to the present embodiment starts a video posting and viewing application to play music from the mobile terminal 3A, and records the first singing video or the second singing video together with the sound of the played music. Furthermore, when recording the second singing video, the recording control unit 72 starts playing the music before the synthesis start position, counts down to the start of recording the second singing video, and starts recording the second singing video when the played music reaches the timing corresponding to the synthesis start position.

[0102] The synthesis start position determination unit 74 determines a synthesis start position at which synthesis of the second singing video, which is to be recorded later, starts with the first singing video, which is to be recorded earlier, based on an operation on the mobile terminal 3A by the singing user.

[0103] The video composition unit 76 composes the first singing video and the second singing video at a composition start position of the first singing video to generate a composite singing video. In addition, the video composition unit 76 according to the present embodiment performs image processing to make the composite portion of the first singing video and the second singing video less noticeable, and composes the first singing video and the second singing video by superimposing the first singing video and the second singing video for a predetermined period (superimposition composition period) from the composition start position.

[0104] [8. Flowchart of video synthesis process] 13 is a flowchart showing the flow of moving image synthesis processing executed by the main control unit 40 included in the mobile terminal 3. A program (movie posting and viewing application) for executing the moving image synthesis processing is stored in a predetermined area of ​​the auxiliary storage unit 44 in advance.

[0105] First, in step S100, the synthesis start position determination unit 74 accepts a selection of a first singing video by a singing user. The first singing video is stored in the auxiliary storage unit 44 of the mobile terminal 3A. Note that the singing user may download a singing video that he or she has uploaded from a video sharing site and use it as the first singing video.

[0106] In the next step S102, the image display control unit 70 displays the first singing video on the screen 3a as shown in FIG. 7, and the synthesis start position determination unit 74 determines the synthesis start position based on an operation on the screen 3a by the singing user.

[0107] In the next step S104, the recording control unit 72 determines whether or not an instruction to start recording the second singing video has been input, and if the determination is affirmative, the process proceeds to step S106. If the determination is negative, the process waits until an instruction to start recording the second singing video is input.

[0108] In step S106, the image display control unit 70 displays the synthesis start position image 64 and the image captured by the camera 3d (currently captured image) on the screen 3a.

[0109] In the next step S108, the recording control unit 72 determines whether or not an instruction to start recording the second singing video has been input, and if the determination is affirmative, the process proceeds to step S110. If the determination is negative, the process waits until an instruction to start recording the second singing video is input.

[0110] In step S110, the recording control unit 72 starts playing the music before the synthesis start position, starts recording the second singing video when the played music reaches the timing corresponding to the synthesis start position, and ends recording the second singing video when the music ends. The recorded second singing video is stored in the auxiliary storage unit 44.

[0111] In step S112, the video synthesis unit 76 synthesizes the first singing video and the second singing video at the synthesis start position of the first singing video to generate a synthesized singing video. Note that the first singing video and the second singing video may be synthesized by superimposing synthesis. The synthesized singing video is stored in the auxiliary storage unit 44.

[0112] In the next step S114, the moving image synthesizing unit 76 performs effect processing on the synthesized singing moving image, and the moving image synthesizing process ends. Note that the type of effect processing to be executed is preset by the singing user.

[0113] The singing user then uploads the composite singing video thus generated to a video posting site via a video posting and viewing application, and the composite singing video can be viewed by a viewing user. If the first singing video has been uploaded to the video posting site, the first singing video may be replaced with the composite singing video by uploading. If the singing user is not satisfied with the completeness of the generated composite singing video, the video synthesis process shown in FIG. 13 is performed again from the beginning to generate the composite singing video again.

[0114] As described above, the mobile terminal 3A, which records the image captured by the camera 3d as a video, determines a synthesis start position for starting synthesis of the second singing video to be recorded later with the first singing video recorded earlier, records the second singing video with the camera 3d, and synthesizes the first singing video and the second singing video at the synthesis start position. Therefore, when the singing user records a video using the camera 3d, the mobile terminal 3A can easily create a video that the singing user finds desirable.

[0115] 9. Other Embodiments The present invention has been described above using the above embodiment, but the technical scope of the present invention is not limited to the scope described in the above embodiment. Various modifications or improvements can be made to the above embodiment without departing from the gist of the invention, and the forms with such modifications or improvements are also included in the technical scope of the present invention. In addition, the above embodiments may be combined as appropriate.

[0116] For example, in the above embodiment, the performance is singing and the video uploaded to the video sharing site is a singing video, but the present invention is not limited to this. For example, the performance may be something other than singing, such as dancing and the video uploaded to the video sharing site is a dance video.

[0117] In the above embodiment, the video composition function is executed by the mobile terminal 3A, but the present invention is not limited to this. For example, a part or all of the video composition function may be executed by the server 4. Even if a part or all of the video composition function is executed by the server 4, the video is photographed and recorded by the camera 3d of the mobile terminal 3A.

[0118] In addition, the video synthesis function may generate a new composite singing video by synthesizing the first singing video and the second singing video, and generate a new composite singing video by synthesizing the first singing video with the new second singing video.

[0119] Furthermore, the flow of the video synthesis process described in the above embodiment is also one example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged within the scope of the present invention. [Explanation of symbols]

[0120] 3. Mobile terminals (information processing devices) 3a screen 3d camera 64 Composite start position image 70 Image display control unit (image display control means) 72 Recording control unit (recording control means) 74 Synthesis start position determining unit (determining means) 76 Video composition unit (video composition means)

Claims

1. a processor, the processor comprising: Identifying a first video from among pre-recorded videos by a user selection; determining a predetermined position in the first moving image before the user issues an instruction to start shooting a second moving image; when the instruction to start shooting is received from the user, starting shooting the second video with a camera; During shooting of the second moving image, a first image, which is an image at the predetermined position, is displayed superimposed on a second image, which is an image of the second moving image; Recording the second moving image based on the predetermined position; displaying the first moving image superimposed on the second moving image from the time at the predetermined position until a predetermined time has elapsed; creating a third moving image, the third moving image including a moving image in which an image of the second moving image is superimposed on an image of the first moving image during the predetermined time period, and including the first moving image and the second moving image from the predetermined position onward; Information processing device.

2. A processor identifies a first video from the pre-recorded videos as selected by a user; a processor determines a predetermined position in the first moving image before the user issues an instruction to start shooting a second moving image; when the processor receives the instruction to start shooting from the user, starts shooting the second video using a camera; a processor, during shooting of the second moving image, displays a first image, which is an image at the predetermined position, superimposed on a second image, which is an image of the second moving image; a processor for recording the second moving image based on the predetermined position; A processor displays the first moving image superimposed on the second moving image from the time at the predetermined position until a predetermined time has elapsed, a processor creates a third moving image, the third moving image including a moving image in which an image of the second moving image is superimposed on an image of the first moving image during the predetermined time, and including the first moving image and the second moving image from the predetermined position onward; Information processing methods.

3. causing a processor to identify a first video from among the pre-recorded videos by a user selection; causing a processor to determine a predetermined position in the first moving image before the user issues an instruction to start shooting a second moving image; causing a processor to start shooting the second video using a camera when the processor receives the instruction to start shooting from the user; causing a processor to display a first image, which is an image at the predetermined position, superimposed on a second image, which is an image of the second moving image, during shooting of the second moving image; causing a processor to record the second moving image based on the predetermined location; causing a processor to superimpose and display an image of the first moving image on an image of the second moving image from a time point at the predetermined position until a time point when a predetermined time has elapsed; causing a processor to create a third moving image, the third moving image including a moving image in which an image of the second moving image is superimposed on an image of the first moving image during the predetermined time period, and including the first moving image and the second moving image from the predetermined position onward; program.

4. A server and a terminal device, the server comprising: accepting a selection of a first moving image from the terminal device from among pre-recorded moving images; determining a predetermined position in the first video before issuing an instruction from the terminal device to start shooting a second video; when the instruction to start shooting is received from the terminal device, starting to shoot the second video by a camera; causing the terminal device to display a first image, which is an image at the predetermined position, superimposed on a second image, which is an image of the second video, during shooting of the second video; Recording the second moving image based on the predetermined position; displaying the first moving image superimposed on the second moving image from the time at the predetermined position until a predetermined time has elapsed; creating a third moving image, the third moving image including a moving image in which an image of the second moving image is superimposed on an image of the first moving image during the predetermined time period, and including the first moving image and the second moving image from the predetermined position onward; system.

Images