Information processing device, information processing method, and program

Abstract

To enable a plurality of remote users to easily capture images of themselves in real time.SOLUTION: In a program according to an aspect of the present technology, a moving image in which a user using a mobile terminal appears is captured, and a live view image generated on the basis of the moving image in which the user appears and a moving image in which another user captured in real time on another mobile terminal used by the other user appears is displayed. In addition, after the start of display of the live view image, a still image is captured at the same timing as that captured by the other mobile terminal, and a composite image generated by synthesizing a still image in which at least the user appears and a still image in which at least the other user appears into one image is displayed. This technology can be applied to a mobile terminal application equipped with a camera.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] This technology relates to an information processing apparatus, an information processing method, and a program, and particularly relates to an information processing apparatus, an information processing method, and a program that enable a plurality of users who are remotely located to easily capture images in which they are reflected in real time.

Background Art

[0002] There is collage photography in which a single image is generated by combining a plurality of captured images. According to collage photography, it is possible to obtain a highly designed image by combining a plurality of captured images.

[0003] Patent Document 1 discloses a technique for generating a collage image by arranging, in a layout frame of a template, an image having a high relevance to a selected reference image together with the reference image. An image having a high relevance is extracted based on the shooting date and time and arranged according to the priority order to generate a collage image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the technique described in Patent Document 1, although it is possible to generate a collage image in which a plurality of people are reflected by combining captured images in which each person is reflected, since the generation of the collage image is performed based on images captured in the past, it is not possible to enjoy collage photography.

[0006] If we were to generate a collage image based on images taken by each user in real time, it would be difficult to take the photos when the users are in different locations, as they wouldn't know what the other is doing. Each user wouldn't know what angle of view to aim for, or what pose to strike in the photo.

[0007] This technology was developed in response to these circumstances, enabling multiple users located remotely to easily capture images of themselves in real time. [Means for solving the problem]

[0008] One aspect of this technology is an information processing device which includes an image processing unit that displays a live view image generated based on multiple moving images of the user using the mobile terminal, each of the multiple mobile terminals, which are captured in real time on each of the multiple mobile terminals, and a shooting control unit that causes each of the multiple mobile terminals to take a still image at the same timing after the start of displaying the live view image, wherein the image processing unit combines the still images of the user captured on each of the multiple mobile terminals into a single image and outputs the combined image on each of the mobile terminals.

[0009] Another aspect of this technology involves a program that captures a video of a user using a mobile device, displays a live view image generated based on the video of the user and a video of the other user captured in real time on another mobile device used by that other user, captures a still image at the same timing as the other mobile device captures a still image after the start of displaying the live view image, and displays a composite image generated by combining a still image that includes at least the user and a still image that includes at least the other user into a single image.

[0010] In one aspect of this technology, a live view image is generated based on multiple video images of the user using the mobile device, which are captured in real time on each of the multiple mobile devices, and this image is displayed on each mobile device. At the same time after the display of the live view image begins, a still image is captured on each of the multiple mobile devices, and the still images captured on each of the multiple mobile devices, which include at least one user, are combined into a single image, and the combined image is output on each of the mobile devices.

[0011] In another aspect of this technology, a video of a user using a mobile device is captured, a live view image is displayed which is generated based on the video of the user and a video of the other user captured in real time by another mobile device used by another user, a still image is captured at the same timing as the other mobile device captures the video after the live view image is displayed, and a composite image is displayed which is generated by combining a still image that includes at least the user and a still image that includes at least the other user into a single image. [Effects of the Invention]

[0012] This technology allows multiple users located remotely to easily capture images of themselves in real time. [Brief explanation of the drawing]

[0013] [Figure 1] This figure shows an example configuration of a remote shooting system according to one embodiment of this technology. [Figure 2] This is a diagram showing an example of a selfie. [Figure 3] This diagram shows how captured images are transmitted during remote shooting. [Figure 4] This figure shows an example of a composite image. [Figure 5] This figure shows how composite images are transmitted during remote shooting. [Figure 6] This figure shows an example of users performing remote shooting together. [Figure 7] It is a diagram showing another example of a user who performs remote shooting together. [Figure 8] It is a diagram showing an example of a composite image. [Figure 9] It is a block diagram showing an example of the hardware configuration of a remote shooting management server. [Figure 10] It is a block diagram showing an example of the functional configuration of a remote shooting management server. [Figure 11] It is a diagram showing an example of setting information. [Figure 12] It is a block diagram showing an example of the configuration of a smartphone. [Figure 13] It is a block diagram showing an example of the functional configuration of a smartphone. [Figure 14] It is a flowchart explaining a series of processes for remote shooting. [Figure 15] It is a flowchart following FIG. 14 and explaining a series of processes for remote shooting. [Figure 16] It is a diagram showing an example of the display of a live view image. [Figure 17] It is a diagram showing an example of the display of a live view image. [Figure 18] It is a diagram showing an example of the display of a live view image. [Figure 19] It is a diagram showing an example of the display of a live view image. [Figure 20] It is a diagram showing an example of the display of a live view image. [Figure 21] It is a diagram showing an example of the display of a live view image. [Figure 22] It is a diagram showing an example of the display of a live view image. [Figure 23] It is a diagram showing an example of the display of a guide. [Figure 24] It is a diagram showing an example of a provided image. [Figure 25] It is a diagram showing an example of a placement position. [Figure 26] It is a diagram showing an example of a composite image in the case of performing one-to-many remote shooting. [Modes for carrying out the invention]

[0014] The following describes the configurations for implementing this technology. The explanation will proceed in the following order. 1. Remote shooting system 2. Example of user invitation 3. Configuration of each device 4. Series of processes for remote shooting 5. Displaying the live view image 6. Provided images 7. Image adjustment 8. Other

[0015] <Remote shooting system> Figure 1 shows an example configuration of a remote shooting system according to one embodiment of this technology.

[0016] The remote shooting system shown in Figure 1 is configured by having multiple smartphones 2 connected to a remote shooting management server 1 via the internet 11. In the example in Figure 1, smartphone 2A used by user A and smartphone 2B used by user B are shown. User A and user B are, for example, friends who are located far apart.

[0017] Other devices equipped with a communication module and a camera, such as tablet devices or PCs, may be used in place of smartphone 2. The camera may be a built-in camera or an external camera.

[0018] Smartphone 2 has a remote shooting app installed. The remote shooting app is an application program that runs on the OS (Operating System) installed on Smartphone 2.

[0019] A remote shooting app is an application that allows multiple users in different locations to take a picture together. Taking a picture using a remote shooting app is called remote shooting.

[0020] Before remote shooting begins, as shown by the white arrows in Figure 1, the user who is hosting the remote shooting configures the settings for remote shooting on the remote shooting application and invites other users to participate in the shooting. In the example in Figure 1, User A is the host user. By accepting User A's invitation, User A and User B will participate in the remote shooting together.

[0021] Remote shooting proceeds according to a predetermined procedure, for example, by following guidance from a remote shooting app. Each user can enjoy remote shooting as if it were a game.

[0022] As shown in Figure 2, each user will point the camera of smartphone 2 at themselves and take a selfie while looking at the screen of the remote shooting app. The front of the smartphone 2's casing is equipped with a camera and a display consisting of an LCD or similar.

[0023] The remote shooting app screen displays a live view image, which is a video of users A and B who are shooting together remotely. Users A and B can check their expressions and decide on poses by looking at the live view image displayed on their respective smartphones 2. During the remote shooting session, each user can also converse with the other.

[0024] User A's shooting on smartphone 2A and User B's shooting on smartphone 2B are performed at the same time (in real time), for example, according to control by the remote shooting management server 1. Remote shooting is performed synchronously on each smartphone 2.

[0025] Captured image P1, which is a still image of user A taken with smartphone 2A, and captured image P2, which is a still image of user B taken with smartphone 2B, are both transmitted to the remote shooting management server 1 as shown by the white arrows in Figure 3.

[0026] As shown in Figure 4, the remote shooting management server 1 combines captured image P1 and captured image P2 into a single image to generate a composite image P11. In the example in Figure 4, the composite image P11 is generated by positioning user A on the left and user B on the right.

[0027] The combination of captured image P1 and captured image P2 is performed, for example, by combining images of at least the human region extracted from captured image P1 and captured image P2 using chroma keying or similar methods. A background image is appropriately composited into the layer below the human region.

[0028] Thus, captured image P1 and captured image P2 are still images used to generate the composite image P11. Hereafter, captured images taken with each smartphone 2 and used to generate the composite image will be referred to as composite images.

[0029] Note that in the example in Figure 4, the dashed lines representing the boundaries of the areas assigned to each user are shown in the center of the composite image P11, but these are not actually displayed on the composite image P11.

[0030] The composite image P11 generated by the remote shooting management server 1 is sent to smartphones 2A and 2B, respectively, as shown by the white arrows in Figure 5. Users A and B can obtain a still image of themselves together, even though they are in different locations.

[0031] In a single remote shooting session, the above type of shooting may occur multiple times. User A and User B can enjoy shooting together while conversing, and obtain multiple composite images.

[0032] Thus, the remote shooting management server 1 is an information processing device that provides a composite image, obtained by combining still images taken at the same time, to users who participated in the remote shooting together.

[0033] <Example of user invitation> Figure 6 shows an example of users performing remote shooting together.

[0034] In the example shown in Figure 6, User A is the organizer, and invitations to remote shooting are sent to both User B and User C.

[0035] After remote shooting begins, smartphone 2A will take a picture of user A, smartphone 2B will take a picture of user B, and smartphone 2C will take a picture of user C. Smartphones 2A through 2C will take pictures at the same time.

[0036] The three images for composite image creation, each captured by smartphones 2A through 2C, are transmitted to the remote image management server 1 and used to generate the composite image.

[0037] A composite image showing three people, users A through C, is sent to smartphones 2A through 2C, respectively. Users A through C can obtain a composite image showing all three of them together by taking a remote photo together.

[0038] As you can see, remote shooting can sometimes be done by three people. The number of users participating in remote shooting is arbitrary, and can be four or more.

[0039] Figure 7 shows another example of users performing remote shooting together.

[0040] In the example shown in Figure 7, users A and B are in the same location and are taking photos together using user A's smartphone, smartphone 2A. User C is a user who is in a different location from users A and B.

[0041] Thus, in remote shooting, a single user can include not only themselves but also their friends in the frame. In the example in Figure 7, User A, who is with User B, is the host and has sent a remote shooting invitation to User C.

[0042] After remote shooting begins, smartphone 2A will take photos of users A and B. Smartphone 2C will take photos of user C. Both smartphones 2A and 2C will take photos at the same time.

[0043] The two images taken with smartphone 2A and smartphone 2C are sent to the remote shooting management server 1 and used to generate the composite image.

[0044] A composite image featuring three people, A through C, is sent to smartphone 2A and smartphone 2C, respectively. For example, if user A sends the composite image to user B using social media, users A through C can obtain a composite image showing all three of them together.

[0045] In this way, a single smartphone 2 may be used to capture a single composite image that includes multiple users. The user performing the remote shooting using smartphone 2 may be the owner of smartphone 2, or it may be the owner and multiple users who are nearby and included in the same image's field of view.

[0046] Figure 8 shows an example of a composite image.

[0047] The composite image shown in Figure 8A is a composite image generated when remote shooting is performed on each of the smartphones 2A to 2C, as explained with reference to Figure 6.

[0048] For example, when the composite images taken with smartphone 2A showing user A, taken with smartphone 2B showing user B, and taken with smartphone 2C showing user C are combined into a single image, a composite image showing three people, A through C, as shown in Figure 8A, is generated.

[0049] The composite image shown in Figure 8B is a composite image generated when remote shooting is performed on smartphone 2A and smartphone 2C, respectively, as explained with reference to Figure 7.

[0050] The composite images of users A and B, taken with smartphone 2A, and user C, taken with smartphone 2C, are combined into a single image, generating a composite image of users A through C as shown in Figure 8B. In the example in Figure 8B, a wider area is allocated to the composite image taken with smartphone 2A, which shows users A and B, than to the composite image taken with smartphone 2C, which shows user C, and the image is then combined.

[0051] <Configuration of each device> • Configuration of Remote Shooting Management Server 1 Figure 9 is a block diagram showing an example of the hardware configuration of the remote shooting management server 1.

[0052] As shown in Figure 9, the remote imaging management server 1 is configured by a computer. The remote imaging management server 1 may also be configured by multiple computers. When the remote imaging management server 1 is configured by multiple computers, various processes for providing composite images are realized through the cooperation of each computer.

[0053] The CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, and RAM (Random Access Memory) 103 are interconnected by a bus 104.

[0054] An input / output interface 105 is further connected to the bus 104. An input unit 106, an output unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

[0055] The input unit 106 consists of a keyboard, mouse, etc. The output unit 107 consists of a display, etc.

[0056] The storage unit 108 consists of a hard disk, non-volatile memory, and the like. The storage unit 108 stores various types of information used to provide a composite image, such as programs executed by the CPU 101 and remote shooting settings.

[0057] The communication unit 109 is an interface to the internet 11. For example, the communication unit 109 communicates with the smartphone 2 and receives the composite images transmitted from the smartphone 2. The communication unit 109 also transmits the composite images generated based on the composite images to each of the smartphones 2.

[0058] Drive 110 controls the writing of data to and reading of data from the removable media 111.

[0059] Figure 10 is a block diagram showing an example of the functional configuration of the remote shooting management server 1.

[0060] As shown in Figure 10, an information processing unit 121 is implemented in the remote shooting management server 1. The information processing unit 121 consists of a game control unit 131, a setting information DB 132, an image acquisition unit 133, and an image processing unit 134.

[0061] The game control unit 131 communicates with the smartphone 2 used by the user who is the organizer of the remote shooting by controlling the communication unit 109, and accepts settings related to remote shooting. Various settings related to remote shooting are made using the remote shooting application. The game control unit 131 stores and manages setting information representing the content of the settings related to remote shooting in the setting information DB 132.

[0062] Figure 11 shows an example of configuration information.

[0063] As shown in Figure 11, the configuration information includes the shooting date and time, the host user, invited users, number of participants, image settings, game time, and URL. Configuration information containing all of this information is generated and managed for each remote shooting session. Not all of the information shown in Figure 11 is included in the configuration information; at least some of it may be included.

[0064] The shooting date and time indicates the start time of remote shooting. When the start time indicated by the shooting date and time arrives, each user participating in remote shooting together will launch the remote shooting app and begin shooting. If the remote shooting app is launched before the start time indicated by the shooting date and time, shooting will be permitted when the start time arrives.

[0065] The organizer user represents the user who will be the organizer of the remote shooting. The game control unit 131 identifies the organizer user and records, for example, the organizer user's ID as organizer user information.

[0066] An invited user is a user who has been invited by the host user. A user is invited, for example, by selecting from a group of users who have been pre-registered as friends of the host user. The host user may also allow invited users to invite other users. For example, an invited user can invite other users with the host user's consent.

[0067] The game control unit 131 identifies users invited by the host user and records, for example, the invited user's ID as information about the invited user. If multiple users are invited, the user ID of each user is recorded. The game control unit 131 also functions as a user management unit that identifies and manages users who will be participating in remote filming together.

[0068] The number of users represents the number of users performing remote shooting. For example, if invitations are made as explained with reference to Figure 7, the information recorded will indicate that two users will perform shooting using smartphone 2A and one user will perform shooting using smartphone 2C. In this way, the number of users performing remote shooting using smartphone 2 is recorded for each smartphone 2.

[0069] Image settings refer to settings related to image quality, such as color tone and brightness. Different image settings may be used for each smartphone, or the same settings may be used for all smartphones.

[0070] The system may integrate with other apps, such as camera apps, so that the image settings selected in those apps are used for remote shooting. Alternatively, the system may use image settings that have been pre-selected using other apps, or the user may be able to choose whether or not to use the image settings of other apps while viewing the live view image.

[0071] The image quality settings of other apps may be applied to all users participating in remote shooting, or they may be applied to each user individually. If applied to each user individually, for example, it may be possible to use camera app image quality setting α for user A and camera app image quality setting β for user B.

[0072] It would be possible to allow each user to select which app's image settings to apply. For example, if multiple camera apps are available, it would be possible to use the image settings of one camera app for user A's photos and the settings of another camera app for user B's photos.

[0073] Game time represents the duration of a single remote shooting session. Multiple shooting sessions (for example, the generation of a composite image) may occur during the game time.

[0074] The URL is an identifier assigned to remote shooting. When the host user configures settings for remote shooting, a URL is issued. The host user invites users to participate in remote shooting by notifying them of the URL issued by the remote shooting management server 1. The notification of the URL to the invited users may also be handled by the remote shooting management server 1.

[0075] A URL could be issued for each remote shoot, or the URL could remain the same, but a different ID and password could be issued for each remote shoot. The user could set the ID and password. The user could set the ID, and the password could be automatically generated. The ID and password could be automatically generated.

[0076] Alternatively, a QR code may be issued instead of a URL as identification information for remote shooting. Alternatively, an invitation code consisting of a predetermined number of digits or letters may be issued as identification information for remote shooting. The invitation code is issued when the host user launches the remote shooting application and configures the remote shooting settings.

[0077] The system can restrict who can be invited to specific users (friend function). For example, it may be possible to invite only users who are set as friends. Users that the organizer follows on a given service, or users who follow the organizer, will be set as friends who can be invited. In addition, users that the organizer has set as friends on the SNS service they use will be set as friends who can be invited. This can prevent unintended users from participating in remote filming and improve security.

[0078] You may also want to generate a URL or QR code that contains the configuration information.

[0079] In this way, the settings for remote shooting between any two users are configured as preparation for remote shooting. The configuration information managed by the game control unit 131 is read from the configuration information DB 132 at the start of remote shooting and supplied to the image processing unit 134.

[0080] Furthermore, the game control unit 131 communicates with multiple smartphones 2 used by users who are performing remote shooting together, and controls the series of remote shooting processes on each smartphone 2.

[0081] In other words, the game control unit 131 works in conjunction with the remote shooting app on each of the smartphones 2 that are performing remote shooting together, and synchronizes the shooting and other operations on each smartphone 2. The game control unit 131 also functions as a shooting control unit that controls the shooting of moving and still images.

[0082] Instead of both video and still image capture being performed in accordance with the control of the game control unit 131, at least one of the captures may be performed by the remote capture application without control from the game control unit 131.

[0083] The image acquisition unit 133 acquires images captured on each of the multiple smartphones 2 used by the user performing the remote shooting together, and transmitted from each smartphone 2, by controlling the communication unit 109.

[0084] Before a still image is taken, each smartphone 2 transmits a live view video, which is a video used to generate a live view image. When a still image is taken, each smartphone 2 transmits a composite image, which is a still image used to generate a composite image. The image acquired by the image acquisition unit 133 is supplied to the image processing unit 134.

[0085] The image processing unit 134 generates a live view image based on the live view video supplied from the image acquisition unit 133.

[0086] The generation of the live view image is performed according to the configuration information supplied by the game control unit 131. For example, based on the number of people information included in the configuration information, the placement pattern of users appearing in each live view video is determined. In addition, image processing is performed on the live view video based on the image quality information included in the configuration information.

[0087] Furthermore, the image processing unit 134 generates a composite image based on the composite image supplied by the image acquisition unit 133. The generation of the composite image is also performed according to the contents of the setting information supplied by the game control unit 131, similar to the generation of the live view image. A composite image is generated in which each user is pictured in the same arrangement as in the live view image.

[0088] The image processing unit 134 controls the communication unit 109 to transmit the live view image and the composite image to the smartphone 2, and to output them on each smartphone 2. The image processing unit 134 controls the output of images on the smartphone 2 used by each user performing remote shooting together, such as displaying the live view image or the composite image, or storing the composite image in memory.

[0089] • Smartphone 2 configuration Figure 12 is a block diagram showing an example configuration of smartphone 2.

[0090] As shown in Figure 12, the smartphone 2 consists of a control unit 301, memory 302, communication unit 303, display 304, operation unit 305, camera 306, speaker 307, and microphone 308.

[0091] The control unit 301 is composed of a CPU, ROM, RAM, etc. The control unit 301 controls the overall operation of the smartphone 2 by executing a predetermined program.

[0092] In the control unit 301, the application execution unit 301A is implemented. The remote shooting application is executed by the application execution unit 301A, thereby realizing a series of processes for remote shooting.

[0093] Memory 302 is composed of flash memory and the like. Memory 302 stores various information such as the composite image transmitted from the remote shooting management server 1 and the application executed by the control unit 301.

[0094] The communication unit 303 communicates with the remote shooting management server 1 via the internet 11. The communication unit 303 receives information transmitted from the remote shooting management server 1 and outputs it to the control unit 301.

[0095] The display 304 is composed of an organic EL display, an LCD, etc. The display 304 displays various screens, such as the screen of a remote shooting application.

[0096] The control unit 305 consists of various buttons and a touch panel superimposed on the display 304. The control unit 305 outputs signals representing the user's actions to the control unit 301.

[0097] Camera 306 captures moving and still images according to the control unit 301. The images captured by camera 306 are supplied to the control unit 301.

[0098] The speaker 307 outputs the voice of the person with whom it is remotely filming, based on the audio data supplied from the control unit 301.

[0099] The microphone 308 detects the user's voice and outputs the voice data to the control unit 301. The user's voice is transmitted to the smartphone 2 of the person performing the remote shooting together via the remote shooting management server 1.

[0100] Figure 13 is a block diagram showing an example of the functional configuration of smartphone 2.

[0101] As shown in Figure 13, in the smartphone 2, the shooting control unit 311, output control unit 312, and remote shooting setting unit 313 are realized when the remote shooting application is executed by the application execution unit 301A.

[0102] The shooting control unit 311 controls the camera 306 to capture moving images and still images. The shooting control unit 311 transmits the captured moving images to the remote shooting management server 1 as moving images for live view. The shooting control unit 311 also transmits the captured still images to the remote shooting management server 1 as images for composite shooting.

[0103] The output control unit 312 displays the screen of the remote shooting application on the display 304 based on the information transmitted from the remote shooting management server 1. For example, the output control unit 312 displays the live view image transmitted from the remote shooting management server 1 on the display 304. The output control unit 312 also displays the composite image transmitted from the remote shooting management server 1 as the image of the shooting result on the display 304.

[0104] The output control unit 312 outputs the composite image transmitted from the remote shooting management server 1 to the memory 302 and stores it there.

[0105] The remote shooting setting unit 313 receives user input regarding remote shooting settings and transmits information representing the settings to the remote shooting management server 1. The remote shooting management server 1 generates the setting information described above based on the information received by the remote shooting setting unit 313.

[0106] <Series of processes for remote shooting> Refer to the flowcharts in Figures 14 and 15 to explain the series of processes involved in remote shooting.

[0107] Here, as explained with reference to Figure 1, User A acts as the host and invites User B, and User A and User B perform remote shooting together. The entire process takes place between User A's smartphone 2A, User B's smartphone 2B, and the remote shooting management server 1.

[0108] If user A instructs to start remote shooting by specifying a pre-issued URL, in step S1, the remote shooting application on smartphone 2A accesses the remote shooting management server 1 via the internet 11.

[0109] Similarly, if user B instructs the start of remote shooting, in step S21, the remote shooting application on smartphone 2B accesses the remote shooting management server 1 via the internet 11.

[0110] In steps S41 and S42, the game control unit 131 of the remote shooting management server 1 accepts access from smartphones 2A and 2B, respectively.

[0111] In step S2, the shooting control unit 311 of the smartphone 2A starts shooting a video for live view.

[0112] In step S3, the shooting control unit 311 of the smartphone 2A transmits the live view video to the remote shooting management server 1.

[0113] In step S22, the shooting control unit 311 of the smartphone 2B starts shooting a video for live view.

[0114] In step S23, the shooting control unit 311 of the smartphone 2B transmits the live view video to the remote shooting management server 1.

[0115] In steps S43 and S44, the image acquisition unit 133 of the remote shooting management server 1 acquires live view video images transmitted from smartphones 2A and 2B.

[0116] In step S45, the image processing unit 134 generates a live view image based on the acquired live view video.

[0117] In step S46, the image processing unit 134 transmits the live view image to smartphones 2A and 2B, respectively.

[0118] In step S4, the output control unit 312 of the smartphone 2A receives the live view image transmitted from the remote shooting management server 1.

[0119] In step S5, the output control unit 312 of the smartphone 2A displays the live view image on the display 304.

[0120] In step S24, the output control unit 312 of the smartphone 2B receives the live view image transmitted from the remote shooting management server 1.

[0121] In step S25, the output control unit 312 of the smartphone 2B displays the live view image on the display 304.

[0122] Figure 16 shows an example of a live view image display.

[0123] As shown in Figure 16, a live view image featuring users A and B, generated from live view video footage captured by smartphone 2A and smartphone 2B, is displayed across almost the entire screen. The person on the left is user A, and the person on the right is user B. The user may be allowed to change the position of the people when the live view image is displayed.

[0124] The live view image is created, for example, by placing live view video footage captured by smartphone 2A and live view video footage captured by smartphone 2B in their respective display areas and combining them. Each display area contains at least an image of a person extracted from each live view video footage.

[0125] The shutter button 401 is displayed overlaid on the live view image. Next to the shutter button 401, guidance is displayed instructing the user to press the shutter button 401 when ready to take a picture. The guidance is also outputted via voice as needed.

[0126] Such live view images are displayed synchronously on both smartphone 2A and smartphone 2B. Users A and B pose while looking at the live view images displayed on the screen, such as while having a conversation.

[0127] For example, if user A presses the capture button 401, in step S6 of Figure 15, the capture control unit 311 of the smartphone 2A sends information to the remote capture management server 1 indicating that the capture button 401 has been pressed.

[0128] In step S47, the game control unit 131 of the remote shooting management server 1 receives information transmitted from the smartphone 2A.

[0129] In step S48, the game control unit 131 instructs the smartphones 2A and 2B to take the first still image.

[0130] In step S7, the shooting control unit 311 of the smartphone 2A receives information transmitted from the remote shooting management server 1, and in step S8, controls the camera 306 to take a still image.

[0131] On the remote shooting app screen, for example, a countdown display starts, and a still image is taken when the countdown ends. Instead of taking a picture immediately when the shooting button 401 is pressed, a predetermined amount of time elapses after the shooting button 401 is pressed, and then, in synchronization between smartphones 2A and 2B, a still image is taken at the same time after the live view image display starts.

[0132] In step S9, the shooting control unit 311 of the smartphone 2A transmits the still image obtained by shooting as a composite image to the remote shooting management server 1.

[0133] In smartphone 2B, images for composite capture are taken in sync with the shooting on smartphone 2A. Specifically, in step S26, the shooting control unit 311 of smartphone 2B receives information transmitted from the remote shooting management server 1, and in step S27, controls the camera 306 to capture a still image.

[0134] In step S28, the shooting control unit 311 of the smartphone 2B transmits the still image obtained by shooting as a composite image to the remote shooting management server 1.

[0135] In steps S49 and S50, the image acquisition unit 133 of the remote shooting management server 1 receives the composite images transmitted from the smartphones 2A and 2B.

[0136] In step S51, the image processing unit 134 generates a composite image based on the captured images for synthesis.

[0137] In step S52, the image processing unit 134 transmits the generated composite image to smartphones 2A and 2B, respectively.

[0138] In step S10, the output control unit 312 of the smartphone 2A receives the composite image transmitted from the remote shooting management server 1.

[0139] In step S11, the output control unit 312 of the smartphone 2A displays the composite image as the image of the captured image on the display 304. The composite image is displayed on the display 304 instead of the live view image.

[0140] In step S29, the output control unit 312 of the smartphone 2B receives the composite image transmitted from the remote shooting management server 1.

[0141] In step S30, the output control unit 312 of the smartphone 2A displays the composite image as the image of the captured image on the display 304. The composite image is displayed on the display 304 instead of the live view image.

[0142] In this way, the composite image resulting from the photo shoot is displayed synchronously on both smartphone 2A and smartphone 2B. User A and User B can enjoy talking while looking at the same composite image.

[0143] The composite image transmitted from the remote shooting management server 1 is displayed on the display 304 and also stored in, for example, the memory 302, so that the user can view it at any time by operating the remote shooting application.

[0144] The above shooting process is repeated a number of times, for example, as set in the remote shooting settings. After multiple shots are taken and multiple composite images are provided to the user, the process is completed.

[0145] Through the above process, each user can obtain a still image of themselves together, even if they are in different locations.

[0146] While the user is expected to press the shutter button 401 each time a still image is taken, it is also possible to have continuous shooting occur after the shutter button 401 is pressed once. Continuous shooting may be performed, for example, at predetermined time intervals.

[0147] After continuous shooting has begun, it may be possible to temporarily pause the shooting. This would allow the user to pose or adjust their appearance.

[0148] Instead of any of the users participating in the photoshoot being able to start the recording, it may be possible to restrict it so that only the host user can start it. In this case, the recording button 401 will only be displayed on the screen of the host user's smartphone 2.

[0149] It may be possible to allow each user to press the shutter button 401. This would serve as a compensatory measure in case of a time lag.

[0150] <Displaying Live View Image> Here, we will explain the details of the live view image display. The following live view image display is realized, for example, through the cooperation of the game control unit 131 (Figure 10) of the remote shooting management server 1 and the remote shooting application (output control unit 312 in Figure 13) of the smartphone 2.

[0151] • Setting the number of divisions Figure 17 shows an example of a live view image display.

[0152] In Figure 17, the display of the shutter button 401, etc., is omitted. The same applies to Figure 18 and other figures described later.

[0153] When two users are taking photos together, the live view image is displayed divided into two display areas, A1 and A2, as shown in Figure 17. Each display area shows the user appearing in the live view video captured by smartphone 2A and smartphone 2B, respectively.

[0154] As shown in Figure 17, if user A is displayed in display area A1 and user B is displayed in display area A2, the composite image obtained after still image capture will also show user A on the left and user B on the left. By displaying such a live view image, users can take pictures while visualizing the final result. The system may also accept changes to the user's position in the composite image so that it differs from the position in the live view image.

[0155] The user could choose whether or not to horizontally flip the live view image and the composite image. By displaying the image as if looking at one's reflection in a mirror, the user can reduce any sense of unease about their own appearance.

[0156] Figure 18 shows another example of how the live view image can be displayed.

[0157] If there are three users taking photos together, the live view image will be displayed divided into three display areas A1, A2, and A3, as shown in Figure 18. Each display area will show the user appearing in the live view video captured on each of the smartphones 2A through 2C.

[0158] As shown in Figure 18, when users A, B, and C are displayed in display areas A1 to A3 respectively, the composite image obtained after still image capture will also show user A on the left, user B in the center, and user C on the right.

[0159] The number of divisions in the live view image is determined by the number of users taking photos together.

[0160] The number of users participating in the shooting is determined by the remote shooting management server 1 to be, for example, the same number as the number of smartphones 2 accessing simultaneously. In this case, the live view image is divided into the same number of display areas as the number of mobile devices used for remote shooting, and each display area shows the user appearing in the live view video captured by each smartphone 2.

[0161] The number of users participating in the photo shoot may be determined based on the settings configured in advance by the user who is organizing the event.

[0162] The remote shooting management server 1 may recognize the faces of users appearing in the live view video transmitted from each smartphone 2, and determine the number of users shooting together based on the number of recognized faces. In this case, for example, the game control unit 131, which functions as a user management unit, analyzes the live view video and performs face recognition.

[0163] Figure 19 shows an example of a live view image display.

[0164] As explained with reference to Figure 7, etc., there may be multiple users using a single smartphone 2. In this case, a larger display area is set to accommodate the larger number of people than the display area for the smaller number of people.

[0165] In the example shown in Figure 19, users A and B, who appear in the live view video captured by one smartphone 2, are displayed in the left display area A1, while user C, who appears in the live view video captured by the other smartphone 2, is displayed in the right display area A2. For example, 2 / 3 of the entire screen area is allocated to display area A1, and the remaining 1 / 3 is allocated to display area A2.

[0166] Thus, the size of each display area that divides the live view image is determined according to the number of users displayed in each area.

[0167] As shown in Figure 20, the user may be able to move the boundaries of the display areas to adjust the size of each display area. In the example in Figure 20, the size of each display area can be adjusted by moving the boundary lines displayed on the live view image left or right with a finger.

[0168] Multiple layouts for the live view image, differing in the number of divisions and the size of the display area, may be provided, allowing the user to select from among the available layouts.

[0169] • Displaying boundary lines The boundary lines representing the display area may be displayed for a certain period of time from the start of the live view image display and disappear after that period has elapsed. A larger screen display makes it easier to see and increases the freedom of posing.

[0170] The user may be able to select whether to display or hide the borders that represent the boundaries of the display areas, as well as the border color, border thickness, and border density. Alternatively, each display area may be represented by a difference in background color instead of a line.

[0171] • Orientation of the video for live view Figure 21 shows another example of how the live view image can be displayed.

[0172] As shown in display area A1 of Figure 21, information that guides the image orientation (the user's orientation as seen in the live view video) may be displayed. In the example in Figure 21, a message guiding the user to the upward direction of the image is displayed along the upper edge of display area A1. In addition to visual information such as messages and symbols, audio guidance may also be provided to guide the user to the image orientation.

[0173] Smartphone 2, used for remote shooting, is a device whose orientation can be easily changed. Therefore, as shown in the upper part of Figure 22, depending on the orientation of the smartphone 2 used by user A, user A's orientation may differ from that of user B.

[0174] By providing information that guides the orientation of the image, the user can adjust the image orientation by changing the orientation of the device or rotating the image. For example, if user A rotates the image, the image of user A will be displayed rotated, as shown at the end of the arrow in Figure 22.

[0175] The system may recognize the orientation of faces in an image and, if the faces are not facing the same direction, provide guidance on the image orientation. Alternatively, guidance on image orientation may only be provided to users whose faces are not facing the same direction as other users.

[0176] Image rotation operations can be restricted to the host user only, or to each individual user. If each user can rotate an image, then while one user is making adjustments, other users cannot. This prevents situations where one user accidentally adjusts the orientation of an image that doesn't need adjusting while another user is working on it.

[0177] • Enlarge image display When a display area is selected, the image displayed in that area may be enlarged. This allows the user to check their appearance in detail before taking a picture.

[0178] • Display of additional information You may also display example pose images along with the live view image.

[0179] The user may be given the option to display pose images before taking a photo. If pose images are displayed, the user may be given the option to select which pose they want to use. The image used as the pose image may be one saved on the user's smartphone 2.

[0180] Images saved by other applications may be used as pose images. For example, images posted on a community site can be used as pose images. A website for posting images taken with the photo sticker creation device described later can be used as the community site. For example, the community site may have functions for searching images by the model of the photo sticker creation device used to take the image, by pose, and by keyword. The community site may also have functions for displaying image rankings, featured images, and adding images to favorites.

[0181] Information that serves as a guide for aligning the face's position and size may be displayed.

[0182] Figure 23 shows an example of how the guide is displayed.

[0183] In the example shown in Figure 23A, vertically elongated elliptical guide images G1 and G2, which serve as reference information for aligning the position and size of faces, are displayed in display area A1 and display area A2, respectively. The live view image shown in Figure 23A is the image displayed when two users, User A and User B, are taking pictures, as explained with reference to Figure 17.

[0184] User A, preparing to take a photo while looking at the display in display area A1, adjusts the distance to smartphone 2 and the relative position of smartphone 2 to align their face with guide image G1. User B, preparing to take a photo while looking at the display in display area A2, adjusts the distance to smartphone 2 and the relative position of smartphone 2 to align their face with guide image G2.

[0185] In the example shown in Figure 23B, guide images G1-1 and G1-2 are displayed in display area A1, and guide image G2 is displayed in display area A2. The live view image shown in Figure 23B is the image that is displayed when three people are taking pictures, for example, as explained with reference to Figure 19, with users A and B using one smartphone 2 and user C using another smartphone 2.

[0186] Users A and B, who are preparing to take a picture while looking at the display in display area A1, adjust the distance to smartphone 2 and their relative positions to smartphone 2 so that their faces are aligned with guide images G1-1 and G1-2, respectively. User C, who is preparing to take a picture while looking at the display in display area A2, also adjusts the distance to smartphone 2 and their relative positions to smartphone 2 so that their face is aligned with guide image G2.

[0187] In this way, a number of guide images corresponding to the number of people visible in each display area are shown.

[0188] The guide image may be composed of a rectangle, a straight line, or other shape, rather than an ellipse.

[0189] The guide images displayed in each display area may be the same size or different sizes. The user may also be able to select the size of each guide image.

[0190] <Provided image> While each user who performs remote shooting is provided with a composite image (a still image), other images may be provided instead.

[0191] Figure 24 shows an example of a provided image.

[0192] As shown in Figure 24A, a video image P52 may be provided along with the composite image P51. The video image P52 is, for example, a video image generated by compositing video images for live view.

[0193] As shown in Figure 24B, individual images P53 may be provided along with the composite image P51. Individual images P53 are still images generated based on the composite image. If the users who performed the remote shooting together are user A and user B, for example, an individual image P53 of user A is provided to user A along with the composite image P51, and an individual image P53 of user B is provided to user B along with the composite image P51.

[0194] In this way, the same image may be provided to each user who performs remote shooting, or different images may be provided.

[0195] <Image adjustment> The composite image provided by remote shooting may be made adjustable after the remote shooting is complete.

[0196] • Position adjustment Figure 25 shows an example of placement.

[0197] For example, if a composite image is taken showing three users standing side-by-side, as explained with reference to Figure 18, the position of each user can be moved to adjust them to the desired position, as shown in Figure 25.

[0198] In the example shown in Figure 25, User B is placed on top of User A and User C. The layering can be adjusted by selecting the layer of each user in this way.

[0199] In addition to adjusting the position, it would also be good to allow adjustment of the size, such as enlarging or shrinking it.

[0200] When adjusting the position, guiding information may be displayed. For example, grid lines or crosshairs could be displayed as guiding information.

[0201] • Adjusting image quality Depending on the specifications of the camera 306 installed in the smartphone 2 and the location of remote shooting, the image quality, brightness, and color of the captured image may vary. Therefore, it may be possible to accept user adjustments to the image quality and perform calibration manually or automatically. For example, adjustments may be made to average image quality and brightness, to adjust to the brighter side, or to adjust to the best image quality.

[0202] It may be possible to use functions similar to those found in displays (monitors), printers, image scanners, digital cameras, etc., to compensate for differences in color reproduction characteristics between devices, individual variations, and changes over time.

[0203] • Image processing such as retouching and makeup Retouching is image processing that adjusts things like eye size and skin tone. Makeup is image processing that makes an image appear as if makeup is being applied. When using retouching or makeup in image processing, it may be helpful to allow comparison with the original image before processing.

[0204] • Content synthesis The composite image may be made editable using content. Examples of content include background images, stamp images, and pen images. A background image is an image composited onto the background layer of a person's layer. A stamp image is an image with a predetermined design that is composited onto the composite image at a position specified by a finger or other object. A pen image is an image that is composited along the trajectory of movement by a finger or other object.

[0205] Image processing using other content such as veils, auras, and filters may be applied to the composite image. Veils, auras, and filters are image processing techniques that adjust color and other aspects.

[0206] The content used for image processing may be selected before or during remote shooting, rather than after. Content selected during remote shooting will be reflected in the live view image display, etc. Content selected before or during remote shooting may also be modified after remote shooting, or its composite position may be changed.

[0207] <Other> • Overlay A feature that allows users to overlay still images from one user onto another user's still image may be implemented. In this case, the still images from one user and the other user may not be taken at the same time, but at different times.

[0208] The overlay shooting function may be made available only when the number of users performing remote shooting together exceeds a threshold set.

[0209] • One-to-many remote shooting It may be possible to allow one user to remotely capture images of multiple other users simultaneously. In the sense that one user captures images of multiple users, this type of remote capture is one-to-many remote capture.

[0210] Figure 26 shows an example of a composite image when performing one-to-many remote imaging.

[0211] In the example shown in Figure 26, User A performs remote imaging together with User B, and also with User C. The former remote imaging is between Users A and B, and the latter remote imaging is between Users A and C. The remote imaging between Users A and B and between Users A and C are performed simultaneously.

[0212] Images of user A taken with smartphone 2A, images of user B taken with smartphone 2B, and images of user C taken with smartphone 2C are each transmitted to the remote shooting management server 1 and used to generate live view images and composite images.

[0213] For example, based on the live view video captured on smartphone 2A and the live view video captured on smartphone 2B, a live view image for remote shooting between users A and B is displayed on both smartphone 2A and smartphone 2B.

[0214] Furthermore, based on the live view video captured on smartphone 2A and the live view video captured on smartphone 2C, a live view image for remote shooting between users A and C will be displayed on smartphone 2A and smartphone 2C, respectively.

[0215] When a still image is captured on each of the smartphones 2A through 2C, the remote shooting management server 1 generates a composite image for each remote shooting session and provides it to the user participating in each remote shooting session.

[0216] In other words, as shown in Figure 26, a composite image P61 is generated based on the composite image captured by smartphone 2A and the composite image captured by smartphone 2B, and is provided to user A and user B as a composite image for remote shooting between users A and B.

[0217] Furthermore, a composite image P62, generated based on the composite image captured by smartphone 2A and the composite image captured by smartphone 2C, is generated as a composite image for remote shooting between users A and C, and is provided to user A and user C.

[0218] In this way, one user may participate in multiple remote shoots simultaneously. For example, a celebrity could be one user and participate in a remote shoot with several other users. The image of the celebrity could be switched to an image of another user after a predetermined amount of time has elapsed.

[0219] • Examples of photographic equipment While remote photography is described as being performed using a smartphone, it may also be performed using other photographic devices equipped with cameras, such as photo sticker creation machines installed in arcades. A photo sticker creation machine is a game machine that photographs users who have paid for the game and provides sticker paper with the image of the user printed on it.

[0220] The composite image generated using the remote shooting application can be printed using the printer of the photo sticker creation device.

[0221] • Program example The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on a computer that is built into dedicated hardware, or on a general-purpose personal computer.

[0222] The programs to be installed are provided on removable media such as optical discs (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or semiconductor memory. They may also be provided via wired or wireless transmission media such as local area networks, the internet, or digital broadcasting. The programs can be pre-installed.

[0223] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when they are called.

[0224] In this specification, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules in one enclosure, are both considered systems.

[0225] The effects described herein are illustrative and not limited to those described herein, and other effects may also occur.

[0226] The embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0227] For example, this technology can be configured as cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network.

[0228] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

[0229] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices. [Explanation of symbols]

[0230] 1 Remote shooting management server, 2 Smartphone, 121 Information processing unit, 131 Game control unit, 132 Configuration information DB, 133 Image acquisition unit, 134 Image processing unit, 301 Control unit, 301A Application execution unit, 311 Shooting control unit, 312 Output control unit, 313 Remote shooting setting unit

Claims

1. A shooting control unit that causes shooting to be performed on a first mobile terminal used by a first user to set up remote shooting, and a second mobile terminal used by a second user, An image processing unit extracts the human region from images captured by the first mobile terminal and the second mobile terminal, and displays a live view image, which is a moving image generated based on the extracted human region image, on the first mobile terminal and the second mobile terminal. Equipped with, The shooting control unit, in response to the shooting button displayed only on the screen of the first mobile terminal being pressed after the start of the display of the live view image, displays a countdown to the same shooting timing on both the first mobile terminal and the second mobile terminal. The image processing unit generates a composite image in which the image of the first user's person region and the image of the second user's person region are arranged, based on the images captured at the same shooting timing on both the first and second mobile terminals, and displays the composite image on both the first and second mobile terminals. Information processing device.

2. The device further comprises a communication unit that transmits and outputs the voice of the first user transmitted from the first mobile terminal to the second mobile terminal, and transmits and outputs the voice of the second user transmitted from the second mobile terminal to the first mobile terminal, The communication unit causes the first user's voice to be output on the second mobile terminal and the second mobile terminal, and the second user's voice to be output on the first mobile terminal, from the time when shooting is started on the first mobile terminal until the composite image is displayed. The information processing apparatus according to claim 1.

3. Further comprising a receiving unit that receives the selection of content of a predetermined design, including a background image to be composited onto the background of a person area. The information processing apparatus according to claim 1 or 2.

4. The receiving unit receives the selection of content, which is performed after the start of image capture for the live view image on each of the first and second mobile terminals. The image processing unit reflects the content selected after the start of shooting onto the live view image. The information processing apparatus according to claim 3.

5. The image processing unit modifies the content included in the composite image generated based on the images and content captured at the same shooting timing after the shooting. The information processing apparatus according to claim 3.

6. The image processing unit generates and displays the live view image in which the images of the person region extracted from the image captured by the first mobile terminal and the images of the person region extracted from the image captured by the second mobile terminal are arranged in areas of different sizes. The information processing apparatus according to any one of claims 1 to 5.

7. The image processing unit adjusts the size of the area according to the number of people placed in each of the areas, or according to the operation of any of the users. The information processing apparatus according to claim 6.

8. The image processing unit rotates the orientation of the image of the person region extracted from the image captured by the first mobile terminal and the image of the person region extracted from the image captured by the second mobile terminal on the live view image in accordance with the operation of the first user. The information processing apparatus according to any one of claims 1 to 7.

9. The image processing unit displays information that serves as a guide for image orientation only on the screen of the mobile terminal among the first and second mobile terminals whose image orientation of the person area differs from the reference orientation. The information processing apparatus according to any one of claims 1 to 8.

10. A shooting control unit that causes shooting to be performed on a first mobile terminal used by a first user to set up remote shooting, and a second mobile terminal used by a second user, An image processing unit extracts the human region from images captured by the first mobile terminal and the second mobile terminal, and displays a live view image, which is a moving image generated based on the extracted human region image, on the first mobile terminal and the second mobile terminal. An information processing device equipped with, When the shutter button displayed only on the screen of the first mobile device is pressed after the start of the display of the live view image, a countdown to the same shooting timing is displayed on both the first and second mobile devices. Based on the images captured at the same time on both the first and second mobile terminals, a composite image is generated in which the image of the first user's person region and the image of the second user's person region are arranged, and this composite image is displayed on both the first and second mobile terminals. Information processing methods.

11. A shooting control unit that causes shooting to be performed on a first mobile terminal used by a first user to set up remote shooting, and a second mobile terminal used by a second user, An image processing unit extracts the human region from images captured by the first mobile terminal and the second mobile terminal, and displays a live view image, which is a moving image generated based on the extracted human region image, on the first mobile terminal and the second mobile terminal. Equipped with a computer, When the shutter button displayed only on the screen of the first mobile device is pressed after the start of the display of the live view image, a countdown to the same shooting timing is displayed on both the first and second mobile devices. Based on the images captured at the same time on both the first and second mobile terminals, a composite image is generated in which the image of the first user's person region and the image of the second user's person region are arranged, and this composite image is displayed on both the first and second mobile terminals. A program to execute a process.

Images