Information processing device, information processing method, and program

Abstract

To enable a plurality of remote users to easily take images of themselves in real time.SOLUTION: An information processing device according to an embodiment of the present technology causes a first portable terminal used by a first user and a second portable terminal used by a second user to capture images, extracts person areas from the images captured by the first portable terminal and the second portable terminal, displays live view images, which are moving images generated on the basis of the extracted person area images, on the first portable terminal and the second portable terminal, and displays information that serves as a guide for the orientation of the images together with the live view images when the orientations of the faces are not aligned. This technology can be applied to a server on the Internet.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable multiple users in remote locations to easily capture images of themselves in real time. [Background technology]

[0002] Collage photography is a technique that combines multiple captured images to create a single image. Collage photography makes it possible to create highly designed images by combining multiple captured images.

[0003] Patent Document 1 discloses a technology for generating a collage image by arranging a selected reference image and images highly related to the reference image in the layout frame of a template. The collage image is generated by extracting highly related images based on the date and time of their capture and arranging them in order of priority. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-46540 Summary of the Invention [Problem to be solved by the invention]

[0005] According to the technology described in Patent Document 1, it is possible to generate a collage image of multiple people by combining photographed images in which each person appears, but since the collage image is generated based on images taken in the past, it is not possible to enjoy collage photography.

[0006] If a collage image were to be generated based on images taken by each user in real time, it would be difficult to take photos when the users are far apart because they cannot see each other's state. Each user would not know what angle of view they should have in the image, or what pose they should strike in the image.

[0007] The present technology has been developed in light of these circumstances, and enables multiple users in remote locations to easily capture images of themselves in real time. [Means for solving the problem]

[0008] An information processing device according to one aspect of the present technology includes an image processing unit that causes each of a plurality of mobile terminals to display a live view image generated based on a plurality of moving images that are captured in real time on each of the mobile terminals and that show a user using the mobile terminal, and an image capture control unit that causes each of the plurality of mobile terminals to capture a still image at the same timing after the start of display of the live view image, and the image processing unit combines the still images that at least show the user captured on each of the plurality of mobile terminals into a single image, and outputs the combined image on each of the mobile terminals.

[0009] A program according to another aspect of the present technology executes a process of capturing a moving image of a user using a mobile terminal, displaying a live view image generated based on the moving image of the user and a moving image of another user captured in real time on another mobile terminal used by the other user, capturing a still image at the same timing as the capturing of the image on the other mobile terminal after display of the live view image begins, and displaying a composite image generated by combining the still image in which at least the user is captured and the still image in which at least the other user is captured into a single image.

[0010] In one aspect of the present technology, a live view image generated based on a plurality of moving images captured in real time on each of a plurality of mobile terminals and showing a user using the mobile terminal is displayed on each mobile terminal, and still images are captured on each of the plurality of mobile terminals at the same timing after the start of display of the live view image, and the still images captured on each of the plurality of mobile terminals and showing at least the user are combined into a single image, and the combined image is output on each of the mobile terminals.

[0011] In another aspect of the present technology, a video image showing a user using a mobile terminal is captured, a live view image is generated based on the video image showing the user and a video image showing another user captured in real time on another mobile terminal used by the other user, and a still image is captured at the same timing as the capture on the other mobile terminal after display of the live view image begins, and a composite image is displayed by combining the still image showing at least the user and the still image showing at least the other user into a single image. [Effects of the Invention]

[0012] According to this technology, multiple users in remote locations can easily capture images of themselves in real time. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a diagram illustrating an example of the configuration of a remote photography system according to an embodiment of the present technology. [Figure 2] FIG. 10 is a diagram showing an example of a selfie. [Figure 3] FIG. 10 is a diagram showing how a captured image is transmitted in remote photography. [Figure 4] FIG. 10 is a diagram illustrating an example of a composite image. [Figure 5] FIG. 10 is a diagram showing how a composite image is transmitted in remote photography. [Figure 6] FIG. 10 is a diagram illustrating an example of users performing remote photography together. [Figure 7] FIG. 10 is a diagram illustrating another example of users performing remote photography together. [Figure 8] FIG. 10 is a diagram illustrating an example of a composite image. [Figure 9] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a remote photography management server. [Figure 10] FIG. 2 is a block diagram illustrating an example of a functional configuration of a remote photography management server. [Figure 11] FIG. 10 is a diagram illustrating an example of setting information. [Figure 12] FIG. 1 is a block diagram illustrating an example of the configuration of a smartphone. [Figure 13] FIG. 2 is a block diagram illustrating an example of a functional configuration of a smartphone. [Figure 14] 10 is a flowchart illustrating a series of processes for remote photography. [Figure 15] 15 is a flowchart illustrating a series of processes for remote photography, continuing from FIG. 14. [Figure 16] FIG. 10 is a diagram showing a display example of a live view image. [Figure 17] FIG. 10 is a diagram showing a display example of a live view image. [Figure 18] FIG. 10 is a diagram showing a display example of a live view image. [Figure 19] FIG. 10 is a diagram showing a display example of a live view image. [Figure 20] FIG. 10 is a diagram showing a display example of a live view image. [Figure 21] FIG. 10 is a diagram showing a display example of a live view image. [Figure 22] FIG. 10 is a diagram showing a display example of a live view image. [Figure 23] FIG. 10 is a diagram showing an example of a guide display. [Figure 24] FIG. 10 is a diagram showing an example of a provided image. [Figure 25] FIG. 10 is a diagram illustrating an example of an arrangement position. [Figure 26] FIG. 10 is a diagram showing an example of a composite image when one-to-many remote photography is performed. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present technology will be described in the following order. 1. Remote photography system 2. User invitation example 3. Configuration of each device 4. Remote shooting process 5. Displaying Live View images 6. Provided Images 7. Adjusting the image 8.Other

[0015] <Remote photography system> FIG. 1 is a diagram illustrating an example of the configuration of a remote photography system according to an embodiment of the present technology.

[0016] The remote photography system in Fig. 1 is configured by connecting multiple smartphones 2 to a remote photography management server 1 via the Internet 11. In the example of Fig. 1, a smartphone 2A used by user A and a smartphone 2B used by user B are shown. User A and user B are, for example, friends who are far away from each other.

[0017] Other devices equipped with a communication module and a camera, such as a tablet terminal or a PC, may be used in place of the smartphone 2. The camera may be a built-in camera or an external camera.

[0018] A remote photography app is installed on the smartphone 2. The remote photography app is an application program that runs on the operating system (OS) installed on the smartphone 2.

[0019] A remote photography app is an application that allows multiple users in different locations to take pictures together. Taking pictures using a remote photography app is called remote photography.

[0020] Before starting remote photography, as shown by the white arrow in Figure 1, the user who will be the organizer of the remote photography event configures the settings for remote photography on the remote photography app and invites users who will participate in the photography. In the example in Figure 1, User A is the organizer. By accepting User A's invitation, User A and User B will take remote photography together.

[0021] Remote photography proceeds in a predetermined sequence, for example, following guidance from the remote photography app. Each user can enjoy remote photography like a game.

[0022] Each user points the camera of the smartphone 2 at themselves and takes a selfie while looking at the screen of the remote photography app, as shown in Figure 2. The front of the housing of the smartphone 2 is provided with a camera and a display such as an LCD.

[0023] The screen of the remote photography app displays a live view image, which is a moving image of User A and User B taking remote photos together. User A and User B can check their expressions and decide on a pose by looking at the live view image displayed on their respective smartphones 2. During remote photography, each user can also talk to the other.

[0024] The smartphone 2A photographs user A and the smartphone 2B photograph user B at the same time (in real time) under the control of, for example, the remote photography management server 1. The smartphones 2 perform remote photography in synchronization with each other.

[0025] A captured image P1, which is a still image taken with smartphone 2A and shows user A as the subject, and a captured image P2, which is a still image taken with smartphone 2B and shows user B as the subject, are each transmitted to the remote photography management server 1 as indicated by the white arrows in Figure 3.

[0026] The remote photography management server 1 composites the photographed image P1 and the photographed image P2 into one image to generate a composite image P11, as shown in Fig. 4. In the example of Fig. 4, user A is positioned on the left side and user B is positioned on the right side to generate the composite image P11.

[0027] The photographed images P1 and P2 are combined by combining at least an image of a person area extracted from the photographed images P1 and P2 by chromakey processing, etc. A background image is appropriately combined on a layer below the person area.

[0028] In this way, the captured image P1 and the captured image P2 are still images used to generate the composite image P11. Hereinafter, the captured images captured by each smartphone 2 and used to generate the composite image will be referred to as captured images for synthesis, as appropriate.

[0029] In the example of FIG. 4, dashed lines representing the boundaries of the areas allocated to each user are shown in the center of the composite image P11, but they are not actually displayed on the composite image P11.

[0030] The composite image P11 generated by the remote photography management server 1 is sent to the smartphone 2A and the smartphone 2B, respectively, as indicated by the white arrows in Fig. 5. User A and User B can obtain a still image of themselves together, even though they are far apart.

[0031] In one remote shooting session, the above-described shooting is performed multiple times, for example. User A and User B can enjoy shooting together while having a conversation, and can obtain multiple composite images.

[0032] In this way, the remote photography management server 1 is an information processing device that provides a composite image obtained by combining still images photographed at the same timing with a user who has taken remote photography together.

[0033] <Example of user invitation> FIG. 6 is a diagram showing an example of users performing remote photography together.

[0034] In the example of FIG. 6, user A is the organizer and has invited users B and C to participate in remote photography.

[0035] After the remote photographing is started, the smartphone 2A photographs the user A as a subject, and the smartphone 2B photographs the user B as a subject. Furthermore, the smartphone 2C takes an image of the user C as a subject. The smartphones 2A to 2C each take an image at the same time.

[0036] The three images to be combined, which are taken by the smartphones 2A to 2C, are transmitted to the remote photography management server 1 and used to generate a composite image.

[0037] The composite image in which the three users A to C appear is transmitted to the smartphones 2A to 2C, respectively. Users A to C can obtain a composite image in which the three users appear together by performing remote shooting together.

[0038] In this way, remote photography can be performed by three people. The number of users who perform remote photography together is arbitrary, such as four or more people.

[0039] FIG. 7 is a diagram showing another example of users performing remote photography together.

[0040] 7, user A and user B are in the same place and take a photo together using smartphone 2A, which is user A's smartphone. User C is a user who is in a place separate from users A and B.

[0041] In this way, a user can take a remote photo of not only themselves but also their friends in the field of view. In the example of Figure 7, user A, who is with user B, is the organizer and has invited user C to take a remote photo.

[0042] After the remote shooting has started, the smartphone 2A shoots images of users A and B as subjects. The smartphone 2C shoots images of user C as a subject. The smartphones 2A and 2C shoot images at the same time.

[0043] The two images to be combined, which are taken by the smartphone 2A and the smartphone 2C, are transmitted to the remote photography management server 1 and used to generate a composite image.

[0044] The composite image showing the three users A to C is sent to smartphone 2A and smartphone 2C, respectively. For example, user A sends the composite image to user B using a social networking service or the like, allowing users A to C to obtain a composite image showing the three users together.

[0045] In this way, a single composite image containing multiple users may be captured using one smartphone 2. The user performing remote photography using the smartphone 2 may be a single user, the owner of the smartphone 2, or may be multiple users, including the owner and other users who are close to the owner and are included in the angle of view of the same image.

[0046] FIG. 8 is a diagram showing an example of a composite image.

[0047] The composite image shown in FIG. 8A is a composite image generated when remote photography is performed on each of the smartphones 2A to 2C as described with reference to FIG.

[0048] For example, by combining a composite image taken with smartphone 2A showing user A, a composite image taken with smartphone 2B showing user B, and a composite image taken with smartphone 2C showing user C into a single image, a composite image showing three users A to C as shown in Figure 8A is generated.

[0049] The composite image shown in FIG. 8B is a composite image generated when remote shooting is performed by each of the smartphone 2A and the smartphone 2C, as described with reference to FIG.

[0050] A captured image for compositing, which is taken by the smartphone 2A and shows users A and B, and a captured image for compositing, which is taken by the smartphone 2C and shows user C, are combined into a single image to generate a composite image showing the three users A to C, as shown in Fig. 8B. In the example of Fig. 8B, a larger area is allocated to the captured image for compositing, which is taken by the smartphone 2A and shows users A and B, than to the captured image for compositing, which is taken by the smartphone 2C and shows user C, and the images are combined.

[0051] <Configuration of each device> Remote photography management server 1 configuration FIG. 9 is a block diagram showing an example of the hardware configuration of the remote photography management server 1. As shown in FIG.

[0052] 9, the remote photography management server 1 is configured by a computer. The remote photography management server 1 may be configured by multiple computers. When the remote photography management server 1 is configured by multiple computers, the computers work together to realize various processes for providing a composite image.

[0053] A CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103 are interconnected by a bus 104.

[0054] An input / output interface 105 is further connected to the bus 104. An input unit 106, an output unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

[0055] The input unit 106 is made up of a keyboard, a mouse, etc. The output unit 107 is made up of a display, etc.

[0056] The storage unit 108 is configured by a hard disk, a non-volatile memory, etc. The storage unit 108 stores various information used to provide a composite image, such as programs executed by the CPU 101 and setting information for remote shooting.

[0057] The communication unit 109 is an interface to the Internet 11. For example, the communication unit 109 communicates with the smartphone 2 and receives the captured images for compositing transmitted from the smartphone 2. The communication unit 109 also transmits a composite image generated based on the captured images for compositing to each smartphone 2.

[0058] The drive 110 controls writing of data to and reading of data from the removable medium 111 .

[0059] FIG. 10 is a block diagram showing an example of the functional configuration of the remote photography management server 1. As shown in FIG.

[0060] 10, an information processing unit 121 is implemented in the remote photography management server 1. The information processing unit 121 includes a game control unit 131, a setting information DB 132, an image acquisition unit 133, and an image processing unit .

[0061] The game control unit 131 controls the communication unit 109 to communicate with the smartphone 2 used by the user who will be the organizer of the remote photography, and accepts settings related to the remote photography. Various settings related to the remote photography are made using a remote photography app. The game control unit 131 stores and manages setting information indicating the contents of the settings related to the remote photography in the setting information DB 132.

[0062] FIG. 11 is a diagram illustrating an example of the setting information.

[0063] As shown in Fig. 11, the setting information includes information on the shooting date and time, the host user, the invited users, the number of users, the appearance settings, the game time, and the URL. Setting information including each piece of information is generated and managed for each remote shooting session. It is not necessary for all of the information shown in Fig. 11 to be included in the setting information, but at least some of the information may be included.

[0064] The shooting date and time indicates the start time of the remote shooting. When the start time indicated by the shooting date and time arrives, each user participating in the remote shooting starts a remote shooting app and starts remote shooting. If the remote shooting app is started before the start time indicated by the shooting date and time, shooting is permitted when the start time arrives.

[0065] The host user represents a user who will be the host of the remote photography. The game control unit 131 identifies the host user and records, for example, the ID of the host user as information about the host user.

[0066] An invited user refers to a user invited by the organizer user. A user is invited, for example, by selecting from among users who have been registered in advance as friends of the organizer user. A user invited by the organizer user may also be able to invite other users. For example, an invited user can invite other users by obtaining the organizer user's consent.

[0067] The game control unit 131 identifies users invited by the host user and records, for example, the IDs of the invited users as information about the inviting users. If multiple users are invited, the user IDs of each user are recorded. The game control unit 131 functions as a user management unit that identifies and manages users who will participate in remote photography together.

[0068] The number of users indicates the number of users who will be performing remote photography. For example, if an invitation is made as described with reference to Fig. 7, information indicating that two users will be performing photography using smartphone 2A and one user will be performing photography using smartphone 2C is recorded as information on the number of users. In this way, information on the number of users who will be performing remote photography using smartphone 2 is recorded for each smartphone 2.

[0069] The appearance settings represent settings related to the image quality of the image, such as color, brightness, etc. As the appearance settings, different settings may be used for each smartphone 2, or the same settings may be used for each smartphone 2.

[0070] The camera may be linked with another app such as a camera app so that the appearance settings selected in that app are used during remote shooting. The appearance settings selected in advance using the other app may be used, or the user may be able to select whether or not to use the appearance settings of the other app while viewing the live view image.

[0071] The appearance settings of other apps may be applied to all users participating in remote photography, or may be applied to each user individually. When applied to each user individually, for example, it is possible to use the appearance setting α of the camera app for the appearance of user A, and the appearance setting β of the camera app for the appearance of user B.

[0072] It may be possible to select which application's image settings to apply for each user. For example, if multiple camera applications are available, it is possible to use the image settings of one camera application for User A's image and the image settings of another camera application for User B's image.

[0073] The game time represents the time it takes to perform one remote photographing session. During the game time, for example, multiple photographing sessions (generation of a composite image) are performed.

[0074] The URL is identification information assigned to remote photography. When the host user configures settings related to remote photography, the URL is issued. The host user invites users to play the remote photography game together by notifying them of the URL issued by the remote photography management server 1. The remote photography management server 1 may notify the inviting users of the URL.

[0075] A URL may be issued for each remote shooting, or a different ID and password may be issued for each remote shooting while keeping the same URL. The ID and password may be set by the user. The ID may be set by the user and the password may be automatically generated. The ID and password may be automatically generated.

[0076] A two-dimensional code may be issued instead of a URL as identification information for remote photography. An invitation code consisting of a predetermined number of digits, letters, etc. may be issued as identification information for remote photography. The invitation code is issued when the organizer's user launches the remote photography app and configures settings related to remote photography.

[0077] Users who can be invited may be limited to specific users (friend function). For example, only users who are set as friends may be invited. Users who the organizer user follows on a specific service, or users who follow the organizer user, are set as friends who can be invited. Also, users who are set as friends on the SNS service used by the organizer user are set as friends who can be invited. This prevents unintended users from participating in remote photography, improving security.

[0078] A URL or two-dimensional code including the contents of the setting information may be generated.

[0079] In this way, settings for remote photography between any two users are made as preparations for remote photography. The setting information managed by the game control unit 131 is read from the setting information DB 132 at the start of remote photography, and is supplied to the image processing unit 134.

[0080] The game control unit 131 also communicates with a plurality of smartphones 2 used by users who will be performing remote photography together, and controls a series of processes for remote photography in each smartphone 2.

[0081] That is, the game control unit 131 cooperates with the remote photography apps of the respective smartphones 2 that perform remote photography together, and synchronizes operations such as photography on the respective smartphones 2. The game control unit 131 also functions as a photography control unit that controls the photography of moving images and still images.

[0082] Instead of capturing both moving images and still images under the control of the game control unit 131, at least one of the images may be captured by a remote photography app without the control of the game control unit 131.

[0083] The image acquisition unit 133 controls the communication unit 109 to acquire images that are taken by each of the multiple smartphones 2 used by users who are performing remote photography together and transmitted from each smartphone 2.

[0084] Before capturing a still image, a live view moving image, which is a moving image to be used for generating a live view image, is transmitted from each smartphone 2. Furthermore, when a still image is captured, a captured image to be synthesized, which is a still image to be used for generating a synthesized image, is transmitted from each smartphone 2. The image acquired by the image acquisition unit 133 is supplied to the image processing unit 134.

[0085] The image processing unit 134 generates a live view image based on the live view moving image supplied from the image acquisition unit 133.

[0086] The live view images are generated in accordance with the contents of the setting information supplied from the game control unit 131. For example, the layout pattern of users appearing in each live view video in the live view images is determined based on the information about the number of people included in the setting information. Also, image processing is performed on the live view video based on the appearance information included in the setting information.

[0087] Furthermore, the image processing unit 134 generates a composite image based on the captured images for synthesis supplied from the image acquisition unit 133. As with the generation of live view images, the generation of a composite image is also performed in accordance with the contents of the setting information supplied from the game control unit 131. A composite image is generated in which each user appears in the same arrangement as in the live view images.

[0088] The image processing unit 134 controls the communication unit 109 to transmit the live view image and the composite image to the smartphones 2 and output them on each smartphone 2. The image processing unit 134 controls the output of images on the smartphones 2, such as displaying the live view image and the composite image on the smartphones 2 used by each user who is performing remote shooting together, or storing the composite image in memory.

[0089] Smartphone 2 configuration FIG. 12 is a block diagram showing an example of the configuration of the smartphone 2.

[0090] As shown in FIG. 12, the smartphone 2 includes a control unit 301, a memory 302, a communication unit 303, a display 304, an operation unit 305, a camera 306, a speaker 307, and a microphone 308.

[0091] The control unit 301 is configured with a CPU, a ROM, a RAM, etc. The control unit 301 controls the overall operation of the smartphone 2 by executing a predetermined program.

[0092] An application execution unit 301A is implemented in the control unit 301. A remote photography application is executed by the application execution unit 301A, thereby implementing a series of processes for remote photography.

[0093] The memory 302 is configured by a flash memory, etc. The memory 302 stores various information such as the composite image transmitted from the remote photography management server 1 and the application executed by the control unit 301.

[0094] The communication unit 303 communicates with the remote photography management server 1 via the Internet 11. The communication unit 303 receives information transmitted from the remote photography management server 1 and outputs it to the control unit 301.

[0095] The display 304 is configured by an organic EL display, an LCD, etc. The display 304 displays various screens such as the screen of the remote photography application.

[0096] The operation unit 305 is configured with various buttons and a touch panel provided over the display 304. The operation unit 305 outputs to the control unit 301 a signal indicating the content of a user's operation.

[0097] The camera 306 captures moving images and still images under the control of the control unit 301. The images captured by the camera 306 are supplied to the control unit 301.

[0098] The speaker 307 outputs the voice of the person participating in the remote photographing based on the voice data supplied from the control unit 301.

[0099] The microphone 308 detects the user's voice and outputs the voice data to the control unit 301. The user's voice is transmitted via the remote photography management server 1 to the smartphone 2 of the other person with whom the user is performing remote photography.

[0100] FIG. 13 is a block diagram showing an example of the functional configuration of the smartphone 2.

[0101] As shown in FIG. 13, in the smartphone 2, a remote photography application is executed by an application execution unit 301A, whereby a photography control unit 311, an output control unit 312, and a remote photography setting unit 313 are realized.

[0102] The photography control unit 311 controls the camera 306 to capture moving images and still images. The photography control unit 311 transmits the captured moving images as live view moving images to the remote photography management server 1. The photography control unit 311 also transmits the captured still images as captured images to be synthesized to the remote photography management server 1.

[0103] The output control unit 312 displays the screen of the remote photography application on the display 304 based on the information transmitted from the remote photography management server 1. For example, the output control unit 312 displays the live view image transmitted from the remote photography management server 1 on the display 304. The output control unit 312 also displays the composite image transmitted from the remote photography management server 1 on the display 304 as the image of the photography result.

[0104] The output control unit 312 outputs the composite image transmitted from the remote photography management server 1 to the memory 302 for storage.

[0105] The remote photography setting unit 313 accepts user operations for settings related to remote photography, and transmits information indicating the contents of the settings to the remote photography management server 1. In the remote photography management server 1, the above-mentioned setting information is generated based on the contents accepted by the remote photography setting unit 313.

[0106] <Remote shooting process> A series of processes for remote photography will be described with reference to the flowcharts of FIGS.

[0107] 1, it is assumed that user A acts as the organizer and invites user B, and that user A and user B perform remote photography together. A series of processes is performed between the smartphone 2A used by user A, the smartphone 2B used by user B, and the remote photography management server 1.

[0108] When user A instructs the start of remote photography by specifying a previously issued URL, for example, the remote photography application of the smartphone 2A accesses the remote photography management server 1 via the Internet 11 in step S1.

[0109] Similarly, when the start of remote photography is instructed by user B, the remote photography application of the smartphone 2B accesses the remote photography management server 1 via the Internet 11 in step S21.

[0110] In steps S41 and S42, the game control unit 131 of the remote photography management server 1 accepts access from the smartphones 2A and 2B, respectively.

[0111] In step S2, the shooting control unit 311 of the smartphone 2A starts shooting a moving image for live view.

[0112] In step S3, the shooting control unit 311 of the smartphone 2A transmits the live view moving image to the remote shooting management server 1.

[0113] In step S22, the shooting control unit 311 of the smartphone 2B starts shooting a moving image for live view.

[0114] In step S23, the shooting control unit 311 of the smartphone 2B transmits the live view moving image to the remote shooting management server 1.

[0115] In steps S43 and S44, the image acquisition unit 133 of the remote photography management server 1 acquires the live view moving images transmitted from the smartphones 2A and 2B.

[0116] In step S45, the image processing unit 134 generates a live view image based on the acquired live view moving images.

[0117] In step S46, the image processing unit 134 transmits the live view image to each of the smartphones 2A and 2B.

[0118] In step S4, the output control unit 312 of the smartphone 2A receives the live view image transmitted from the remote photography management server 1.

[0119] In step S5, the output control unit 312 of the smartphone 2A causes the display 304 to display a live view image.

[0120] In step S24, the output control unit 312 of the smartphone 2B receives the live view image transmitted from the remote photography management server 1.

[0121] In step S25, the output control unit 312 of the smartphone 2B causes the display 304 to display a live view image.

[0122] FIG. 16 is a diagram showing a display example of a live view image.

[0123] 16, a live view image showing user A and user B, which has been generated based on a live view video captured by smartphone 2A and a live view video captured by smartphone 2B, is displayed on almost the entire screen. The person on the left is user A, and the person on the right is user B. The user may be able to change the position of the people when the live view image is displayed.

[0124] The live view image is constructed by, for example, arranging a live view moving image captured by smartphone 2A and a live view moving image captured by smartphone 2B in their respective display areas and combining them. At least an image of a person area extracted from each live view moving image is arranged in each display area.

[0125] A shooting button 401 is displayed superimposed on the live view image. Next to the shooting button 401, guidance is displayed to guide the user to press the shooting button 401 when preparations for shooting are complete. The guidance is output using audio as appropriate.

[0126] Such live view images are displayed in synchronization on the smartphones 2 A and 2 B. User A and User B look at the live view images and strike a pose while having a conversation or the like.

[0127] For example, when the user A presses the photographing button 401, the photographing control unit 311 of the smartphone 2A transmits, to the remote photographing management server 1, information indicating that the photographing button 401 has been pressed, in step S6 of FIG.

[0128] In step S47, the game control unit 131 of the remote photography management server 1 receives the information transmitted from the smartphone 2A.

[0129] In step S48, the game control unit 131 instructs the smartphones 2A and 2B to capture a first still image.

[0130] In step S7, the photography control unit 311 of the smartphone 2A receives the information transmitted from the remote photography management server 1, and in step S8, controls the camera 306 to take a still image.

[0131] On the screen of the remote photography app, for example, a countdown display starts, and a still image is captured when the countdown ends. Photography does not start immediately when the photography button 401 is pressed, but after a predetermined time has passed since the photography button 401 was pressed, the smartphones 2A and 2B synchronize and capture a still image at the same timing after the display of a live view image starts.

[0132] In step S9, the photography control unit 311 of the smartphone 2A transmits the still image obtained by photography to the remote photography management server 1 as a photographed image to be synthesized.

[0133] The smartphone 2B also captures a composite image in synchronization with the image capture by the smartphone 2A. That is, in step S26, the image capture control unit 311 of the smartphone 2B receives the information transmitted from the remote image capture management server 1, and in step S27, controls the camera 306 to capture a still image.

[0134] In step S28, the photography control unit 311 of the smartphone 2B transmits the still image obtained by photography to the remote photography management server 1 as a photographed image to be synthesized.

[0135] In steps S49 and S50, the image acquisition unit 133 of the remote photography management server 1 receives the photographed images to be combined transmitted from the smartphones 2A and 2B.

[0136] In step S51, the image processing unit 134 generates a composite image based on the captured images for synthesis.

[0137] In step S52, the image processing unit 134 transmits the generated composite image to each of the smartphones 2A and 2B.

[0138] In step S10, the output control unit 312 of the smartphone 2A receives the composite image transmitted from the remote photography management server 1.

[0139] In step S11, the output control unit 312 of the smartphone 2A causes the composite image to be displayed as a captured image on the display 304. The composite image is displayed on the display 304 in place of the live view image.

[0140] In step S29, the output control unit 312 of the smartphone 2B receives the composite image transmitted from the remote photography management server 1.

[0141] In step S30, the output control unit 312 of the smartphone 2A causes the composite image to be displayed as a captured image on the display 304. The composite image is displayed on the display 304 in place of the live view image.

[0142] In this way, the composite image of the photographed results is displayed synchronously on each of the smartphones 2A and 2B. User A and User B can enjoy a conversation while looking at the same composite image.

[0143] The composite image transmitted from the remote photography management server 1 is displayed on the display 304 and is also stored in, for example, the memory 302, so that the user can view it at any time by operating the remote photography application.

[0144] The above-described photographing is repeated the number of times set in the remote photographing settings, for example. After photographing is performed multiple times and multiple composite images are provided to the user, the process ends.

[0145] Through the above process, each user can obtain a still image of themselves together even if they are in different locations.

[0146] Although the user is required to press the capture button 401 for each still image capture, continuous capture may be performed after pressing the capture button 401 once. Continuous capture may be performed at predetermined time intervals, for example.

[0147] After continuous shooting has begun, the shooting may be paused, allowing the user to strike a pose or adjust their appearance.

[0148] It may be possible to allow only the organizer user to start shooting, rather than allowing any of the users taking photos together to start shooting. In this case, the shooting button 401 is displayed only on the screen of the smartphone 2 used by the organizer user.

[0149] Each user may be allowed to press the shooting button 401. This is a relief measure in case of a time lag.

[0150] <Live View image display> Here, the display of the live view image will be described in detail. The display of the live view image as shown below is realized, for example, by cooperation between the game control unit 131 (FIG. 10) of the remote photography management server 1 and the remote photography application (output control unit 312 of FIG. 13) of the smartphone 2.

[0151] -Setting the number of divisions FIG. 17 is a diagram showing a display example of a live view image.

[0152] 17, the display of the shooting button 401 and the like is omitted. The same applies to FIG. 18 and the like, which will be described later.

[0153] When two users are shooting together, a live view image is displayed divided into two left and right display areas A1 and A2, as shown in Fig. 17. In each display area, the users appearing in the live view moving images captured by the smartphone 2A and the smartphone 2B are displayed.

[0154] 17, when user A is displayed in display area A1 and user B is displayed in display area A2, the composite image obtained after capturing a still image will also have user A on the left and user B on the right. By displaying such a live view image, the user can capture the image while imagining the final result. The system may also be configured to accept changes to the user's position on the composite image so that the position differs from that on the live view image.

[0155] It may be possible to select whether or not to flip the live view image and the composite image horizontally. By displaying the image as if you were looking at your own reflection in a mirror, the user can reduce the sense of discomfort they may feel about their own reflection.

[0156] FIG. 18 is a diagram showing another display example of a live view image.

[0157] When there are three users taking pictures together, a live view image is displayed divided into three display areas A1, A2, and A3, as shown in Fig. 18. In each display area, a user who appears in a live view moving image taken by each of the smartphones 2A to 2C is displayed.

[0158] As shown in Figure 18, when user A, user B, and user C are displayed in display areas A1 to A3, respectively, the composite image obtained after a still image is captured will also show user A on the left, user B in the center, and user C on the right.

[0159] The number of divisions of the live view image is determined according to the number of users who will be taking pictures together.

[0160] The number of users taking pictures together is determined by the remote photography management server 1 to be, for example, the same as the number of smartphones 2 accessing the server simultaneously. In this case, the live view image is divided into display areas equal in number to the number of mobile terminals used for remote photography, and the users appearing in the live view moving images taken by each smartphone 2 are displayed in each display area.

[0161] The number of users who will be taking photos together may be determined based on the contents of a setting made in advance by the user of the organizer.

[0162] The remote shooting management server 1 may recognize the faces of users appearing in the live view moving images transmitted from each smartphone 2, and determine the number of users who will be shooting together based on the number of recognized faces. In this case, for example, the game control unit 131 functioning as a user management unit analyzes the live view moving images and recognizes faces.

[0163] FIG. 19 is a diagram showing a display example of a live view image.

[0164] As described with reference to Fig. 7 etc., there may be a plurality of users using one smartphone 2. In this case, a display area showing a larger number of users is set to be larger than a display area showing a smaller number of users.

[0165] 19, users A and B appearing in a live view video captured by one smartphone 2 are displayed in display area A1 on the left, and user C appearing in a live view video captured by the other smartphone 2 is displayed in display area A2 on the right. For example, 2 / 3 of the entire screen is allocated to display area A1, and the remaining 1 / 3 is allocated to display area A2.

[0166] In this way, the size of each display area into which the live view image is divided is determined according to the number of users shown in each display area.

[0167] As shown in Fig. 20, the user may be able to move the boundaries of the display areas to adjust the size of each display area. In the example of Fig. 20, the size of each display area can be adjusted by moving the boundary line displayed on the live view image left or right with a finger.

[0168] A plurality of layouts for live view images, each differing in the number of divisions and the size of the display area, may be prepared, and the user may be allowed to select from the prepared layouts.

[0169] Border display The borders representing the boundaries of the display area may be displayed for a certain period of time from the start of the live view image display, and then disappear after the certain period of time has passed. By displaying the screen in a larger size, it is easier to see and the degree of freedom in posing increases.

[0170] The user may be able to select whether to display or not display the border line representing the boundary of the display area, as well as the color, thickness, and density of the border line. Instead of lines, each display area may be represented by a different background color.

[0171] Live View video orientation FIG. 21 is a diagram showing another display example of a live view image.

[0172] As shown in display area A1 in Fig. 21, information guiding the orientation of the image (the orientation of the user in the live view moving image) may be displayed. In the example of Fig. 21, a message guiding the upward direction of the image is displayed along the upper edge of display area A1. In addition to visual information such as messages and symbols, guidance on the orientation of the image may also be provided by audio guidance.

[0173] The smartphone 2 used for remote photography is a device whose orientation can be easily changed. Therefore, as shown in the upper part of Fig. 22, depending on the orientation of the smartphone 2 used by user A, the orientation of user A may be different from the orientation of user B.

[0174] By presenting information that serves as a guide for the image orientation, the user can adjust the image orientation by changing the orientation of the casing or rotating the image. For example, when user A performs an operation to rotate the image, the image in which user A appears is displayed rotated, as shown by the arrow in Figure 22.

[0175] The orientation of the face in the image may be recognized, and if the orientation of the face is not aligned, information that serves as a guide for the orientation of the image may be presented. Information that serves as a guide for the orientation of the image may be presented only to a user whose facial orientation is not aligned with that of other users.

[0176] The operation to rotate an image may be allowed only for the organizer user, or may be allowed for each user. If each user is allowed to rotate an image, while one user is making an adjustment, other users cannot make adjustments. This makes it possible to prevent a situation where one user is making an adjustment and accidentally adjusts the orientation of an image that does not need to be adjusted.

[0177] Enlarged image display When a display area is selected, the image displayed in that display area may be enlarged and displayed, allowing the user to check in detail how they look before taking a photo.

[0178] - Display of additional information A pose image serving as a sample pose may be displayed together with the live view image.

[0179] The user may be allowed to select whether or not to display a pose image before taking a photograph. If a pose image is displayed, the user may be allowed to select a pose in which the user wants to take the photograph. An image stored in the user's smartphone 2 may be used as the pose image.

[0180] Images saved by a function of another application may be used as the pose image. For example, images posted to a community site may be used as the pose image. The community site may be a posting site for images taken with a photo sticker creation device (described later). For example, the community site may provide functions for searching for images by model of the photo sticker creation device used to take the image or by pose, and for searching for images using words. The community site may also provide functions for displaying image rankings, highlighting, and adding to favorites.

[0181] Information that serves as a guide for adjusting the position and size of the face may be displayed.

[0182] FIG. 23 is a diagram showing an example of a guide display.

[0183] In the example of Fig. 23A, vertically elongated oval guide images G1 and G2, which are reference information for adjusting the position and size of the face, are displayed in display area A1 and display area A2, respectively. The live view image shown in Fig. 23A is an image that is displayed when, for example, user A and user B are taking a photo together, as described with reference to Fig. 17.

[0184] User A, who is preparing to take a photograph while looking at the display in display area A1, adjusts the distance from the smartphone 2 and the positional relationship between the smartphone 2 and himself so that his face is aligned with guide image G1. User B, who is preparing to take a photograph while looking at the display in display area A2, adjusts the distance from the smartphone 2 and the positional relationship between the smartphone 2 and himself so that his face is aligned with guide image G2.

[0185] In the example of Fig. 23B, guide images G1-1 and G1-2 are displayed in display area A1, and guide image G2 is displayed in display area A2. The live view image shown in Fig. 23B is an image that is displayed when, for example, as described with reference to Fig. 19, user A and user B are using one smartphone 2, and user C is using one smartphone 2 to take a photo together.

[0186] Users A and B, who are preparing to take a photograph while looking at the display in display area A1, adjust the distance from the smartphone 2 and their positional relationship with the smartphone 2 so that their faces align with guide images G1-1 and G1-2, respectively. User C, who is preparing to take a photograph while looking at the display in display area A2, adjusts the distance from the smartphone 2 and their positional relationship with the smartphone 2 so that their faces align with guide image G2.

[0187] In this way, the number of guide images displayed corresponds to the number of people shown in each display area.

[0188] The guide image may be configured as a rectangular image, a straight line image, or the like, instead of an oval image.

[0189] The guide images displayed in the respective display areas may be the same size or different sizes, and the size of each guide image may be selectable by the user.

[0190] <Provided image> Although a composite image, which is a still image, is provided to each user who has taken remote photographs, other images may also be provided.

[0191] FIG. 24 is a diagram showing an example of a provided image.

[0192] As shown in Fig. 24A, a moving image P52 may be provided together with a composite image P51. The moving image P52 is, for example, a moving image generated by combining moving images for live view.

[0193] 24B, an individual image P53 may be provided together with the composite image P51. The individual image P53 is a still image generated based on the captured images for synthesis. If users A and B take remote photos together, for example, an individual image P53 of user A is provided to user A together with the composite image P51, and an individual image P53 of user B is provided to user B together with the composite image P51.

[0194] In this way, the same image may be provided to each user who has taken a remote photograph, or different images may be provided to each user.

[0195] <Image adjustment> The appearance of the composite image provided by remote photography may be adjusted after the remote photography.

[0196] Position adjustment FIG. 25 is a diagram showing an example of the arrangement position.

[0197] For example, when a composite image is captured showing three users side by side as described with reference to FIG. 18, the positions of each user may be moved and adjusted to their preferred positions, as shown in FIG. 25.

[0198] In the example of Fig. 25, user B is placed over user A and user C. In this way, the anteroposterior relationship may be adjusted by selecting the layer of each user.

[0199] In addition to adjusting the position, the size may be adjusted by enlarging or reducing it.

[0200] When adjusting the position, guide information may be displayed, such as grid lines or crosses.

[0201] Adjusting the image quality The image quality, brightness, color, etc. of the captured image may vary depending on the specifications of the camera 306 installed in the smartphone 2, the location of the remote image capture, etc. Therefore, the image quality may be adjusted by the user, and calibration may be performed manually or automatically. For example, adjustments may be made to average image quality and brightness, to adjust to a brighter image, or to adjust to a better image quality.

[0202] It may be possible to use functions similar to those installed in displays (monitors), printers, image scanners, digital cameras, etc. to correct differences in color development characteristics between devices, individual differences, changes over time, etc.

[0203] Image processing such as retouching and makeup Retouching is image processing that adjusts the size of eyes and skin color. Makeup is image processing that makes the image appear to be made up. When image processing using retouching or makeup is performed, it may be possible to compare the image with the image before the processing.

[0204] Content synthesis The composite image may be edited using content. The content may include, for example, a background image, a stamp image, and a pen image. A background image is an image that is composited into a background layer of a person's layer. A stamp image is an image of a predetermined design that is composited into a position on the composite image specified by a finger or the like. A pen image is an image that is composited along the trajectory of the movement of a finger or the like.

[0205] The composite image may be subjected to image processing using other content such as a veil, aura, filter, etc. The veil, aura, and filter are image processing that adjust the color tone, etc.

[0206] The content to be used for image processing may be selected before the start of remote shooting or during remote shooting, rather than after the remote shooting. The content selected during remote shooting is reflected in the display of a live view image, etc. The content selected before the start of remote shooting or during remote shooting may be changed after the remote shooting, or the composition position may be changed.

[0207] <Other> Overlapping shots A function of superimposing a still image of one user onto a still image of another user may be used. In this case, the still images of one user and the still images of the other user may be taken at different times rather than at the same time.

[0208] The function of taking multiple pictures may be made available when the number of users taking remote pictures together is greater than a threshold value.

[0209] One-to-many remote shooting One user may be able to simultaneously perform remote photography with multiple other users. This type of remote photography is one-to-many remote photography in the sense that one user photographs with each of multiple users.

[0210] FIG. 26 is a diagram showing an example of a composite image when one-to-many remote photography is performed.

[0211] In the example of Fig. 26, user A performs remote photography together with user B and also performs remote photography together with user C. The former remote photography is remote photography between users A and B, and the latter remote photography is remote photography between users A and C. The remote photography between users A and B and the remote photography between users A and C are performed simultaneously.

[0212] An image of user A taken with smartphone 2A, an image of user B taken with smartphone 2B, and an image of user C taken with smartphone 2C are each sent to remote photography management server 1 and used to generate live view images and composite images.

[0213] For example, based on a live view video captured by the smartphone 2A and a live view video captured by the smartphone 2B, a live view image for remote capture between users A and B is displayed on each of the smartphones 2A and 2B.

[0214] Furthermore, based on the live view moving images captured by the smartphone 2A and the live view moving images captured by the smartphone 2C, live view images for remote capture between users A and C are displayed on the smartphone 2A and the smartphone 2C, respectively.

[0215] When a still image is taken by each of the smartphones 2A to 2C, the remote photography management server 1 generates a composite image for each remote photography and provides it to the users participating in each remote photography.

[0216] That is, as shown in FIG. 26, a composite image P61 generated based on a composite image taken with smartphone 2A and a composite image taken with smartphone 2B is generated as a composite image for remote shooting between users A and B and is provided to users A and B.

[0217] In addition, a composite image P62 generated based on a composite image taken by smartphone 2A and a composite image taken by smartphone 2C is generated as a composite image for remote shooting between users AC and is provided to users A and C.

[0218] In this way, one user may participate in multiple remote photography sessions simultaneously. For example, a celebrity could be one user and take remote photos with multiple other users. The image showing the celebrity may be switched to an image showing another user after a predetermined time has elapsed.

[0219] Examples of imaging devices Although the remote photography is performed using a smartphone in the above example, it may be performed using other photography devices with a camera, such as a photo sticker creation device installed in a game center. The photo sticker creation device is a game machine that takes a photo of a user who has paid a game fee and provides sticker paper with an image of the user printed on it.

[0220] A composite image generated using the remote photography application may be printed using a printer of the photo sticker creation device.

[0221] Program example The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0222] The program to be installed is provided by being recorded on removable media such as an optical disc (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or semiconductor memory. It may also be provided via wired or wireless transmission media such as a local area network, the Internet, or digital broadcasting. The program can be pre-installed.

[0223] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0224] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0225] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0226] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0227] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0228] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0229] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices. [Explanation of symbols]

[0230] 1 Remote photography management server, 2 Smartphone, 121 Information processing unit, 131 Game control unit, 132 Setting information DB, 133 Image acquisition unit, 134 Image processing unit, 301 Control unit, 301A Application execution unit, 311 Photography control unit, 312 Output control unit, 313 Remote photography setting unit

Claims

1. an imaging control unit that causes a first portable terminal used by a first user and a second portable terminal used by a second user to perform imaging; an image processing unit that extracts a person area from an image captured by each of the first portable terminal and the second portable terminal, displays a live view image, which is a moving image generated based on the image of the extracted person area, on the first portable terminal and the second portable terminal, and displays information serving as a guide for the orientation of the image together with the live view image when the orientation of the face is not aligned; Equipped with the photographing control unit, in response to a photographing button being pressed after display of the live view image has started, causes the first portable terminal and the second portable terminal to display a countdown until a photographing timing; The image processing unit generates a composite image in which an image of a person area of ​​the first user and an image of a person area of ​​the second user are arranged based on images captured at the capturing timings on the first portable terminal and the second portable terminal, and displays the composite image on the first portable terminal and the second portable terminal. Information processing device.

2. The image processing unit displays images of person regions extracted from images captured by the first portable terminal and the second portable terminal in an area having a size corresponding to the number of people appearing in the captured images. The information processing device according to claim 1 .

3. The image processing unit determines display areas for the image of the person area of ​​the first user and the image of the person area of ​​the second user according to a layout selected from a plurality of types of layouts having different numbers of divisions and sizes of display areas. The information processing device according to claim 2 .

4. The image processing unit displays information serving as a guide for the image orientation only on the screen of the mobile terminal in which the orientation of the face is different from the reference orientation. The information processing device according to claim 1 .

5. The information processing device photographing the first portable terminal used by the first user and the second portable terminal used by the second user; extracting a person area from an image captured by each of the first and second portable terminals; a live view image, which is a moving image generated based on an image of the extracted person area, is displayed on the first portable terminal and the second portable terminal, and information serving as a guide for the orientation of the image is displayed together with the live view image when the orientation of the face is not aligned; In response to a photographing button being pressed after the display of the live view image has started, a countdown to a photographing timing is displayed on each of the first portable terminal and the second portable terminal; A composite image in which an image of the person area of ​​the first user and an image of the person area of ​​the second user are arranged is generated based on the images captured at the capturing timing on each of the first portable terminal and the second portable terminal, and the composite image is displayed on each of the first portable terminal and the second portable terminal. Information processing methods.

6. photographing the first portable terminal used by the first user and the second portable terminal used by the second user; extracting a person area from an image captured by each of the first and second portable terminals; a live view image, which is a moving image generated based on an image of the extracted person area, is displayed on the first portable terminal and the second portable terminal, and information serving as a guide for the orientation of the image is displayed together with the live view image when the orientation of the face is not aligned; In response to a photographing button being pressed after the display of the live view image has started, a countdown to a photographing timing is displayed on each of the first portable terminal and the second portable terminal; A composite image in which an image of the person area of ​​the first user and an image of the person area of ​​the second user are arranged is generated based on the images captured at the capturing timing on each of the first portable terminal and the second portable terminal, and the composite image is displayed on each of the first portable terminal and the second portable terminal. A program that causes a computer to execute a process.

Images