Imaging apparatus and program

Abstract

To generate a composite video in which a user and a performer are captured in a natural overlapping manner.SOLUTION: An imaging apparatus includes a camera for capturing a video of a user, a display unit, a storage unit for storing a first video and a second video, a key signal generating unit for generating a key signal from a video of the user captured by the camera, a third video generating unit for generating a third video from the captured video using the key signal of the captured video, a composite unit for generating a composite video for determining the order in which the second video and the third video are superimposed on the first video on the basis of the relationship between the depth position of the user and a reference position of the second video and superimposing the second video and the third video on the first video and composing them to generate a composite video, and a display control unit for displaying the composite video on a display unit.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an imaging device and a program, and more particularly to an imaging device and a program for imaging a user and generating a virtual reality image. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there has been an automatic photo creation device for amusement that takes a photograph of a subject, including a user, with a camera, and outputs a composite image, in which the photographed image is combined with an additional image, as a printed photograph.

[0003] However, in the above-mentioned automatic photo creation devices, the device manufacturer separately prepares image sources as additional images and stores them in the device in advance, and the user uses these in editing work. Therefore, the user has no choice but to use the additional images prepared in advance by the device manufacturer, and cannot achieve a satisfactory feeling.

[0004] Therefore, an automatic photo creation device has been invented that can register a photographed image of a preceding subject and its edited image as a new additional image to be added to a photographed image of a user as a subsequent subject (Patent Document 1). The preceding subject is a person with added value who can be granted publicity rights or merchandising rights, such as a celebrity. This allows a photo to be taken of the user alongside the celebrity, providing a sense of satisfaction not previously available. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-005401 Summary of the Invention [Problem to be solved by the invention]

[0006] Incidentally, the additional image of the automatic photo creation device is a still image, and the purpose of the automatic photo creation device is to provide a photo in which the user and a celebrity or the like are side by side.

[0007] However, the above-mentioned automatic photo creation device does not generate a natural composite image depending on the standing positions in front of and behind the user, and there are cases where the user is not given a sufficient sense of satisfaction.

[0008] Therefore, the object of the present invention is to provide a filming device and program that can generate a composite image in which the user and performer are filmed with a natural overlap, allowing the user to have the experience of sharing the same space as the performer. [Means for solving the problem]

[0009] One aspect of the present invention is an imaging device comprising a camera that images a user, a display unit, a memory unit that stores a first image and a second image, a key signal generation unit that generates a key signal from the image of the user captured by the camera, a third image generation unit that generates a third image from the captured image using the key signal of the captured image, a synthesis unit that determines the depth position of the user on the image from the key signal, determines the order in which the second image and the third image are to be superimposed on the first image based on the relationship between the depth position of the user and the reference position of the second image, and superimposes the second image and the third image on the first image in accordance with the order to synthesize them and generate a composite image, and a display control unit that displays the composite image on the display unit.

[0010] One aspect of the present invention is a booth having a first surface, a second surface facing the first surface, and a shooting space for a user between the first surface and the second surface, a camera provided on the first surface for shooting the user, a display unit provided on the first surface, a background unit of a specific color provided on the second surface, a storage unit storing a first video and a second video, a key signal generation unit for generating a key signal from a shooting video of the user shot by the camera, a third video generation unit for generating a third video from the shooting video using the key signal of the shooting video, determining a depth position of the user on the video from the key signal, and determining the second video and the third video based on the relationship between the depth position of the user and the reference position of the second video. A shooting apparatus comprising: a synthesizing unit that determines an order of superimposing on the first video, superimposes and synthesizes the second video and the third video on the first video in accordance with the order to generate a synthesized video; and a display control unit that displays the synthesized video on the display unit.

[0011] One aspect of the present invention is a program that causes a computer to function as acquisition means for acquiring a first video and a second video from a storage unit, key signal generation means for generating a key signal from a shooting video of a user shot by a camera, and using the key signal of the shooting video. Third video generation means for generating a third video from the shooting video, determining the depth position of the user on the video from the key signal, and determining the relationship between the depth position of the user and the reference position of the second video. A synthesizing means for determining an order of superimposing the second video and the third video on the first video, superimposing and synthesizing the second video and the third video on the first video in accordance with the order to generate a synthesized video, and a display control means for displaying the synthesized video on the display unit.

Advantages of the Invention

[0012] The present invention can provide an experience in which a user can share the same space as a performer such as a talent, and can generate a synthesized video in which the user and the performer are photographed with a natural overlapping state.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Embodiments for Carrying Out the Invention

[0014] [Overall Configuration] Figure 1 is a diagram showing an example of the overall configuration of the imaging system in the present embodiment. As shown in Figure 1, the imaging system includes an imaging device 1, a management server 2, a settlement server 3, and a terminal 4. The imaging device 1, the management server 2, the settlement server 3, and the terminal 4 can be connected to the communication line N and can communicate with each other.

[0015] The communication line N means a communication path capable of data communication. That is, the communication line N includes a dedicated line (dedicated cable) for direct connection, a LAN such as Ethernet (registered trademark), and communication networks such as a telephone communication network, a cable network, and the Internet, and the communication method is not limited to wired / wireless.

[0016] The imaging device 1 is installed in facilities such as event venues, commercial facilities, and movie theaters, and is a device that generates a composite video obtained by synthesizing a captured video of a user (subject) and a content video. The content video is a video (for example, a moving image) of entertainers (casts in dramas, movies, etc., idols, etc.), sports players, etc. (hereinafter, these are collectively referred to as performers). The imaging device 1 also has a digital signage function. Basically, there are a plurality of imaging devices 1, which are installed in facilities such as each event venue and movie theater.

[0017] The management server 2 is a server that manages the content videos and advertisements used by the imaging device 1 and stores the recorded images provided to the user (subject).

[0018] The settlement server 3 is a server that settles the usage fees of the imaging device 1.

[0019] The terminal 4 is a terminal possessed by the user (subject), is connected to the communication line N via a wireless communication base station or the like, and can perform data communication with the management server 2. The terminal 4 is, for example, a smartphone, a mobile phone, a personal computer, a tablet computer, or the like. Basically, there are a plurality of terminals 4, which are operated by each user (subject). And the terminal 4 has at least a function of receiving image data and storing or displaying the image data, a photographing function, and the like.

[0020] [Configuration of Each Device] The configuration of the imaging device 1 will be described. FIG. 2 is an external view of the imaging device 1, and FIG. 3 is a block diagram of the imaging device 1.

[0021] The imaging device 1 is, for example, a box-shaped booth. A camera 10 and a display 11 are provided on the first surface 100 of the booth. The second surface 101 is provided at a position facing the first surface 100, and the background surface 102 on the side of the second surface 101 facing the first surface 100 serves as the background surface of the user, and the background surface is colored with a specific color. The specific color is, for example, green. A photographing space 103 for photographing the user (subject) is provided between the first surface 100 and the second surface 101.

[0022] In FIG. 3, 10 is a camera, 11 is a display, 12 is a speaker, 13 is an operation input unit, 14 is a short-range wireless communication unit, 15 is a processing unit, 16 is a transmission / reception unit, and 17 is a storage unit.

[0023] The camera 10 is a camera that captures a user (subject) and a background surface 102 of a shooting space 103 in which the user (subject) is present. The video (including both still images and videos) captured by the camera 10 will be referred to as a captured video hereinafter. This captured video is input to the processing unit 15.

[0024] The display 11 displays an image obtained by combining the image captured by the camera 10 and the content image.

[0025] The speaker 12 outputs the sound of the content.

[0026] The operation input unit 13 is used by the user (subject) to input various operations related to photography, and outputs an operation input signal corresponding to the operation input to the processing unit 15. The functions of the operation input unit 13 can be realized by, for example, elements operated by the user's fingers, such as a keyboard, mouse, touch operation pad, home button, button switch, joystick, or trackball, as well as elements that detect movement or posture, such as an acceleration sensor, angular velocity sensor, tilt sensor, or geomagnetic sensor, or a two-dimensional code input from a camera (for example, a two-dimensional code of information specifying content). The operation input unit 13 in this embodiment is a touch operation panel equipped with a display on which operation information is displayed. The display may also serve as the display 11 that displays content.

[0027] The short-distance wireless communication unit 14 is a reader / writer for a contactless IC card used for payment of usage fees.

[0028] The processing unit 15 comprehensively controls the operation of the image capturing device 1 based on programs and content data stored in the storage unit 17, operation input signals from the operation input unit 13, etc. The functions of the processing unit 15 can be realized by electronic components such as a microprocessor such as a CPU or GPU, an ASIC, an IC memory, etc.

[0029] As the main functional units, the processing unit 15 includes a management unit 150, an operation control unit 151, a settlement processing unit 152, a key signal generation unit 153, a user video generation unit 154, a video composition unit 155, a signage control unit 156, an audio control unit 157, and a storage location prompting unit 158.

[0030] The management unit 150 manages the settings and data of the imaging device 1. The settings of the imaging device 1 are made by the administrator after authentication with the assigned account and password of the imaging device 1. Representative settings of the imaging device 1 include the settings of the content used by the imaging device 1. The content is downloaded from the management server 2 and stored in the storage unit 17. The content download has an automatic update mode in which the content is automatically downloaded and a manual update mode in which the administrator selects the desired content for download. By setting either mode by the administrator, the content is downloaded and stored in the imaging device 1.

[0031] The content includes content videos to be synthesized with the captured videos of users (subjects) and signage videos (advertising videos) used for digital signage.

[0032] The content videos are videos of performers such as entertainers (casts of dramas, movies, idols, etc.) and sports players. The content videos are basically moving images. And the content videos include a first video and a second video. FIG. 4 is a diagram showing an example of a background video (first video), and FIG. 5 is a diagram showing an example of a performer video (second video). The first video is a background video that serves as the background of the composite video. The second video is a performer video of only the performer excluding the background video, and is also called a key fill video.

[0033] The signage video is an advertisement video and includes both videos and still images. Note that the content data is described to include the content video and the metadata (such as identification information and usage fees) related to those videos. Similarly, the signage data is described to include the signage video and the metadata (such as identification information) related to those videos.

[0034] The operation control unit 151 displays the content of the content (for example, the thumbnail video) and operation information on the operation input unit 13. Also, it receives the operation signal from the operation input unit 13.

[0035] The settlement processing unit 152 receives the settlement information of the contactless IC card read by the short-range wireless communication unit 14 and transmits the settlement information to the settlement server 3. Also, when the settlement is completed, it receives the settlement completion information from the settlement server 3.

[0036] The key signal generation unit 153 inputs the captured video of the user (subject) captured by the camera 10 and generates a key signal including the signal component of the user from the captured video. As a method for generating the key signal, a conventional chroma key method can be used. FIG. 6 is a diagram for explaining the key signal. The key signal generation unit 153 compares the captured video of the user (subject) with the color signal of a specific color (for example, green) of the reference background surface 102, and generates a valid key signal if they match within the specified error range. This key signal is used to generate the third video described later.

[0037] Note that the key signal generation process of the key signal generation unit 153 is an example, and other methods may be used. For example, a method of automatically detecting the contour of the user from the captured video by a method such as AI and extracting only the video of the user may be used.

[0038] The user video generation unit 154 uses the key signal generated by the key signal generation unit 153 to generate a user video (third video) of only the user (subject) by removing the background from the captured video of the user. FIG. 7 is a diagram for explaining the generation of the user video (third video). The generation process generates the user video (third video) by cutting out a video of a region specified based on the key signal from the entire region of the captured image. Here, for ease of understanding, a case will be described in which the transmittance of the key signal is 0 percent (opaque) or 100 percent (transparent). As is clear from FIG. 7, of the entire region of the user video obtained by the synthesis process, the captured video is used in regions where the transmittance of the key signal is 100 percent (white regions in FIG. 7), and the captured video is masked in regions where the transmittance of the key signal is 0 percent (black regions in FIG. 7), resulting in the generation of a user video (third video) in which the user region is transparent.

[0039] The video composition unit 155 composes the background video (first video), the performer video (second video), and the user video (third video) to generate a single composite video. As for the order in which the videos are superimposed, the video composition unit 155 determines the depth position of the user on the video from the key signal, and determines the superimposition order in which the performer video (second video) and the user video (third video) are superimposed on the background video (first video) based on the relationship between the depth position of the user and the reference position of the performer video (second video). Then, the performer video (second video) and the user video (third video) are superimposed on the background video (first video) according to the superimposition order to compose the video, thereby generating a composite video.

[0040] A method for determining the superimposition order will now be described. FIG. 8 is a diagram for explaining a method for determining the superimposition order. First, as shown in FIG. 8, a depth reference position of the performer video (second video) is determined. This depth reference position is a position where, when a user stands at a predetermined position in the shooting space corresponding to the depth reference position and a video is shot, the user in the shot video and the performer in the performer video appear to be standing at the same depth position. A predetermined Y coordinate in the depth direction (vertical Y direction) on the performer video (second video) is set as the depth reference position. The video composition unit 155 stores this depth reference position.

[0041] The video composition unit 155 inputs the key signal of the captured video, obtains the lowest Y coordinate of the signal component indicating the user's area in the key signal, sets this coordinate as the user's position coordinate, and compares the Y coordinate of the user's position coordinate with the Y coordinate of the depth reference position. When the Y coordinate of the user's position coordinate is greater than the Y coordinate of the depth reference position (when, on the video, the Y coordinate of the user's position coordinate is below the Y coordinate of the depth reference position), it is determined that the user's standing position is in front of the depth reference position, and the superimposition order for superimposing on the background video is set as the order of the performer video and the user video. On the other hand, when the Y coordinate of the user's position coordinate is less than the Y coordinate of the depth reference position (when, on the video, the Y coordinate of the user's position coordinate is above the Y coordinate of the depth reference position), it is determined that the user's standing position is behind the depth reference position, and the superimposition order for superimposing on the background video is set as the order of the user video and the performer video.

[0042] The video composition unit 155 superimposes the performer video and the user video on the background video according to the superimposition order to generate a composite video. FIG. 9 is an example of a composite video when the Y coordinate of the user's position coordinate is greater than the Y coordinate of the depth reference position (when, on the video, the Y coordinate of the user's position coordinate is below the Y coordinate of the depth reference position). FIG. 10 is an example of a composite video when the Y coordinate of the user's position coordinate is less than the Y coordinate of the depth reference position (when, on the video, the Y coordinate of the user's position coordinate is above the Y coordinate of the depth reference position). As shown in FIG. 9, the user is positioned in front of the performer, and the overlap between the user and the performer is also such that the performer is hidden behind the user, resulting in a natural video. Also, as shown in FIG. 10, the user is positioned behind the performer, and the overlap between the user and the performer is also such that the user is hidden behind the performer, resulting in a natural video.

[0043] The video composition unit 155 performs the determination of the above-described superimposition order and the composition of the video in units of frames.

[0044] In addition, the video composition unit 155 also has a function of recording the video at a certain moment in the composite video (video) as a still image.

[0045] These composite images (moving images) or composite images (still images) are stored in a storage location of the management server 2, which will be described later. Note that the composite images (moving images) may be all or part of the moving images.

[0046] The signage control unit 156 uses the signage data to display an advertisement on the display 11 during times when the image capturing device 1 is not being used for image capturing.

[0047] The audio control unit 157 uses the audio data included in the content data to output the voice of the performer on the video, the audio of usage instructions, etc. from the speaker 12 when shooting the video of the user (subject).

[0048] The storage location presenting unit 158 notifies the user (subject) of the storage location of the composite image described above. One notification method is to display a two-dimensional code (QR code (registered trademark) or the like) indicating the address of the storage location of the composite image on the display 11. However, the present invention is not limited to this, and the address of the storage location may be displayed directly on the display 11, or another method may be used.

[0049] The transmitting / receiving unit 16 transmits and receives data to and from the management server 2 and the settlement server 3 via the communication line N.

[0050] The storage unit 17 stores programs that cause the processing unit 15 to function as a management unit 150, an operation control unit 151, a payment processing unit 152, a key signal generation unit 153, a user image generation unit 154, an image synthesis unit 155, a signage control unit 156, an audio control unit 157, and a storage location presentation unit 158. The storage unit 17 also stores various data necessary for the image capture device 1, such as a content database and setting data.

[0051] Fig. 11 is a diagram showing an example of a content database. The database shown in Fig. 11 stores content data of performers used in filming with users and signage data used as advertisements. The content data includes content identification information (ID), a content name, video data including background video (first video) and performer video (second video), audio data, and metadata. The signage data includes signage identification information (ID), an advertisement name, video data, audio data, and metadata. Note that Fig. 11 is an example and is not limited to this.

[0052] Fig. 12 is a diagram showing an example of setting data. The setting data shown in Fig. 12 includes the account of the camera device 1, an administration password, whether or not automatic updates are enabled, the advertisement start time (the time elapsed since the camera device 1 was last used), and auxiliary data. Note that Fig. 12 is just an example and is not limited to this.

[0053] Next, a description will be given of the configuration of the management server 2. FIG.

[0054] In FIG. 13, 20 is an input unit, 21 is a transmitting / receiving unit, 22 is a processing unit, and 23 is a storage unit.

[0055] The input unit 20 is used by the administrator to input various operations, and outputs an operation input signal corresponding to the operation input to the processing unit 22. The functions of the input unit 20 can be realized by elements that are operated by the user's fingers, such as a keyboard, mouse, touch operation pad, home button, button switch, joystick, or trackball, as well as elements that detect movement or posture, such as an acceleration sensor, angular velocity sensor, tilt sensor, or geomagnetic sensor.

[0056] The transmitting / receiving unit 21 transmits and receives data to and from the photographing device 1 via the communication line N.

[0057] The processing unit 22 comprehensively controls the operation of the management server 2 based on programs and content data stored in the storage unit 23, operation input signals from the input unit 20, and the like. The functions of the processing unit 15 can be realized by electronic components such as microprocessors such as CPUs and GPUs, ASICs, and IC memories. This processing unit 22 includes, as main functional units, a content management unit 121 and a captured data storage management unit 122.

[0058] The content management unit 121 manages content data to be set in the imaging device 1. Specifically, using the imaging device management database stored in the storage unit 23, it manages content used in the imaging device 1 and the like. FIG. 14 is an example of the imaging device management database. The imaging device management database shown in FIG. 14 records, for each imaging device 1, identification information of the imaging device, an account, a password, whether there is automatic update, and identification information of available content.

[0059] The captured data storage management unit 122 stores the recorded images captured by the imaging device 1 in the storage unit 23 and notifies the imaging device 1 of the address of the storage location.

[0060] Next, the configuration of the settlement server 3 will be described. FIG. 15 is a block diagram of the settlement server 3.

[0061] In FIG. 15, 30 is a transmission / reception unit, 31 is a settlement processing unit, and 32 is a storage unit.

[0062] The transmission / reception unit 30 transmits and receives data to and from the imaging device 1 via the communication line N.

[0063] The settlement processing unit 31 receives settlement information from the imaging device 1, performs settlement processing based on programs and settlement data stored in the storage unit 32, and transmits settlement completion information to the imaging device 1. The settlement processing unit 31 can be realized by electronic components such as microprocessors such as CPUs and GPUs, ASICs, and IC memories. In addition, a conventional method can be used for the settlement of the fees of non-contact IC cards.

[0064] [Operation of each device] The operation of the first embodiment will be described with reference to Fig. 16, which is a sequence diagram between devices.

[0065] First, a setting process is performed to make the image capturing device 1 available (Step 1). Then, in response to a setting request from the image capturing device 1, the management server 2 performs a management process and transmits content data and signage data (Step 2).

[0066] Here, the operation of the setting process (Step 1) of the photographing device 1 will be described using a flowchart. Fig. 17 is an operation flowchart of the setting process (Step 1) of the photographing device 1.

[0067] First, the administrator of the image capturing device 1 displays the management screen by operating the operation input unit 13. The operation control unit 151 displays the management screen on the operation input unit 13 (Step 100).

[0068] The administrator operates the operation input unit 13 to input identification information (account) and a password (Step 101).

[0069] The management unit 150 verifies the input identification information (account) and password (Step 102), and if authentication is successful (Step 103), the process proceeds to selecting an update mode. On the other hand, if authentication is not successful, the setting process ends.

[0070] If authentication is successful, the management screen displays automatic update mode and manual update mode, and the administrator can select either mode.

[0071] If the administrator selects the automatic update mode (Step 104), content data and signage data available for the image capture device 1 are transmitted from the management server 2, and the management unit 150 stores the content data and signage data in the content database of the storage unit 17, completing the content setting. Then, a setting completion notification is transmitted to the management server 2.

[0072] On the other hand, if the administrator selects the manual update mode (Step 104), a list of content data and signage data available for use by the image capture device 1 is transmitted from the management server 2, and the management unit 150 displays the list information of the content data and signage data (Step 105). The administrator operates the operation input unit 13 to select the content data and signage data they wish to use from the list information (Step 106). The management unit 150 then downloads the selected content data and signage data from the management server 2 (Step 107), stores the selected content data and signage data in the content database of the storage unit 17, and completes the content setting (Step 108). The management unit 150 then transmits a setting completion notification to the management server 2.

[0073] Next, the operation of the management process (Step 2) of the management server 2 will be described with reference to a flowchart.

[0074] The management server 2 verifies the identification information (account) in response to the setting request from the image capturing device 1 (Step 200). If the identification information (account) is registered (Step 201), the management server 2 performs processing corresponding to the mode request from the image capturing device 1.

[0075] If the mode requested by the camera device 1 is the manual update mode (Step 202), the control unit 21 transmits a list of content data and signage data corresponding to the content ID or signage ID entered in the available content field of the account of the camera device 1 that requested the setting in the camera device management database of the storage unit 23 (Step 203). When the control unit 21 receives selection information of the content data and signage data from the camera device 1, it transmits the selected content data and signage data to the camera device 1 (Step 204). Then, when the control unit 21 receives a setting completion notification from the camera device 1, it ends the management process (Step 205).

[0076] On the other hand, if the mode requested by the imaging device 1 is the automatic update mode, the flag in the field "ON" for whether or not the account of the imaging device 1 that has been requested to set is set in the recording device management database of the storage unit 23 (Step 206). Then, among the contents registered in the content database, the content data and signage data corresponding to the content ID or signage ID listed in the available content field of the account of the photographing device 1 that requested the setting is searched (Step 207), and the available content data and signage data are transmitted (Step 208). When the setting completion notification is received from the photographing device 1, the management process is terminated (Step 205).

[0077] Furthermore, for camera devices 1 with accounts for which automatic updates are enabled, if there is a change in the available content, the content data or signage data corresponding to the ID listed in the available content field is automatically sent to the camera device 1, and the content of the camera device 1 is automatically updated.

[0078] Returning to the explanation of the sequence diagram of FIG.

[0079] Once the settings of the photographing device 1 are complete, the user (subject) can use the photographing device 1. Then, when the user (subject) starts using the photographing device 1, the photographing device 1 executes a photographing process (Step 3).

[0080] Here, the operation of the photographing process (Step 3) will be described using a flowchart. Fig. 19 is an operational flowchart of the photographing process (Step 3).

[0081] The operation control unit 151 of the imaging device 1 displays a list of contents on the operation input unit 13 (Step 300). Fig. 20 is an example of a content selection screen displayed on the operation input unit 13. The example of Fig. 20 is an example of a screen from which three types of content, content AAA, content BBB, and content CCC, can be selected.

[0082] A user who uses the imaging device 1 selects desired content from a list of content displayed on the operation input unit 13. The selection is made by touching the desired content from the list of content (Step301).

[0083] When the content is selected, a screen for the usage fee of the content is displayed on the screen of the operation input unit 13 (Step302). FIG. 21 is an example of the usage fee screen displayed on the operation input unit 13. In the example of FIG. 21, the usage fee for the content AAA is 600 yen, and it is an example of a screen that requests touching an IC card to the short-range wireless communication unit 14.

[0084] When the user touches the IC card to the short-range wireless communication unit 14 according to the instruction on the usage fee screen, the settlement processing unit 152 of the imaging device 1 receives the settlement information of the fee (Step303), and transmits the settlement information to the settlement server 3 (Step304).

[0085] The settlement processing unit 152 of the imaging device 1 receives the settlement completion information from the settlement server 3 (Step305), displays on the operation input unit 13 that the settlement has been completed, and notifies that the imaging is to be started (Step306).

[0086] When the imaging of the user (subject) by the camera is started, the captured image is displayed on the display 11. Subsequently, the imaging device 1 performs an image synthesis process (Step307). Here, the operation of the image synthesis process will be described. FIG. 22 is an operation flowchart of the image synthesis process.

[0087] The key signal generation unit 153 performs frame determination (Step400). When the frame is updated (Step401), the key signal generation unit 153 generates a key signal from the captured image as shown in FIG. 6 (Step401).

[0088] The user image generation unit 154 generates a user image from the captured image using the key signal as shown in FIG. 7 (Step403).

[0089] As shown in Fig. 8, the video synthesis unit 155 uses a key signal to compare the user position (Y coordinate) with the depth reference position (Y coordinate) (Step 404). If the Y coordinate of the user's position coordinates is greater than the Y coordinate of the depth reference position (Step 405), the video synthesis unit 155 determines that the user's standing position is closer to the depth reference position, and sets the superimposition order of the performer video and the user video to be superimposed on the background video (Step 406). On the other hand, if the Y coordinate of the user's position coordinates is smaller than the Y coordinate of the depth reference position (Step 405), the video synthesis unit 155 determines that the user's standing position is deeper than the depth reference position, and sets the superimposition order of the user video and the performer video to be superimposed on the background video (Step 407).

[0090] The video composition unit 155 superimposes the user video and the performer video on the background video in the determined superimposition order to generate a composite video (Step 408).

[0091] The video synthesis unit 155 determines whether it is the final frame of the content (Step 407), and if it is the final frame, ends the video synthesis process. If it is not the final frame, the process returns to Step 400 and continues. This concludes the explanation of the video synthesis process, and the flow returns to the operational flowchart of the shooting process (Step 3) in FIG. The video synthesis unit 155 displays the synthesized video (video) on the display 11 (Step 308). At this time, the voice of the performer is also output from the speaker based on the audio data of the content. For example, a voice such as "This is AAA. Thank you for participating in the photo session today" is output. In this way, the display 11 displays an image of the performer and the user existing together in the same space (see the synthesized video in Figures 9 and 10), and the voice can also be heard, so the user can have a simulated experience of sharing the same space with the performer. The video composition unit 155 records the composite video (moving image) (Step 309).

[0092] When the shooting is completed, the storage location prompt section 158 uploads the composite video to the management server 2 (Step 310), and displays the address of the storage location on the display 11 using a two-dimensional code (Step 311). FIG. 23 is an example of a screen on which a two-dimensional code indicating the address of the storage location is displayed.

[0093] This concludes the description of the shooting process.

[0094] Return to the description of the sequence diagram in FIG. 16.

[0095] Step 4 is the payment process performed by the payment server 3. It receives the payment information of the user's IC card transmitted from the imaging device 1, and transmits payment completion information to the imaging device 1 based on the payment information.

[0096] Step 5 is the storage process performed by the management server 2. It stores the recorded video transmitted from the imaging device 1, and transmits the address of the storage location to the imaging device 1.

[0097] The user uses the shooting function of the terminal 4 to shoot the two-dimensional code displayed on the screen of the operation input unit 13, and accesses the address indicated by the two-dimensional code (Step 6). Then, the terminal 4 downloads the recorded video shot to the terminal 4 (Step 7). FIG. 24 is an example of the case where the recorded video downloaded to the terminal 4 is displayed on the display of the terminal 4.

[0098] On the other hand, when there is no use of the imaging device 1 by the user for a certain period of time (for example, 3 minutes), the signage control unit 156 of the imaging device 1 acquires video data and audio data from the signage data in the content database of the storage unit 17, displays an advertisement using the video data on the display 11, and outputs audio using the audio data from the speaker 12 (Step 8).

[0099] This concludes the description of the operations of the embodiment.

[0100] In this embodiment, the performer's video (video) and the user's video are combined and displayed on the display, and audio is also audible, allowing the user to experience sharing the same space as the performer. Furthermore, the recorded images recorded as a memento of the experience are stored on a management server, and the user can download them to their device, improving the convenience of sharing via, for example, social networking sites, and reducing costs by eliminating the need for physical printing.

[0101] Furthermore, this embodiment compares the depth reference position of the performer video with the user's standing position to determine the anteroposterior relationship between the performer video and the user video, and based on this determination, determines the order in which the performer video and the user video will be superimposed (overlaid) on the background video, and synthesizes the background video, performer video, and user video in that order, thereby generating a composite video in which the performer and user video overlap naturally. Furthermore, the anteroposterior relationship between the performer video and the user video is determined by comparing the reference depth position with the user position using a key signal from the user's captured video, so the processing load is small and the synthesis process can be performed almost in real time.

[0102] Furthermore, in this embodiment, the management server collectively manages the content used in the imaging device, thereby realizing immediate setting of the content used in the imaging device and flexible registration.

[0103] Furthermore, the image capturing device of this embodiment is also equipped with a digital signage function, so that it is possible to effectively utilize the time when the image capturing device is not in use.

[0104] <First Modification of the Embodiment> In the above-described embodiment, an example in which content is managed by the management server 2 has been described, but it is also possible to store content in advance in the image capture device 1, making the content on the management server 2 unnecessary. In this case, the management server 2 only functions as a storage location for recorded video. Even with this configuration, it is possible to generate a composite video in which the performer and the user overlap naturally.

[0105] <Modification 2 of the embodiment> In the above-described embodiment, a configuration has been described in which selectable content is displayed on the imaging device 1 and the user is allowed to select one. However, without being limited to this, for example, the content to be executed may be specified by having the imaging device 1 read an information recording medium (e.g., a two-dimensional code) containing information specifying the content. The information recording medium may be displayed on the user's terminal (smartphone, tablet, PC), for example, or may be written on a paper medium. Furthermore, in the above-described embodiment, an example has been described in which a contactless IC card reader / writer is used as a payment method for content photography. However, without being limited to this, for example, electronic payment such as QR code payment may also be used.

[0106] Although the present invention has been described above by way of preferred embodiments, the present invention is not necessarily limited to the above-described embodiments, and can be modified and implemented in various ways within the scope of its technical concept. [Explanation of symbols]

[0107] 1. Imaging device 2 Management Server 3. Payment Server 4. Terminal 5 Performer filming equipment 10 Camera 11 Display 12 speakers 13 Operation input section 14 Near Field Communication Department 15 Processing section 16 Transmitter / Receiver 17 Memory section 20 Input section 21 Transmitter / receiver 22 Processing section 23 Memory section 30 Transmitter / Receiver 31 Payment processing section 32 Storage section 150 Management Department 151 Operation control section 152 Payment processing unit 153 Key signal generation unit 154 User image generation unit 155 Video synthesis unit 156 Signage control unit 157 Audio control section 158 Storage location presentation section

Claims

1. A camera that photographs a user, A display unit, A storage unit that stores a first video and a second video, A key signal generation unit that generates a key signal from the captured video of the user captured by the camera, A third video generation unit that generates a third video from the captured video using the key signal of the captured video, A composition unit that determines the depth position of the user in the video from the key signal, determines the order of superimposing the second video and the third video on the first video based on the relationship between the depth position of the user and the reference position of the second video, and superimposes and composes the second video and the third video on the first video according to the order to generate a composite video, A display control unit that displays the composite video on the display unit, A photographing apparatus comprising the same.

2. The composition unit, When the depth position of the user is deeper than the reference position of the second video, superimpose and compose on the first video in the order of the third video and the second video, When the depth position of the user is in front of the reference position of the second video, superimpose on the first video in the order of the second video and the third video, The photographing apparatus according to Claim 1.

3. The composition unit performs determination of the order for each frame, The photographing apparatus according to Claim 2.

4. The first video is a background video, The second video is a video of a performer with the background removed, The third video is a video of the user with the background removed, The photographing apparatus according to Claim 1 or Claim 2.

5. Comprising a booth including a photographing space for the user, The camera and the display unit are provided on the first surface of the booth, On the second surface facing the first surface, a background portion of a specific color is provided, The photographing apparatus according to Claim 1 or Claim 2.

6. Transmission means for transmitting the composite video to a storage location, Storage location presentation means for displaying storage location information indicating the storage location of the composite video on the display unit, The photographing apparatus according to Claim 1 or Claim 2, comprising the same.

7. A booth having a first surface, a second surface facing the first surface, and a photographing space for the user between the first surface and the second surface, A camera provided on the first surface for photographing the user, A display unit provided on the first surface, A background portion of a specific color provided on the second surface, A storage unit that stores a first video and a second video, A key signal generation unit that generates a key signal from the captured video of the user captured by the camera, A third video generation unit that generates a third video from the captured video using the key signal of the captured video; A depth position of the user in the video is determined from the key signal, and an order in which the second video and the third video are superimposed on the first video is determined based on a relationship between the depth position of the user and a reference position of the second video. According to the order, the second video and the third video are superimposed and combined on the first video to generate a combined video; A display control unit that displays the combined video on the display unit; An imaging device comprising: **Claim 8** A computer, An acquisition unit that acquires a first video and a second video from a storage unit, A key signal generation unit that generates a key signal from a captured video of a user captured by a camera, A third video generation means that generates a third video from the captured video using the key signal of the captured video, A depth position of the user in the video is determined from the key signal, and an order in which the second video and the third video are superimposed on the first video is determined based on a relationship between the depth position of the user and a reference position of the second video. According to the order, the second video and the third video are superimposed and combined on the first video to generate a combined video; A display control means for displaying the combined video on a display unit, A program for causing the computer to function as such. **Claim 9** The combining means, When the depth position of the user is deeper than the reference position of the second video, the third video and the second video are superimposed and combined on the first video in this order, When the depth position of the user is in front of the reference position of the second video, the second video and the third video are superimposed on the first video in this order, The program according to claim 8. **Claim 10** The combining unit performs the order determination for each frame. The program according to claim 9. **Claim 11** The first video is a background video, The second video is a video of a performer with the background removed, The third video is a video of the user with the background removed, The program according to claim 8 or claim 9. **Claim 12** A computer, A transmission unit that transmits the combined video to a storage location, A storage location presentation unit that displays storage location information indicating the storage location of the combined video on the display unit, The program according to claim 8 or claim 9 for causing the computer to function as such.

Images