Imaging output device, method for operating imaging output device, and program for operating imaging output device

Abstract

An image capture output device is provided with a processor that performs: when a user performs an image capture command to a first image capture device existing in a first space, the processor performs processing of performing an image capture command to a second image capture device existing in a second space different from the first space; and outputting, from the second imaging device, a second image corresponding to the first image captured by the first imaging device.

Description

Photographic output device, method of operating a photographic output device, and program for operating a photographic output deviceTechnical Field

[0001] The technology of the present invention relates to a photographic output device, a method of operating a photographic output device, and a program for operating a photographic output device Background Art

[0002] Japanese Unexamined Patent Application Publication No. 2009-176025 discloses a virtual space communication system including: a server device that provides a virtual space in which a character that acts as a user avatar is arranged; and a user terminal that is connected to the server device via a network and is used by a user to operate the character. In the virtual space communication system described in Japanese Unexamined Patent Application Publication No. 2009-176025, in the virtual space, the character has a virtual camera for taking pictures in the virtual space, and the user terminal is connected to a printing device that prints images. The user terminal acquires a photographic image in the virtual space obtained by taking a picture using the virtual camera from the server device according to the operation of the character by the user, and causes the printing device to print it

[0003] There is described a method of taking an instant photo of a virtual space with an instant camera operated by an avatar in a virtual space in "[Instant camera (Polaroid camera) for 'Azul2', a virtual avatar with griffin wings and a black panther orc that can take stereoscopic photos! Can also draw handwritten characters! Can also fly! Supports physbone (physical bones)' for VRChat]" retrieved on February 26, 2023, Internet <URL: https: / / booth.pm / ja / items / <3961856> (the "Polaroid" described in the above Internet title is a registered trademark) Summary of the Invention

[0004] One embodiment of the technology of the present invention provides a photographic output device, a method of operating a photographic output device, and a program for operating a photographic output device that can meet the requirements of a user

[0005] Means for Solving the Technical Problem

[0006] The photographic output device of the present invention includes a processor that performs the following processing: when a user gives a photographing command for an image to a first photographing device existing in a first space, giving a photographing command for an image to a second photographing device existing in a second space different from the first space; and outputting a second image corresponding to a first image photographed by the first photographing device from the second photographing device

[0007] Preferably, the processor generates a second image that reproduces the imaging state of the subject of the first image​

[0008] Preferably, the first space is a real space and the second space is a virtual space. The first imaging device is equipped with a sensor that detects the relative position and relative posture of the first imaging device relative to the reference position and reference posture. The processor generates a second image that reproduces the imaging state of the subject in the first image based on the detection results of the sensor when the imaging command is executed.

[0009] Preferably, the first space is the real space and the second space is the virtual space. When the processor commands the first user to take a picture in the real space, the first virtual image of the first user who is active in the virtual space and the second virtual image of the second user who is different from the first user appear in the second image, and only the first user appears in the first image, the second user is composited into the first image.

[0010] Preferably, the first space is a real space, the second space is a virtual space, the first photographic device has the function of printing images, and the processor prints the second image not only from the second photographic device, but also from the first photographic device.

[0011] The operation method of the photographic output device of the present invention includes the following steps: when a user gives a photographic command to a first photographic device existing in a first space, a photographic command is given to a second photographic device existing in a second space different from the first space; and a second image corresponding to the first image captured by the first photographic device is output from the second photographic device.

[0012] The operating procedure of the photographic output device of the present invention causes a computer to perform a process including the following steps: when a user gives a photographic command to a first photographic device existing in a first space, a photographic command is given to a second photographic device existing in a second space different from the first space; and a second image corresponding to the first image captured by the first photographic device is output from the second photographic device. Attached Figure Description

[0013] Figure 1 is a diagram showing a real-time camera, user terminal, and virtual space management server existing in the real space.

[0014] Figure 2 is a block diagram showing the internal structure of a real-time camera.

[0015] Figure 3 is a diagram representing virtual space and virtual avatars.

[0016] Figure 4 is a block diagram showing the computer that constitutes the user terminal and the virtual space management server.

[0017] Figure 5 is a block diagram showing the processing unit of the CPU in a user terminal.

[0018] Figure 6 is a block diagram showing the processing unit of the CPU in the virtual space management server.

[0019] Figure 7 is a diagram representing object information.

[0020] Figure 8 is a diagram representing user information.

[0021] Figure 9 is a diagram illustrating the process of photographic output processing in both real and virtual spaces.

[0022] Figure 10 is a diagram showing the process of photographic output processing in real space and virtual space.

[0023] Figure 11 is a flowchart showing the processing sequence of the virtual space management server.

[0024] Figure 12 is a diagram showing the process of photographic output processing in real space and virtual space in the second embodiment.

[0025] Figure 13 is a diagram illustrating the process of photographic output processing in the virtual space of the second embodiment.

[0026] Figure 14 is a diagram illustrating the process of photographic output processing in real space and virtual space in the second embodiment. Detailed Implementation

[0027] [First Implementation]

[0028] As an example, as shown in Figure 1, in the real space RS, user U possesses a live camera 10 and a user terminal 11. The live camera 10 is used to capture a live photograph 95 (refer to Figure 10). The live camera 10 is connected to the user terminal 11 via short-range wireless communication such as Bluetooth (registered trademark) to enable mutual communication. The real space RS is an example of the "first space" according to the technology of this invention. Furthermore, the live camera 10 is an example of the "first photographic device" according to the technology of this invention. Moreover, the live photograph 95 is an example of the "first image" according to the technology of this invention.

[0029] Specifically, user terminal 11 is a smartphone, tablet, laptop, or desktop PC, etc. Furthermore, user terminal 11 can also be a wearable device such as a head-mounted display with a display function in the part covering the eyes. User terminal 11 can communicate with virtual space management server 13 via network 12. Network 12 is, for example, the Internet or a public communication network (WAN).

[0030] The virtual space management server 13, such as a server computer or workstation, is an example of a "photographic output device" involved in the technology of this invention. Multiple user terminals 11, including multiple users U, are connected to the virtual space management server 13 via a network 12.

[0031] As an example, as shown in Figure 2, the instant camera 10 includes an imaging optical system 20 and an imaging element 21. The imaging optical system 20 has various lenses for imaging the subject onto the imaging element 21. Specifically, the imaging optical system 20 has an objective lens 22 and a focusing lens 23. These objective lenses 22 and focusing lenses 23 are arranged sequentially from the object side (subject side) towards the imaging side (imaging element 21 side). Although simplified in Figure 1, the objective lenses 22 and focusing lenses 23 are actually lens groups composed of multiple lenses. The imaging optical system 20 also has an aperture 24. The aperture 24 is located on the imaging side closest to the imaging optical system 20. In addition, the imaging optical system 20 may have a zoom lens.

[0032] A focusing lens drive mechanism 25 is provided on the focusing lens 23, and an aperture opening adjustment mechanism 26 is provided on the aperture 24. The focusing lens drive mechanism 25 includes, for example: a focusing cam ring that holds the focusing lens 23 and has a cam groove formed on its outer periphery; a focusing motor that moves the focusing cam ring along the optical axis OA by rotating the focusing cam around the optical axis OA; and a driver for the focusing motor, etc.

[0033] Aperture 24, for example, is an iris aperture, which is constructed by combining multiple aperture blades. Aperture 24 adjusts the amount of light passing through by simultaneously moving the aperture blades through a cam mechanism, thereby opening and closing the central opening formed by the inner edges of the aperture blades, i.e., changing the opening degree (hereinafter referred to as the aperture 24 opening degree). The aperture opening degree adjustment mechanism 26 includes an aperture motor for opening and closing the aperture blades and a driver for the aperture motor, etc.

[0034] The focusing motor and the aperture motor are, for example, stepper motors. In this case, the position of the focusing lens 23 on the optical axis OA and the opening of the aperture 24 can be derived from the driving amount of the focusing motor and the aperture motor. Alternatively, a position sensor can be used to detect the position of the focusing lens 23 instead of the driving amount of the focusing motor.

[0035] The electrical components, such as the focusing motor and aperture motor or driver, of the focusing lens drive mechanism 25 and the aperture adjustment mechanism 26 are connected to the control unit 27. The electrical components of the focusing lens drive mechanism 25 and the aperture adjustment mechanism 26 are driven under the control of the control unit 27. More specifically, the control unit 27 sends a drive signal corresponding to the distance to the subject or the brightness of the subject to drive the electrical components of the focusing lens drive mechanism 25 and the aperture adjustment mechanism 26. For example, the control unit 27 sends a drive signal to the driver of the focusing motor of the focusing lens drive mechanism 25, causing the focusing lens 23 to move along the optical axis OA, thereby performing automatic focusing. Furthermore, the control unit 27 sends a drive signal to the driver of the aperture motor of the aperture adjustment mechanism 26, changing the aperture 24, thereby performing automatic exposure adjustment.

[0036] The focusing motor and the aperture motor output drive quantity to the control unit 27. The control unit 27 determines the position of the focusing lens 23 on the optical axis OA and the opening of the aperture 24 based on the drive quantity.

[0037] Imaging element 21 is, for example, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, having an imaging surface 29 for capturing subject light. Imaging surface 29 is formed by a plurality of pixels arranged in a two-dimensional shape. Each pixel accumulates a signal charge corresponding to the subject light and outputs an image signal (voltage signal) corresponding to the signal charge. Imaging element 21 is configured such that the center of imaging surface 29 is aligned with the optical axis OA, and imaging surface 29 is orthogonal to the optical axis OA. Furthermore, the terms "aligned" and "orthogonal" used herein refer not only to complete alignment and orthogonality, but also to alignment and orthogonality in the sense of errors generally permissible in the technical field to which this invention pertains.

[0038] An imaging element driver 30 is connected to the imaging element 21. The imaging element driver 30 is connected to the control unit 27. Under the control of the control unit 27, the imaging element driver 30 supplies vertical scanning signals and horizontal scanning signals to the imaging element 21, thereby controlling the imaging timing based on the subject light of the imaging element 21. Furthermore, the imaging element driver 30 sets the gain assigned to the image signal output from the pixel to correspond to the ISO (International Organization for Standardization) sensitivity.

[0039] A shutter 31 is provided between the camera optical system 20 and the imaging element 21. The shutter 31 is, for example, a focal plane shutter with a front curtain and a rear curtain. A shutter drive mechanism 32 is connected to the shutter 31. The shutter drive mechanism 32 includes an electromagnet and its actuator for holding the front curtain and the rear curtain in place and for releasing the front curtain and the rear curtain to move. The shutter drive mechanism 32 is driven under the control of the control unit 27 to open and close the shutter 31.

[0040] The control unit 27 is connected to various units such as the image input controller 33, image memory 34, and image processing unit 35 via the bus 36. The bus 36 also connects to VRAM (Video Random Access Memory) 37, display control unit 38, communication unit 39, print driver 40, position / pose sensor 41, command receiver 42, etc. Additionally, although not shown in the diagram, the bus 36 also connects to a flash drive control unit that controls the flash device, a media controller that controls the writing of image data to the memory card, etc.

[0041] Image data obtained by capturing light from the subject is input from the imaging element 21 to the image input controller 33. The image input controller 33 outputs the image data to the image memory 34. The image memory 34 is, for example, SDRAM (Synchronous Dynamic Random Access Memory), which temporarily stores the image data.

[0042] The image processing unit 35 reads unprocessed image data from the image memory 34. The image processing unit 35 performs various image processing operations on the image data. These operations include, for example, offset correction, sensitivity correction, pixel interpolation, white balance correction, gamma correction, demosaicing, brightness and chromatic aberration signal generation, contour enhancement, and color correction. The image processing unit 35 then rewrites the processed image data back into the image memory 34.

[0043] Image data, including image data after various image processing, is input from image memory 34 into VRAM 37 for display as an instant preview image (also known as a live view image).

[0044] VRAM37 has a region for storing two consecutive frames of image data. The image data stored in VRAM37 is sequentially rewritten with new image data. VRAM37 then sequentially outputs the new image data from the two consecutive frames of image data to the display control unit 38.

[0045] The display control unit 38 is responsible for converting image data from VRAM 37 into video data and outputting it to the rear monitor 43, functioning as a so-called video encoder. Thus, the user U can visually recognize the real-time preview image through the rear monitor 43. The display frame rate of the real-time preview image is, for example, 30 fps (frames per second).

[0046] The communication unit 39 is responsible for wireless communication with external devices such as the user terminal 11 based on short-range wireless communication such as Bluetooth (registered trademark). Furthermore, the communication unit 39 is also responsible for wired communication with external devices via connection terminals such as USB (Universal Serial Bus) terminals.

[0047] When a command to take a photograph 95 is executed via the release button on the operation unit 28, the image data after various image processing steps of the image memory 34 is output to the print driver 40. A printer 44 is connected to the print driver 40. Under the control of the print driver 40, the printer 44 prints the image data onto a built-in instant imaging film 45 as the instant photograph 95. The instant imaging film 45 with the instant photograph 95 printed on it is output to the outside from a film outlet provided on one side of the instant camera 10. The print driver 40 and the printer 44 are an example of the "image printing function" involved in the technology of the present invention. In addition, the instant imaging film 45 can be either silver halide or thermal.

[0048] The position / attitude sensor 41 consists of an accelerometer and a gyroscope. The position / attitude sensor 41 detects the relative position and relative attitude of the real-time camera 10 relative to a reference position and a reference attitude at a preset sampling interval (e.g., a few milliseconds).

[0049] The position / pose sensor 41 sequentially outputs the detection results (hereinafter referred to as sensor detection results) 90 (refer to FIG. 9) of the relative position and relative pose of the real-time camera 10 with respect to the reference position and reference pose to the communication unit 39. The communication unit 39 wirelessly transmits the sensor detection results 90 to the user terminal 11.

[0050] The command receiving unit 42 receives various operation commands input from the user U via the operation unit 28 and the touch panel 46 integrated with the rear monitor 43. The command receiving unit 42 outputs the received operation commands to the control unit 27 via the bus 36.

[0051] The release button of the operation unit 28 is, for example, a two-stage press button capable of both half-press and full-press operations. A half-press operation of the release button indicates preparation for taking the instant photo 95, while a full-press operation commands the taking of the instant photo 95. When the command to take the instant photo 95 is given by a full-press operation of the release button, the communication unit 39 wirelessly transmits a photography command signal 96 (refer to FIG. 10) indicating that the command to take the instant photo 95 has been given to the user terminal 11.

[0052] The operation unit 28 also includes menu buttons for displaying various setting menus on the rear monitor 43, a cross-shaped key for numerical settings and option switching, and an confirmation button for confirming settings. A touchpad 46 is superimposed on the display surface of the rear monitor 43. The touchpad 46 recognizes various operation commands from the user by detecting contact from the user's finger or a dedicated command device such as a stylus.

[0053] As an example, as shown in Figure 3, the virtual space management server 13 manages the virtual space VS provided to the user U through the user terminal 11. The virtual space VS is a three-dimensional CG (Computer Graphics) space available in a computer, and it is a place where multiple users U communicate with each other while engaging in various social activities such as learning, working, shopping, and playing, separate from the real space RS. Multiple users U act as their respective virtual avatars 50 within the virtual space VS. Figure 2 illustrates the state of virtual avatars 501 and 502 standing side-by-side.

[0054] As an example, as shown in Figure 4, the computers constituting the user terminal 11 and the virtual space management server 13 have essentially the same structure, including a storage device 55, a memory 56, a CPU (Central Processing Unit) 57, a communication unit 58, a display 59, and an input device 60. They are connected to each other via a bus 61.

[0055] Storage device 55 is a hard disk drive built into the computer constituting user terminal 11 and virtual space management server 13 or connected via cable or network. Alternatively, storage device 55 is a hard disk array consisting of multiple hard disk drives. Storage device 55 stores control programs such as operating systems, various application programs (hereinafter referred to as APs), and various data associated with these programs. Alternatively, a solid-state drive can be used instead of a hard disk drive.

[0056] Memory 56 is a working memory used for processing performed by CPU 57. CPU 57 loads programs stored in storage device 55 into memory 56 and executes processing according to the programs. Thus, CPU 57 centrally controls the various parts of the computer. CPU 57 is an example of a "processor" according to the technology of this invention. Alternatively, memory 56 may also be built into CPU 57.

[0057] The communication unit 58 is a network interface for controlling the transmission of various information via network 12, etc. The display 59 shows various screens. These screens have GUI (Graphical User Interface) based operation functions. The computer constituting the user terminal 11 and the virtual space management server 13 receives operation commands from the input device 60 through these screens. The input device 60 includes a keyboard, mouse, touchpad, and microphone for voice input, etc.

[0058] In addition, in the following description, the symbols are distinguished as follows: the components of the computer constituting the user terminal 11 (storage device 55, CPU 57, display 59 and input device 60) are marked with the suffix "A", and the components of the computer constituting the virtual space management server 13 (storage device 55, CPU 57 and input device 60) are marked with the suffix "B".

[0059] As an example, as shown in Figure 5, a virtual space AP 70 is stored in the storage device 55A of the user terminal 11. The virtual space AP 70 is installed on the user terminal 11 by the user U. The virtual space AP 70 is an AP used by the user U to enjoy the virtual space VS in the user terminal 11. When the virtual space AP 70 is started, the CPU 57A of the user terminal 11, together with the memory 56, etc., functions as a browser control unit 72. The browser control unit 72 controls the operation of the dedicated web browser of the virtual space AP 70.

[0060] The browser control unit 72 receives various information from the real-time camera 10, such as sensor detection results 90 and photography command signals 96. It then transmits the received information to the virtual space management server 13. Furthermore, the browser control unit 72 also receives various information from the virtual space management server 13, such as print command signals 99 (refer to FIG. 10). It then transmits the received information to the real-time camera 10. In other words, the user terminal 11 functions as a relay device for transmitting and receiving various information between the real-time camera 10 and the virtual space management server 13.

[0061] The browser control unit 72 reproduces various screens based on various screen data from the virtual space management server 13, and displays the reproduced screens on the monitor 59A. The screen data may be web distribution screen data created using a markup language such as XML (Extensible Markup Language). Alternatively, other data description languages ​​such as JSON (Javascript Object Notation) may be used instead of XML.

[0062] Furthermore, the browser control unit 72 receives various operation commands input by the user U from the input device 60A through various screens. The browser control unit 72 sends the received operation commands along with the terminal ID (Identification Data) used to uniquely identify each user terminal 11 to the virtual space management server 13.

[0063] As an example, as shown in FIG6, an operating program 75 is stored in the storage device 55B of the virtual space management server 13. The operating program 75 is an application program (AP) for enabling the computer constituting the virtual space management server 13 to function as the "photographic output device" according to the technology of the present invention. That is, the operating program 75 is an example of the "operating program of the photographic output device" according to the technology of the present invention.

[0064] The storage device 55B also stores display data 76, object information 77, user information 78, etc. Display data 76 is various 3D CG data required for displaying the virtual space VS. Object information 77 is information about various objects contained in the virtual space VS. Objects are, for example, virtual objects that the virtual avatar 50 can handle, manipulate, ride, wear, etc. Specifically, objects are various characters, buildings, and items configured within the virtual space VS. Characters include the virtual avatar 50 shown in Figure 2 and the roles created by the creator of the virtual space VS. Furthermore, the virtual real-time camera 92 (refer to Figure 9) and the virtual real-time imaging film 98 (refer to Figure 10), described later, are also types of objects. User information 78 is information related to each user U.

[0065] If the working program 75 is started, the CPU 57B and memory 56 of the virtual space management server 13 work together as the receiving unit 80, the processing unit 81 and the sending control unit 82.

[0066] The receiving unit 80 receives various information transmitted from the real-time camera 10 by the user terminal 11, such as sensor detection results 90 and photography command signals 96. Furthermore, the receiving unit 80 receives various operation commands and terminal IDs from the user U of the user terminal 11. The receiving unit 80 outputs the various information from the real-time camera 10 and the various operation commands from the user terminal 11 to the processing unit 81, and outputs the terminal ID to the transmission control unit 82. Additionally, the receiving unit 80 also receives various operation commands from the creator of the virtual space VS via the input device 60B.

[0067] The processing unit 81 is responsible for overall processing related to the management of the virtual space VS, based on various information and operation commands received from the receiving unit 80. The processing performed by the processing unit 81 includes, for example, user U authentication processing, virtual space VS display update processing, object location update processing, display data 76 update processing, object information 77 update processing, user information 78 update processing, and photographic output processing.

[0068] The authentication process for user U involves comparing the user ID and password, which are uniquely used to identify each user U and included in the authentication instruction that serves as the operation command, with the user ID and password stored in user information 78 (refer to Figure 8). The display update process for virtual space VS is as follows: display data 76 corresponding to the movement instruction of the virtual avatar 50 that serves as the operation command is read from storage device 55B, screen data for the virtual space display screen is generated based on the read display data 76, and the generated screen data is output to the transmission control unit 82.

[0069] The object position update process updates the object's position information based on the object's movement instruction, which is used as an operation command. The display data 76 update process is as follows: based on the update instruction of display data 76 as an operation command, the newly created display data 76 by the creator of the virtual space VS is stored in storage device 55B. The object information 77 update process updates the various items constituting the object information 77 based on the update instruction of object information 77 as an operation command, and adds information about newly created objects by the creator of the virtual space VS. The user information 78 update process updates the various items constituting the user information 78 based on the update instruction of user information 78 as an operation command, and adds information about new user U. The photographic output process will be described later in Figures 9 and 10.

[0070] The transmission control unit 82 controls the distribution of various information to the user terminal 11. For example, the transmission control unit 82 controls the distribution of screen data for the virtual space display screen. At this time, the transmission control unit 82 determines the user terminal 11 to which the virtual space display screen is distributed based on the terminal ID from the receiving unit 80. Furthermore, the transmission control unit 82 controls the distribution of the print command signal 99 to the user terminal 11.

[0071] As an example, as shown in Figure 7, object information 77 registers information such as the object type, object name, and ownership information, using object IDs to uniquely identify each object. Among the types, there are various objects, such as virtual avatar 50, virtual instant camera 92 (referred to as instant camera in Figure 7), and virtual instant imaging film 98 (referred to as instant imaging film in Figure 7), for example, cars, clothing, etc. In the name field, if it is virtual avatar 50, the name given to that virtual avatar 50 by user U is registered. The names of items such as virtual instant camera 92 and cars are registered. No name is registered for virtual instant imaging film 98.

[0072] Ownership information indicates which user U owns the object. Specifically, ownership information is the user ID of the user U who owns the object. For example, the virtual avatar 501 is owned by user U with user ID "U0000001".

[0073] Objects can be bought, sold, or transferred. The processing unit 81 updates the object's ownership information, in this case, the user ID, upon the buying, selling, or transfer of the object. In the buying and selling of objects, for example, a currency universally used within the virtual space VS may be used. Furthermore, "buying and selling" includes situations where user U purchases objects prepared by the creator of the virtual space VS, and situations where an object is sold from one user U to another user U.

[0074] Although the illustration is omitted, the virtual real-time imaging film 98 also stores the shooting date and time based on the moment in the virtual space VS, the shooting location in the virtual space VS, etc. Furthermore, the virtual real-time imaging film 98 contains not only images taken by user U, but also images prepared by the creator of the virtual space VS. In the virtual real-time imaging film 98 prepared by the creator of the virtual space VS, the user ID is not registered as ownership information in the initial state.

[0075] Additionally, the virtual avatar 50 represented by object ID "OBA000001" is the virtual avatar 501 shown in Figure 3. Furthermore, the virtual avatar 50 represented by object ID "OBA000002" is the virtual avatar 502 shown in Figure 3.

[0076] As an example, as shown in Figure 8, user information 78 includes user ID, password, camera ID of each real-time camera 10, terminal ID of each user terminal 11, age, and gender. Furthermore, user information 78 includes a registration image 85. The registration image 85 is an image registered by the user U during the installation of the virtual space AP 70, etc., and is an image captured of the user U as the subject.

[0077] Next, referring to Figures 9 and 10, the photographic output processing will be explained.

[0078] As an example, as shown in Figure 9, various events occur in the virtual space VS. In Figure 9, the upgrade of virtual avatar 501 is illustrated as an event. Hereinafter, user U, who owns the user ID "U0000001" of virtual avatar 501, will be labeled as user U1.

[0079] In the event that occurred in the virtual space VS, in the real space RS, user U1 wants to capture their emotions, such as joy, and therefore wishes to take a live selfie 95 using the live camera 10. To begin, user U1 turns on the power to the live camera 10 and sets the reference position and pose of the live camera 10.

[0080] The reference position is, for example, the center of the user U1's chest. The reference posture is, for example, a state in which the front of the live camera 10, with the objective lens 22 exposed, faces the user U1 and the vertical direction of the live camera 10 is aligned with the vertical direction. The setting sequence for the reference position and reference posture can be, for example, as follows: First, after turning on the power to the live camera 10, a guiding sound is played to set the live camera 10 to the reference position and reference posture, or a guiding screen is displayed on the rear monitor 43. In response, the user U1 moves the live camera 10 to the reference position and places it in the reference posture. Then, by selecting the setting button on the operation unit 28, the setting of the reference position and reference posture is completed.

[0081] After the reference position and reference pose are set, user U1 sets up the live camera 10 in front of their face for a selfie. Position / pose sensor 41 detects the relative position and relative pose of live camera 10 relative to the reference position and reference pose, and outputs sensor detection result 90. Communication unit 39 sends sensor detection result 90 to user terminal 11. Sensor detection result 90 includes the camera ID of live camera 10 and the terminal ID of user terminal 11. Browser control unit 72 of user terminal 11 receives sensor detection result 90 and transmits it to virtual space management server 13. Receiving unit 80 of virtual space management server 13 receives sensor detection result 90. Receiving unit 80 outputs sensor detection result 90 to processing unit 81.

[0082] The processing unit 81 determines the relative position and relative posture of the real-time camera 10 currently set up by the user U in the real space RS based on the sensor detection result 90. In the virtual space VS, the processing unit 81 causes the virtual real-time camera 92 to appear in a position and posture corresponding to the determined relative position and posture, and causes the virtual real-time camera 92 to be held by the hand of the virtual avatar 501. At this time, the processing unit 81 generates, for example, a virtual space display screen including an animation that gives the virtual real-time camera 92 a visual effect such as falling from the sky with a shooting star. The virtual real-time camera 92 is an example of the "second photographic device" according to the technology of this invention.

[0083] User U changes the position and orientation of the real-time camera 10 to determine the composition. The processing unit 81 determines the current relative position and orientation of the real-time camera 10 based on the sensor detection results 90 sequentially sent from the user terminal 11. Then, it also changes the position and orientation of the virtual real-time camera 92 in the virtual space VS, thereby changing the display of the virtual avatar 501 with the virtual real-time camera 92.

[0084] With the composition determined, as an example, as shown in Figure 10, user U operates the release button of instant camera 10 to issue a command to take an instant photograph 95. Printer 44, under the control of print driver 40, prints the instant photograph 95 onto instant imaging film 45.

[0085] The communication unit 39 sends the sensor detection result 90 and the photography command signal 96 to the user terminal 11 when a photography command is issued. Similar to the sensor detection result 90, the photography command signal 96 includes the camera ID of the real-time camera 10 and the terminal ID of the user terminal 11. The browser control unit 72 of the user terminal 11 receives the sensor detection result 90 and the photography command signal 96, and transmits them to the virtual space management server 13. The receiving unit 80 of the virtual space management server 13 receives the sensor detection result 90 and the photography command signal 96. The receiving unit 80 outputs the sensor detection result 90 and the photography command signal 96 to the processing unit 81.

[0086] The processing unit 81 receives the sensor detection result 90 and the photography command signal 96, and also issues a photography command for virtual real-time photo 97 to the virtual real-time camera 92. Based on the sensor detection result 90, the processing unit 81 determines the relative position and relative posture of the real-time camera 10 when the photography command was issued. Based on this determination result, the processing unit 81 generates a virtual real-time photo 97 corresponding to the real-time photo 95 captured by the real-time camera 10. More specifically, the processing unit 81 generates a virtual real-time photo 97 that reproduces the subject of the real-time photo 95 (in this case, the imaging state of user U1). Figure 10 illustrates a virtual real-time photo 97 that reproduces the size and orientation of the face of the user U1 captured by a virtual image 501, which is configured to be positioned slightly above and to the right of the user's face. The virtual real-time photo 97 is an example of the "second image" according to the technology of this invention.

[0087] The processing unit 81 prints the virtual real-time photograph 97 onto the virtual real-time imaging film 98. At this time, the processing unit 81 generates a virtual space display screen containing an animation that makes the virtual real-time camera 92 produce a visual effect as if the virtual real-time photograph 97 has been printed out. As described above, the virtual real-time imaging film 98 is treated as an object in the virtual space VS, and the virtual real-time photograph 97 can be viewed, or bought, sold, and transferred.

[0088] The transmission control unit 82 sends a print command signal 99 for the virtual instant photo 97 to the user terminal 11. The browser control unit 72 of the user terminal 11 receives the print command signal 99 and transmits it to the instant camera 10. The communication unit 39 of the instant camera 10 receives the print command signal 99 and outputs it to the print driver 40. Under the control of the print driver 40, the printer 44 prints the virtual instant photo 97 onto the instant imaging film 45.

[0089] Next, as an example, the function of the above structure will be explained with reference to the flowchart shown in FIG11. As shown in FIG5, the CPU57A of the user terminal 11 functions as the browser control unit 72 upon startup of the virtual space AP70. Furthermore, as shown in FIG6, the CPU57B of the virtual space management server 13 functions as the receiving unit 80, processing unit 81, and transmission control unit 82 upon startup of the working program 75.

[0090] As shown in Figures 9 and 10, when an event such as an upgrade of the virtual avatar 50 occurs in the virtual space VS, the user U takes a real-time selfie 95 using the real-time camera 10. At this time, the receiving unit 80 receives the sensor detection result 90 from the position / pose sensor 41 of the real-time camera 10 (step ST100). The sensor detection result 90 is output from the receiving unit 80 to the processing unit 81.

[0091] The processing unit 81 determines the relative position and relative posture of the real-time camera 10 based on the sensor detection result 90. Then, a virtual real-time camera 92 appears with a position and posture corresponding to the determined relative position and posture, and the virtual real-time camera 92 is held by the hand of the virtual avatar 501. If the position and posture of the real-time camera 10 are changed by the user U, the position and posture of the virtual real-time camera 92 are also changed (step ST110).

[0092] When a command to capture a real-time photo 95 is issued by the user U operating the release button of the real-time camera 10, the receiving unit 80 receives the sensor detection result 90 and the capture command signal 96 ("Yes" in step ST120). The sensor detection result 90 and the capture command signal 96 are output from the receiving unit 80 to the processing unit 81. Additionally, in the real-time camera 10, the printer 44 prints the real-time photo 95 onto the real-time imaging film 45.

[0093] The processing unit 81 also issues a photography command for virtual real-time photo 97 to the virtual real-time camera 92. The processing unit 81 determines the relative position and relative posture of the real-time camera 10 when the photography command was issued based on the sensor detection result 90. Then, based on the determination result, a virtual real-time photo 97 is generated that reproduces the imaging state of the user U in the real-time photo 95 (step ST130).

[0094] The processing unit 81 prints the virtual instant photo 97 onto the virtual instant imaging film 98 (step ST140). Then, the transmission control unit 82 sends a print command signal 99 for the virtual instant photo 97 to the user terminal 11 (step ST150).

[0095] The print command signal 99 is received by the communication unit 39 of the instant camera 10. The print command signal 99 is output from the communication unit 39 to the print driver 40. Then, the printer 44 prints the virtual instant photo 97 onto the instant imaging film 45.

[0096] As explained above, the CPU 57B of the virtual space management server 13 functions as the processing unit 81. When user U commands the real-time camera 10 in the real-time space RS to take a real-time photo 95, the processing unit 81 commands the virtual real-time camera 92 in the virtual space VS to take a virtual real-time photo 97. The processing unit 81 outputs the virtual real-time photo 97 corresponding to the real-time photo 95 taken by the real-time camera 10 from the virtual real-time camera 92. Therefore, it is possible to coordinate photography based on the real-time camera 10 in the real-time space RS and photography based on the virtual real-time camera 92 in the virtual space VS. In both the real-time space RS and the virtual space VS, user U's request to enjoy photography without separation of spaces can be fulfilled. Furthermore, it is also possible to fulfill user U's request to retain memories of events such as upgrades experienced in the virtual space VS in the real-time space RS as well as in the real-time space RS.

[0097] The processing unit 81 generates a virtual instant photo 97 that reproduces the imaging state of the user U in the instant photo 95. Therefore, it is possible to obtain a virtual instant photo 97 that is more attractive to the user U. Furthermore, compared to the situation where the imaging state of the user U in the instant photo 95 and the virtual instant photo 97 is completely different, it is easier for the user U to recall the emotions experienced in the virtual space VS.

[0098] A position / pose sensor 41 is provided in the real-time camera 10 to detect the relative position and relative pose of the real-time camera 10 with respect to a reference position and a reference pose. The processing unit 81 generates a virtual real-time photograph 97 that reproduces the imaging state of the user U in the real-time photograph 95 based on the sensor detection result 90 of the position / pose sensor 41 at the time the shooting command is issued. Therefore, compared to generating a virtual real-time photograph 97 without referring to the sensor detection result 90, a virtual real-time photograph 97 that more accurately reproduces the imaging state of the user U in the real-time photograph 95 can be generated. The quality of the virtual real-time photograph 97 can be further improved.

[0099] The instant camera 10 has the function of printing images (printer driver 40 and printer 44). The processing unit 81 not only causes the virtual instant camera 92 to print the virtual instant photo 97, but also causes the instant camera 10 to print the virtual instant photo 97. Therefore, even in the real space RS, one can view the virtual instant photo 97 of the object. Furthermore, by viewing the virtual instant photo 97 in pair with the instant photo 95, it is easier for the user U to recall the emotions experienced in the virtual space VS. Conversely, the virtual instant camera 92 can also print the instant photo 95, and the virtual instant imaging film 98 printed with the instant photo 95 will appear in the virtual space VS.

[0100] [Second Implementation]

[0101] Referring to Figures 12 to 14, the photographic output processing of the second embodiment will be described. Similar to the first embodiment, the second embodiment also uses the case of user U1 taking a selfie with the instant camera 10 in the real-world space RS as an example. User U1 is an example of the "first user" involved in the technology of this invention.

[0102] As an example, as shown in FIG12, in the second embodiment, the instant camera 10 includes an instant photo 95 in the photography command signal 96. The processing unit 81 receives the photography command signal 96 and performs the following processing. First, using well-known facial recognition technology, it checks the face appearing in the instant photo 95 against the face in the registration image 85 of the user information 78, and determines whether only user U1 appears in the instant photo 95. Furthermore, similarly to the first embodiment described above, the processing unit 81 generates a virtual instant photo 97 based on the photography command signal 96, but determines whether only user U1's virtual avatar 501 appears in the virtual instant photo 97. If user U1's virtual avatar 501 and a virtual avatar 50 of a user U different from user U1 appear in the virtual instant photo 97, and only user U1 appears in the instant photo 95, the processing unit 81 further performs the following processing. Furthermore, Figure 12 illustrates a scenario where the virtual avatar 502 owned by user U with user ID "U0000002" appears alongside the virtual avatar 501 in the virtual instant photo 97. Hereinafter, user U with user ID "U0000002" who owns the virtual avatar 502 will be designated as user U2 (refer to Figure 13). User U2 is an example of the "second user" according to the technology of this invention. Furthermore, virtual avatar 501 is an example of the "first virtual avatar" according to the technology of this invention, and virtual avatar 502 is an example of the "second virtual avatar" according to the technology of this invention.

[0103] As an example, as shown in Figure 13, the processing unit 81 refers to the object information 77 to determine the user U (user U2) who owns the virtual avatar 502 that appears in the virtual instant photo 97 along with the virtual avatar 501. Next, the processing unit 81 selects the registration image 85 of the determined user U2 from the user information 78. The processing unit 81 composites the selected user U2 appearing in the registration image 85 into the instant photo 95 sent along with the photography command signal 96, generating a composite instant photo 110. At this time, the processing unit 81 uses a machine learning model specifically for image compositing of people to generate a seamless composite instant photo 110.

[0104] As an example, as shown in Figure 14, the transmission control unit 82 sends a print command signal 99 for the virtual instant photo 97 and the composite instant photo 110 to the user terminal 11. The browser control unit 72 of the user terminal 11 receives the print command signal 99 and transmits it to the instant camera 10. The communication unit 39 of the instant camera 10 receives the print command signal 99 and outputs it to the print driver 40. Under the control of the print driver 40, the printer 44 prints the virtual instant photo 97 and the composite instant photo 110 onto the instant imaging film 45.

[0105] Thus, in the second embodiment, the processing unit 81 displays the virtual image 501 of user U1 who gave the photography command to the real-time camera 10 in the real space RS, and the virtual image 502 of user U2, which is different from user U1, in the virtual real-time photo 97. Furthermore, if only user U1 appears in the real-time photo 95, user U2 is composited into the real-time photo 95 as a composite real-time photo 110. Therefore, user U2, who acts alongside virtual image 501 in the virtual space VS and whose virtual image 502 appears in the virtual real-time photo 97, can be included in the real-time photo of a physical object in a way that makes it seem as if user U2 is present in the real space RS, even though user U2 is not physically present there.

[0106] Since they appear together in the virtual instant photo 97, it can be said that virtual avatar 502 has a close relationship with virtual avatar 501, at least in the virtual space VS. Therefore, by generating a composite instant photo 110 in the real space RS that inherits the close relationship in the virtual space VS, it is possible to further arouse the interest of user U. It can satisfy the request of user U who wants to share the feelings of the activities experienced in the virtual space VS with a close relationship who is not present in the real space RS.

[0107] Furthermore, the second user added to the composite instant photo 110 is not limited to the one person shown in the example. It can also be two or more people. Moreover, the user U to be composited can be limited to users U who have registered as friends. In this way, it is possible to prevent users U who accidentally appear in the virtual instant photo 97 and have no relation to each other from being composited into the instant photo 95.

[0108] The virtual real-time photo 97 can be a slightly zoomed-out composition so that the virtual avatar 50 can be seen by setting up a virtual real-time camera 92.

[0109] Events are not limited to the upgrades shown. They can also occur when a partner joins the team, or when a key item for the game strategy is obtained. Furthermore, events can also be seasonal gatherings such as Christmas parties and New Year's parties held by each virtual space VS, or non-seasonal gatherings such as a user U's birthday party, a virtual space VS's anniversary party, or a virtual culture lecture.

[0110] Alternatively, instead of referencing the sensor detection result 90 based on the position / pose sensor 41, a virtual instant photo 97 that reproduces the imaging state of user U in instant photo 95 can be generated by image analysis of the imaging state of user U's face appearing in instant photo 95. A machine learning model that outputs a virtual instant photo 97 reproducing the imaging state of user U based on the input of instant photo 95 can also be used.

[0111] The description describes setting the first space as the real space RS, the second space as the virtual space VS, the first camera device as a live camera 10, the second camera device as a virtual live camera 92, the first image as a live photograph 95, and the second image as a virtual live photograph 97, but is not limited to this. Conversely, it is also possible to set the first space as the virtual space VS, the second space as the real space RS, the first camera device as the virtual live camera 92, the second camera device as the live camera 10, the first image as a virtual live photograph 97, and the second image as a live photograph 95. In this case, in order to generate a live photograph 95 that reproduces the imaging state of the virtual image 50 that is the virtual live photograph 97, the live camera 10 plays a guiding sound for setting the position and posture of the virtual image 50 to the same imaging state as the virtual live photograph 97, or displays a guiding screen on the rear monitor 43.

[0112] For example, it could be set up as a virtual space VS that recreates a scene from an era older than the real space RS, such as the Edo period. In this case, if the current scene is captured by the real-time camera 10 in the real space RS, then the scene from an older era is captured from the same perspective by the virtual real-time camera 92 in the virtual space VS. Thus, the photography method of the real-time camera 10 is not limited to the user U's selfie.

[0113] Examples are given of various information such as sensor detection results 90, photography command signals 96, and printing command signals 99 being sent and received between the real-time camera 10 and the virtual space management server 13 via the user terminal 11, but this is not limited to these examples. It is also possible to send and receive various information directly between the real-time camera 10 and the virtual space management server 13 without going through the user terminal 11.

[0114] The hardware structure of the computer constituting the virtual space management server 13 can be modified in various ways. For example, to improve processing power and reliability, the virtual space management server 13 can be composed of multiple computers that are separate hardware components. For example, the functions of the receiving unit 80 and the transmitting control unit 82 and the processing unit 81 can be distributed between two computers. In this case, the virtual space management server 13 is composed of two computers. The user terminal 11 can perform some or all of the functions of the virtual space management server 13.

[0115] Thus, the hardware structure of the computer of the virtual space management server 13 can be appropriately changed according to the required performance such as processing power, security, and reliability. Furthermore, not limited to hardware, the virtual space AP 70 and the working program 75, for the purpose of ensuring security and reliability, can of course be dual-stored or distributed across multiple storage devices.

[0116] In the above embodiments, for example, as the hardware structure of the processing unit that performs various processes, such as the browser control unit 72, the receiving unit 80, the processing unit 81, and the transmitting control unit 82, various processors as shown below can be used. Among the various processors, in addition to general-purpose processors such as CPU57A and 57B that execute software (virtual space AP70 and working program 75) and function as various processing units, processors that can change their circuit structure after manufacturing, such as FPGA (Field Programmable Gate Array), are included; processors that have a circuit structure specifically designed for performing specific processes, such as PLD and / or ASIC (Application Specific Integrated Circuit).

[0117] A processing unit can be composed of one of these various processors, or it can be composed of a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA). Furthermore, multiple processing units can also be composed of a single processor.

[0118] As examples of a single processor comprising multiple processing units, firstly, as exemplified by computers such as client and server computers, there are processors that consist of a combination of one or more CPUs and software, and that processor functions as multiple processing units. Secondly, there are approaches such as System-on-Chips (SoCs), which use a single IC (Integrated Circuit) chip to implement the overall system functionality including multiple processing units. Thus, various processing units are constructed using one or more of the aforementioned processors as their hardware architecture.

[0119] Furthermore, as the hardware structure of these various processors, more specifically, circuits composed of circuit elements such as semiconductor elements can be used.

[0120] Based on the above records, one can master the techniques described in the following notes.

[0121] [Note 1]

[0122] A photographic output device includes a processor that performs the following processing: when a user commands a first photographic device existing in a first space to take an image, the processor commands a second photographic device existing in a second space different from the first space to take an image; and outputs a second image from the second photographic device that corresponds to a first image taken by the first photographic device.

[0123] [Note 2]

[0124] According to the photographic output device described in Appendix 1, the processor generates the second image, which reproduces the imaging state of the subject in the first image.

[0125] [Note 3]

[0126] According to the photographic output device described in Appendix 2, wherein the first space is a real space and the second space is a virtual space, a sensor is provided in the first photographic device to detect the relative position and relative posture of the first photographic device relative to a reference position and a reference posture, and the processor generates the second image that reproduces the imaging state of the subject of the first image based on the detection results of the sensor when the photographic command is executed.

[0127] [Note 4]

[0128] According to any one of the appendices 1 to 3, the photographic output device, wherein the first space is a real space, the second space is a virtual space, a first virtual image of a first user who performs the photographic command on the first photographic device in the real space and a second virtual image of a second user different from the first user appear in the second image, and when only the first user appears in the first image, the second user is composited into the first image.

[0129] [Note 5]

[0130] The photographic output device according to any one of appendices 1 to 4, wherein the first space is a real space, the second space is a virtual space, the first photographic device has the function of printing images, and the processor prints the second image not only from the second photographic device but also from the first photographic device.

[0131] The technology of the present invention can also appropriately combine the various embodiments and / or variations described above. Furthermore, it is not limited to the embodiments described above; various structures can be adopted as long as they do not depart from the spirit of the invention. Moreover, in addition to programs, the technology of the present invention also relates to storage media for non-transitory storage of programs and computer program products containing programs.

[0132] The descriptions and illustrations above are detailed explanations of the parts related to the technology of this invention, and are merely one example of the technology of this invention. For example, the descriptions related to the above-described structure, function, effect, and effect are examples of the structure, function, effect, and effect of the parts related to the technology of this invention. Therefore, it is self-evident that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the descriptions and illustrations above without departing from the spirit of this invention. Furthermore, to avoid complications and to facilitate understanding of the parts related to the technology of this invention, explanations of common technical knowledge that do not require special explanation based on the technology enabling the implementation of this invention have been omitted from the descriptions and illustrations above.

[0133] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, when "and / or" is used in this specification to represent three or more items together, the same concept as "A and / or B" can also be applied.

[0134] All documents, patent applications and technical standards described in this specification are incorporated herein by reference to the same extent as the specific documents, patent applications and technical standards described therein.

Claims

1. A photographic output device comprising a processor, the processor performing the following processing: when a user commands a first photographic device existing in a first space to take an image, a second photographic device existing in a second space different from the first space to take an image; and causing a second image corresponding to a first image taken by the first photographic device to be output from the second photographic device.

2. The photographic output device according to claim 1, wherein, The processor generates a second image that reproduces the imaging state of the subject in the first image.

3. The photographic output device according to claim 2, wherein, The first space is a real space, and the second space is a virtual space. The first photographic device is equipped with a sensor that detects the relative position and relative posture of the first photographic device with respect to a reference position and a reference posture. The processor generates the second image, which reproduces the imaging state of the subject in the first image, based on the detection results of the sensor when the photographing command is executed.

4. The photographic output device according to claim 1, wherein, The first space is a real space, and the second space is a virtual space. A first virtual image of a first user who gave the photography command to the first photography device in the real space and a second virtual image of a second user different from the first user appear in the second image. If only the first user appears in the first image, the processor will composite the second user into the first image.

5. The photographic output device according to claim 1, wherein, The first space is a real space, the second space is a virtual space, the first photographic device has the function of printing images, and the processor prints the second image not only from the second photographic device but also from the first photographic device.

6. A method of operating a photographic output device, comprising the following steps: when a user commands a first photographic device existing in a first space to take an image, a second photographic device existing in a second space different from the first space is commanded to take an image; and outputting a second image from the second photographic device corresponding to a first image taken by the first photographic device.

7. An operating procedure for a photographic output device, which causes a computer to perform a process comprising the following steps: when a user commands a first photographic device existing in a first space to take an image, commands a second photographic device existing in a second space different from the first space to take an image; and outputs a second image from the second photographic device corresponding to a first image taken by the first photographic device.

Images