Imaging system and imaging method

Abstract

An embodiment of the present invention is to provide an imaging device, an imaging system and an imaging method wherein the imaging in a real space and the imaging in a virtual space are possible. In an imaging device according to an aspect of the present invention, a processor sets first imaging parameters, which are imaging parameters used when imaging an object in a real space to acquire a first image, and second imaging parameters, which are imaging parameters used when virtually imaging an object in a virtual space to acquire a second image; acquires the first image on the basis of the first imaging parameters in response to a first imaging instruction operation; transmits, in a virtual imaging mode, an imaging request for a virtual image and the second imaging parameters through a communication line to a virtual space management server for managing the virtual space; receives image data of the second image corresponding to the second imaging parameters from the virtual space management server via the communication line; and causes an output device to output at least the second image.

Description

Imaging device, imaging system, and imaging method

[0001] The present invention relates to an imaging device, an imaging system, and an imaging method, and more particularly to an imaging device, an imaging system, and an imaging method that can capture images of objects in real space and objects in virtual space.

[0002] For example, Patent Documents 1 to 3 are known as technologies for capturing an image of an object in a virtual space.

[0003] JP 2020-129272 A JP 2020-140383 A JP 2021-101358 A

[0004] One embodiment of the technique of the present disclosure provides an imaging device, an imaging system, and an imaging method that are capable of imaging in real space and virtual space.

[0005] An imaging device according to a first aspect of the present invention is an imaging device equipped with a processor, in which the processor sets first imaging parameters, which are imaging parameters when an object in real space is imaged using an imaging optical system and an imaging element to obtain a first image, and second imaging parameters, which are imaging parameters when an object in virtual space is virtually imaged to obtain a second image, and in response to a first imaging instruction, obtains the first image based on the first imaging parameters, and when a predetermined condition is satisfied, the imaging device transitions to a virtual imaging mode in which an imaging request for the second image and setting of the second imaging parameters are accepted, and when an instruction to set the second imaging parameters is given in the virtual imaging mode, the processor transmits the second imaging parameters that have been instructed to be set to a virtual space management server that manages the virtual space via a communication line, and in response to the second imaging instruction, transmits an imaging request for at least the second image to the virtual space management server via the communication line, receives image data of the second image corresponding to the transmitted second imaging parameters from the virtual space management server via the communication line, and causes an image output device to output at least the second image corresponding to the image data. In the first aspect, the devices and the like for realizing the functions of the imaging device may be housed in a single housing, or may be housed in multiple separate or separable housings.

[0006] In the imaging device of the second aspect of the present invention, in the first aspect, the processor sets the second imaging parameter in response to an operation on the imaging parameter setting member, and when an operation to change the second imaging parameter is performed on the imaging parameter setting member, the processor transmits the changed second imaging parameter to the virtual space management server.

[0007] In the imaging device of the third aspect, in the first or second aspect, the processor sets, as the second imaging parameter, an actual parameter realized by a real imaging optical system provided in the imaging device, or a virtual parameter realized by a virtual imaging optical system virtually provided in the imaging device.

[0008] The imaging device of the fourth aspect is any one of the first to third aspects and includes a measurement unit that measures a first imaging position and a first imaging direction of the imaging device in real space, and the processor sets a second imaging position and a second imaging direction of the imaging device in virtual space based on the measured first imaging position and first imaging direction, transmits information indicating the second imaging position and information indicating the second imaging direction to the virtual space management server, and receives image data of a second image corresponding to the second imaging position and the second imaging direction from the virtual space management server.

[0009] In the imaging device of the fifth aspect, in the fourth aspect, when the measured first imaging position and first imaging direction change, the processor transmits information indicating the changed second imaging position and information indicating the changed second imaging direction to the virtual space management server.

[0010] In the imaging device of the sixth aspect, in the fourth or fifth aspect, the processor causes the virtual space management server to fix the second imaging position and the second imaging direction in the virtual space in response to a user's instruction to the imaging device.

[0011] In the imaging device of the seventh aspect, in any one of the first to sixth aspects, the processor sends an imaging request for the second image to the virtual space management server after a predetermined time has elapsed since the second imaging instruction was issued.

[0012] The imaging device of the eighth aspect is any one of the first to seventh aspects, and includes a first memory unit that stores avatar information indicating an avatar of a user of the imaging device, and the processor transmits the stored avatar information to a virtual space management server, causing the virtual space management server to display the user's avatar in the virtual space.

[0013] The imaging device of the ninth aspect is the eighth aspect, and is provided with a second memory unit that stores data indicating the three-dimensional shape of the imaging device, and the processor transmits the stored data to a virtual space management server, and the virtual space management server displays the three-dimensional shape of the imaging device in association with the avatar near the avatar in the virtual space.

[0014] The imaging device according to a tenth aspect is the imaging device according to any one of the first to ninth aspects, wherein the second image is a still image or a moving image.

[0015] An imaging system according to an eleventh aspect is an imaging system having an imaging device and a server connection device, wherein the imaging device includes a first processor, and the first processor sets first imaging parameters which are imaging conditions when an object in real space is imaged using an imaging optical system and an imaging element to acquire a first image, and second imaging parameters which are imaging conditions when an object in virtual space is virtually imaged to acquire a second image, and acquires the first image based on the first imaging parameters in response to a first imaging instruction, and when predetermined conditions are satisfied, the imaging device and the server connection device transition to a virtual imaging mode in which an imaging request for the second image and setting of the second imaging parameters are accepted, and the first processor In the case where an instruction to set second imaging parameters is given, the instructed second imaging parameters are transmitted to a virtual space management server that manages the virtual space via a server connection device, and in response to the second imaging instruction, an imaging request for at least a second image is transmitted to the virtual space management server via the server connection device, the server connection device having a second processor and an image output device, and the second processor, in a virtual imaging mode, transmits the imaging request for the second image and the second imaging parameters to the virtual space management server via a communication line, receives image data of the second image corresponding to the transmitted second imaging parameters from the virtual space management server via the communication line, and causes the image output device to output at least the second image corresponding to the received image data.

[0016] The imaging system according to a twelfth aspect is the eleventh aspect, wherein the server connection device is a goggle-type device worn by a user, and includes a display device that displays the second image as the image output device.

[0017] In the imaging system of the thirteenth aspect, in the eleventh or twelfth aspect, the system further includes a virtual space management server, and the virtual space management server generates image data of the second image corresponding to the second imaging parameters in accordance with the imaging request and imaging conditions of the second image received from the server connection device, and transmits the generated image data to the server connection device.

[0018] An imaging system according to a fourteenth aspect is the thirteenth aspect, wherein the virtual space management server causes the virtual space management server to display a second image in the virtual space in response to an operation by a user of the imaging device.

[0019] In the imaging system of the 15th aspect, in the 13th or 14th aspect, the virtual space management server transmits image data of a second image generated in response to an imaging request received from one imaging device connected to the virtual space management server to another imaging device in response to a request from the other imaging device connected to the virtual space management server.

[0020] According to a sixteenth aspect, there is provided an imaging method executed by an imaging device including a processor, wherein the processor sets first imaging parameters, which are imaging parameters for capturing an image of an object in real space using an imaging optical system and an imaging element to acquire a first image, and second imaging parameters, which are imaging parameters for virtually capturing an image of an object in a virtual space to acquire a second image, in response to a first imaging instruction, to acquire the first image based on the first imaging parameters, and the imaging device transitions to a virtual imaging mode in which, if a predetermined condition is satisfied, an imaging request for the second image and setting of the second imaging parameters is accepted, and when an instruction to set the second imaging parameters is received in the virtual imaging mode, the processor transmits the second imaging parameters to a virtual space management server that manages the virtual space via a communication line, and in response to the second imaging instruction, transmits an imaging request for the second image to the virtual space management server via the communication line, receives image data of the second image corresponding to the transmitted second imaging parameters from the virtual space management server via the communication line, and outputs at least the second image corresponding to the image data to an image output device. The imaging method according to the sixteenth aspect may further include configurations similar to those of the second to tenth aspects.

[0021] An imaging method according to a seventeenth aspect is an imaging method executed by an imaging system having an imaging device including a first processor, and a server connection device including a second processor and an image output device, wherein the first processor sets first imaging parameters which are imaging conditions when an object in real space is imaged using an imaging optical system and an imaging element to obtain a first image, and second imaging parameters which are imaging conditions when an object in a virtual space is virtually imaged to obtain a second image, and acquires the first image based on the first imaging parameters in response to a first imaging instruction, and the imaging device and the server connection device issue an imaging request for the second image and the setting of the second imaging parameters when predetermined conditions are satisfied. The first processor transitions to a virtual imaging mode in which it accepts settings for the second imaging parameters, and when there is an instruction to set the second imaging parameters in the virtual imaging mode, the first processor transmits the second imaging parameters that have been instructed to be set to a virtual space management server that manages the virtual space via the server connection device, and transmits an imaging request for the second image to the virtual space management server via the server connection device in response to the second imaging instruction, and the second processor transmits the imaging request for the second image to the virtual space management server via a communication line in the virtual imaging mode, receives image data of the second image corresponding to the second imaging parameters from the virtual space management server via the communication line, and causes the image output device to output at least the second image corresponding to the received image data.

[0022] FIG. 1 is a diagram illustrating the configuration of an imaging system according to a first embodiment. FIG. 2 is a diagram illustrating the configuration of a user system according to the first embodiment. FIG. 3 is a diagram illustrating the configuration of an imaging device. FIG. 4 is a diagram illustrating the functional configuration of a processor of the imaging device. FIG. 5 is a diagram illustrating the configuration of goggles. FIG. 6 is a diagram illustrating the configuration of a virtual space management server. FIG. 7 is a flowchart illustrating processing of an imaging method according to the first embodiment. FIG. 8 is a flowchart (continuation of FIG. 7) illustrating processing of an imaging method according to the first embodiment. FIG. 9 is a diagram illustrating a state in which three-dimensional shapes of an avatar and an imaging device are displayed in a virtual space. FIG. 10 is a diagram illustrating an example of mode transition based on a user's action. FIG. 11 is a diagram illustrating a state in which a virtual live view image is displayed on goggles. FIG. 12 is a diagram illustrating how a second imaging parameter is set or changed in a virtual space. FIG. 13 is a diagram illustrating an example of a virtual live view image when the focal length of the imaging device is changed. FIG. 14 is a diagram illustrating an example of a virtual live view image when the exposure is changed. FIG. 15 is a diagram illustrating how virtual images are displayed in a virtual space. FIG. 16 is a diagram illustrating how an imaging position and an imaging direction are fixed in a virtual space. Fig. 17 is a diagram showing how imaging is performed using the self-timer function. Fig. 18 is another diagram showing how imaging is performed using the self-timer function. Fig. 19 is a diagram showing a modified example of a user system. Fig. 20 is a diagram showing the configuration of a user system according to a second embodiment. Fig. 21 is a diagram showing the configuration of an imaging device according to the second embodiment.

[0023] An embodiment of an imaging device, an imaging system, and an imaging method according to the present invention is as follows: In the description, reference will be made to the accompanying drawings as necessary.

[0024] [First embodiment] [Configuration of imaging system] Fig. 1 is a diagram showing the configuration of an imaging system according to the first embodiment. As shown in the figure, an imaging system 1 (imaging system) includes a user system 10 (imaging system) and a virtual space management server 20 (virtual space management server), which are connected via a network NW (communication line) such as the Internet. There may be one or more user systems 10.

[0025] [Configuration of User System] Fig. 2 is a diagram showing the configuration of the user system according to the first embodiment. As shown in the figure, the user system 10 includes an imaging device 100 (imaging device), goggles 200 (server connection device, goggle-type device), and a router 300, and the goggles 200 are connected to a network NW via the router 300. The imaging device 100 and the goggles 200 can be connected via wired communication or wireless communication such as Bluetooth (registered trademark), and the goggles 200 and the router 300 can be connected via wired communication or wireless communication such as Wi-Fi (registered trademark).

[0026] 3 is a block diagram showing the configuration of the imaging device 100. As shown in the figure, the imaging device 100 includes an imaging optical system 102 (imaging optical system, actual imaging optical system), an imaging element 104 (imaging element), a processor 106 (processor), a display 108 (display device), an operation unit 110 (imaging parameter setting member), a flash ROM (ROM is read only memory; a non-transitory and tangible recording medium), a RAM 114 (RAM is random access memory), a memory card 116, a motion sensor 118 (measurement unit), and a wireless communication interface 120.

[0027] The imaging optical system 102 includes optical components such as an aperture stop in addition to lenses (not shown) (such as an imaging lens, a zoom lens, and a focus lens), and forms an optical image of a subject (object) existing in real space on the imaging element 104. As will be described in detail later, a real image (first image; a still image or a moving image) of the subject is generated from this optical image through processing by the processor 106. The imaging element 104 can be of various types, such as a complementary metal-oxide semiconductor (CMOS) type or a charge-coupled device (CCD) type. The generated real image can be displayed on the display 108. The display 108 is configured by a device such as a liquid crystal monitor.

[0028] The operation unit 110 is composed of operation members such as a release button, a cross button, a MENU / OK button, a dial, a switch, and a lever for setting the self-timer, and can be used in common to set first imaging parameters, which are imaging parameters when acquiring an actual image, and second imaging parameters, which are imaging parameters when acquiring a virtual image. The setting of imaging parameters will be described in detail later. The operation unit 110 can also be used for operations such as issuing imaging instructions (image acquisition requests) and setting the self-timer.

[0029] The configurations and functions of the processor 106, flash ROM 112, and RAM 114 will be described later.

[0030] The memory card 116 is a non-transitory, tangible recording medium that can be attached to or detached from the imaging device 100, and can record information such as captured real images (first images) and virtual images (second images). The memory card 116 can be configured using, for example, a flash memory as a storage element. Note that avatar information indicating an avatar of a user of the imaging device and data indicating the three-dimensional shape of the imaging device may be stored in the flash ROM 112 (first storage unit, second storage unit) and / or the memory card 116 (first storage unit, second storage unit).

[0031] The motion sensor 118 includes sensors such as an acceleration sensor and an angular velocity sensor, and can output information indicating the imaging position (first imaging position) and imaging direction (first imaging direction) in real space of the imaging device 100. In addition to these sensors, the imaging device 100 may also be equipped with a GPS receiver (GPS: Global Positioning System).

[0032] The wireless communication interface 120 includes an antenna for wireless communication such as Bluetooth (registered trademark), and is used for communication with the goggles 200 .

[0033] 4 is a diagram showing the functional configuration of the processor 106 (processor, first processor) of the imaging device 100. As shown in the figure, the processor 106 has functions as an image acquisition unit 106A, an imaging parameter setting unit 106B, a measurement unit 106C, a mode control unit 106D, a communication control unit 106E, a display control unit 106F, and a recording control unit 106G. If the imaging device 100 has a GPS receiver, the processor 106 may also have a function as a positioning unit that performs positioning based on GPS signals.

[0034] The processor 106 is composed of various processors and electrical circuits, such as a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), and a programmable logic device (PLD). When these processors and electrical circuits execute software (programs), computer-readable code for the software to be executed (e.g., various processors and electrical circuits constituting the processor, and / or a combination thereof) is stored in a non-transitory tangible recording medium, such as a flash ROM 112, and the computer accesses the software. The software stored in the non-transitory tangible recording medium includes the imaging program of the present invention (a program for executing the imaging method of the present invention) and data used in its execution. The code may be recorded in a non-transitory tangible recording medium, such as a ROM or an EEPROM (Electronically Erasable and Programmable Read Only Memory), instead of the flash ROM 112. Note that this "non-transitory tangible recording medium" does not include non-tangible recording media such as carrier signals or propagation signals themselves (the same applies to processors 202 and 22, described below). When processing using software, the RAM 114 is used as a temporary storage area or a working area.

[0035] The processing using the processor 106 having the above-described configuration will be described in detail later.

[0036] [Goggle Configuration] FIG. 5 is a diagram showing the configuration of the goggles 200. As shown in the figure, the goggles 200 include a processor 202, a display 204, an operation unit 206, a flash ROM 208, a RAM 210, a wireless communication interface 212, a microphone 214, and a speaker 216. The goggles 200 communicate with the image capture device 100 and with the virtual space management server 20 via the router 300 and the network NW. That is, the goggles 200 function as a server connection device and an image display device (image output device) for virtual images, and can be configured as a VR goggle-type device (VR: Virtual Reality) worn on the user's head. The goggles 200 may further include a memory card and / or a motion sensor.

[0037] The processor 202 (processor, second processor) of the goggles 200 can be configured using a CPU, GPU, etc., similar to the processor 106 of the imaging device 100 described above, and executes various programs such as the imaging program according to the present invention (a program that executes the imaging method according to the present invention). When executing the programs and data stored in the flash ROM 208, the programs and data can be used, and when processing, the RAM 210 can be used as a temporary storage area or a working area.

[0038] The display 204 is used to display various information such as a virtual image (second image), imaging parameters, an operation menu, and messages to the user. Furthermore, the operation unit 206 of the virtual image goggles 200 may be used in combination with the operation unit 110 of the imaging device 100 for operations such as setting imaging parameters (second imaging parameters), switching (transitioning) imaging modes, requesting imaging, and displaying images. Even if the operation unit 110 of the imaging device 100 is difficult to operate because the goggles 200 cover the eyes, using the operation unit 110 and the operation unit 206 in combination improves operability.

[0039] The goggles 200 also allow for operation by voice input via a microphone 218 and a speaker 220, for listening to sounds in the virtual space, and for conversation with other users connected (participating) in the virtual space. Messages to the user may be output via these devices. The processor 202 is capable of voice recognition of voice input via the microphone 218.

[0040] The processing using the goggles 200 having the above-described configuration will be described in detail later.

[0041] 6 is a diagram showing the configuration of the virtual space management server 20. As shown in the figure, the virtual space management server 20 includes a processor 22 (a third processor), a ROM 24, a RAM 26, and a database 28, and manages the positions of objects and avatars in the virtual space, audio and lighting (brightness, saturation, etc.) in the virtual space, and changes thereto.

[0042] The processor 22 can be configured using a CPU, GPU, etc., similar to the processor 106 of the imaging device 100 and the processor 202 of the goggles 200 described above, and executes various programs, such as an imaging program according to the present invention (a program for executing an imaging method according to the present invention). These programs can be executed using programs and data stored in a ROM 24 (a non-transitory, tangible recording medium), and the RAM 26 can be used as a temporary storage area or work area during processing. The database 28 includes various types of magneto-optical recording devices, semiconductor recording devices, etc. (non-transitory, tangible recording media), and can record and output various information necessary for managing the virtual space (e.g., virtual space environment setting information, object placement, user accounts, etc.).

[0043] [Image Capture by Image Capture System] The image capture system 1 configured as described above can capture a real image (first image) and a virtual image (second image), as will be described below.

[0044] [Acquisition of Real Images] The imaging device 100 can acquire real images, similar to general imaging devices. For example, the processor 106 (processor, first processor) of the imaging device 100 sets first imaging parameters, which are imaging conditions when capturing an object in real space and acquiring a real image (first image), in response to a user's operation. The first imaging parameters may include at least one of focal length, focal length, shutter speed, aperture value, and image quality. They may also include the imaging mode (photography scene), the presence and content of image processing, the presence or absence of a flash, etc. Furthermore, "image quality" may include at least one of resolution, sharpness, ISO sensitivity (ISO: International Organization for Standardization), and white balance, and may further include settings that can be set when capturing a real image, such as film simulation, tone, saturation, contrast, hue, noise, dynamic range, and brightness (the same applies to virtual images, described below).

[0045] The user can perform a setting operation of the first imaging parameter using a dial, button, or the like (not shown) of the operation unit 110. The setting operation may be a zoom operation and / or a focus operation for the imaging optical system 102 (for example, rotation of a zoom ring or a focus ring on a lens barrel, forward / backward movement, etc.), or, if the display 108 is a touch panel type, an operation via the touch panel may be performed.

[0046] In response to a user operation (first imaging instruction, imaging request) such as pressing a release button (not shown), the imaging device 100 can capture an image of an object in real space using the imaging optical system 102 and the imaging element 104 based on first imaging parameters to obtain a real image (first image).

[0047] [Capturing a Virtual Image] [Basic Mode of Capturing a Virtual Image] FIGS. 7 and 8 are flowcharts showing the procedure of a basic mode of capturing a virtual image.

[0048] [Pairing of Imaging Device and Goggles] The mode control unit 106D (processor 106) sets the imaging device 100 to the virtual space connection mode, and the processor 202 sets the goggles 200 to the virtual space connection mode (step S100). The setting may be triggered, for example, by detecting an operation on the operation unit 110, detecting a predetermined action (such as shaking the imaging device 100) with the motion sensor 118, detecting an operation on the operation unit 206, or recognizing a voice input to the microphone 214 (in this case, it is assumed that the processor 202 of the goggles 200 has a voice recognition function), etc.

[0049] Once the imaging device 100 and the goggles 200 are set to the virtual space connection mode, the mode control unit 106D and the processor 202 pair the imaging device 100 and the goggles 200 (step S100). Communication between the imaging device 100 and the goggles 200 can be performed via wireless communication such as Bluetooth (registered trademark) via the wireless communication interface 120 and the wireless communication interface 212. The pairing process is similar to pairing a personal computer (PC) or smartphone with a peripheral device (printer, speaker, earphones, etc.). If pairing is successful (YES in step S120), the process proceeds to step S130. From this point on, the imaging device 100 operates as a controller for the goggles 200. Note that in the example of FIG. 7 , pairing is repeated until successful (until YES in step S120), but the process may also end if pairing is not successful within a predetermined number of times or within a predetermined time period.

[0050] [Transmission of Avatar Information] If pairing in step S120 is successful, the user system 10 transmits avatar information indicating the avatar of the user of the imaging device 100 to the virtual space management server 20 (step S130). Specifically, the communication control unit 106E (processor 106) and the processor 202 transmit the avatar information to the virtual space management server 20 via the goggles 200 (server connection device) and the router 300 (step S130). As described above, the avatar information can be stored in the flash ROM 112 (first storage unit) and / or the memory card 116 (first storage unit). The avatar information may be created by the user or downloaded (purchased). The avatar may be a representation of a human body, an animal, a plant, or another object. The avatar may have the user's "emotions" or "facial expressions," or attributes equivalent thereto.

[0051] In the imaging system 1, the motion sensor 118 of the imaging device 100 detects the user's movements, allowing the avatar to change its position and posture in the virtual space to reflect those movements. Furthermore, in order to accurately reflect the user's movements in the avatar, the user system 10 may further include motion sensors (acceleration sensors, angular velocity sensors, etc.) attached to the user's hands, feet, etc. The user system 10 can be configured so that these motion sensors communicate with the virtual space management server 20 via the goggles 200 (server connection device).

[0052] [Transmission of 3D Shape Data of Imaging Device] [Significance of 3D Shape Data] In recent years, taking photos of people, landscapes, and the like in virtual spaces (virtual reality spaces, VR spaces) has become an established use case, creating a need for 3D shape data of digital cameras (imaging devices) that can be used for avatars in virtual spaces. While polygon models of digital cameras that exist in real space have begun to be provided as virtual space items, these are low-polygon models used as props to decorate avatars in virtual spaces. This trend is likely to lead to an increasing shift in virtual space avatars from the property of specific platform providers to personal possessions, and items used to decorate one's avatars will become personal assets. This could accelerate the distribution of virtual space data for not only cameras but also various objects (items) surrounding one's daily life. In light of this situation, the present embodiment enables the 3D shape data of an imaging device 100 to be displayed in virtual space.

[0053] The communication control unit 106E (processor 106; first processor) transmits data indicating the three-dimensional shape of the image capture device 100 to the virtual space management server 20 via the goggles 200 (server connection device), the router 300, and the network NW (communication line) (step S140). This three-dimensional shape data can be stored in the flash ROM 112 (second storage unit) and / or the memory card 116 (second storage unit). The communication control unit 106E can transmit the three-dimensional shape data using, for example, a Web API (Application Programming Interface). Note that this Web API preferably communicates status information (such as power ON / OFF status, tilt information (attitude around three axes), focus and zoom information, release status, etc.) and second image capture parameters of the image capture device 100 in addition to the three-dimensional shape data.

[0054] The three-dimensional shape data can be provided by a company that manufactures and sells the imaging device 100. As described below in the section "Configuration of Virtual Space," the data may be made available to users within the virtual space. Furthermore, the three-dimensional shape data may accurately represent the shape, size, color, texture, etc. of the imaging device 100, or may be a pseudo- or deformed representation of the imaging device 100, which differs from the actual imaging device 100. For example, if the actual imaging device 100 is a popular or low-cost model, displaying data of an actual high-end model or data that evokes a high-end model can satisfy a user's desire to "own a high-end camera in the virtual space." Data for a model that does not actually exist (a model that exists only in the virtual space) may also be used. Furthermore, the three-dimensional shape data may display the brand name and / or logo of the imaging device 100. These displays satisfy the user's desire to own the device and also provide advertising effects for the manufacturer or seller.

[0055] [Configuration of Virtual Space] The virtual space may be a single space, or may be divided into multiple spaces (for example, a personal room or conference room, a virtual store such as a retail store or restaurant, a facility such as a live music venue, a city, etc.). Furthermore, the virtual space management server 20 may provide within the virtual space "a virtual store where users can obtain (download, purchase, etc.; the same applies below) information indicating avatars," "a virtual store that displays or exhibits three-dimensional shape data of an imaging device and allows users to obtain that data," or "a virtual exhibition room that exhibits captured virtual images or virtual prints thereof and allows users to obtain those images" (see FIG. 18), which will be described later.

[0056] [Output of Virtual Space Status] The communication control unit 22A and virtual space management unit 22B (third processor) of the virtual space management server 20 transmit the status of the virtual space to the user system 10 via the network NW (communication line) and output it to the goggles 200 (image output device) (step S150). The "status of the virtual space" may include the background, object, and avatar positions, shapes, and movements in the virtual space, as well as audio and lighting (brightness, saturation, etc.) within the virtual space and their temporal changes. The "output" may include display on the display 204 (image output device, image display device) and audio output from the speaker 216. The communication control unit 22A and virtual space management unit 22B may display the avatar of the user of the imaging device 100 as well as the avatars of other users participating in the virtual space. The virtual space management server 20 preferably updates the above-mentioned output at a predetermined rate.

[0057] The virtual space management server 20 can display an image (such as that shown in FIG. 15 described later) showing a bird's-eye view of the virtual space from a predetermined position (viewpoint) on the user system 10, similar to a game played on a smartphone or PC. The virtual space management server 20 can change the viewpoint (position and / or direction) in response to a user's operation or movement within the virtual space, and can change the image and / or sound that is output in response to a change in the viewpoint.

[0058] Users participating in the virtual space can converse (communicate with other users) with other users under the control of the virtual space management unit 22B, thereby enabling them to hold conferences, etc. in the virtual space. Note that conversations in the virtual space can be conducted not only by normal voice conversations via the microphone 214 and speaker 216, but also by text chats between specific users, or by displaying comments in text in the virtual space using speech bubbles or the like around avatars (which may be visible to all participants in the virtual space), etc.

[0059] It is preferable that the virtual space management server 20 continuously performs the above-mentioned "output of the state of the virtual space" at a predetermined rate.

[0060] [Display of Avatar and Three-Dimensional Shape Data of Imaging Device] When the user of the imaging device 100 instructs transmission of the above-mentioned three-dimensional shape data, the virtual space management server 20 displays the three-dimensional shape of the imaging device 100 in association with the avatar near the avatar in the virtual space (step S150). Specifically, the virtual space management unit 22B and the communication control unit 22A update the virtual space to a state in which "the three-dimensional shape of the imaging device 100 is displayed in association with the avatar near the avatar," and transmits information about the virtual space to the user system 10 to display it on the display 204 of the goggles 200.

[0061] Here, "displaying nearby" may mean, for example, displaying the three-dimensional shape data within a predetermined distance from the avatar, or displaying it on a predetermined part of the avatar, such as the avatar's hand or chest. Such a display can inform (appeal to) other users that the user has an imaging device and can capture virtual images. When displaying an avatar, the virtual space management server 20 may also display the user's ID, nickname, etc., near the avatar in association with the avatar. This allows participants in the virtual space to understand who the user represented by the avatar is.

[0062] 9 is a diagram showing a state (a virtual live view image 500, described later) in which a user's (the user of the imaging device 100's) own avatar 600, three-dimensional shape data 610 of the imaging device 100, and another user's avatar 700 are displayed in a virtual space. In the example shown in the figure, the three-dimensional shape data 610 is displayed in the hand of the avatar 600. This type of display is one example of "displaying data indicating the three-dimensional shape of the imaging device in association with the avatar, near the avatar."

[0063] The virtual space management server 20 may turn on or off the display of the three-dimensional shape and ID of the image capture device 100 in response to a user instruction (operation, movement, voice input, etc.). If the connection between the image capture device 100 and the goggles 200 is interrupted, the image capture device 100 and the goggles 200 may attempt to reconnect, or the goggles 200 may notify the virtual space management server 20 of this and delete the drawing of the image capture device data from the virtual space.

[0064] [Transition to Virtual Imaging Mode] When the mode control unit 106D (processor 106; first processor) and the processor 202 (second processor) determine that the predetermined conditions are met (YES in step S160), the imaging device 100 and the goggles 200 transition to a virtual imaging mode in which an imaging request for a virtual image (second image) and settings of second imaging parameters are accepted (step S170). The second imaging parameters are imaging conditions for virtually capturing an image of an object in a virtual space to acquire the virtual image (second image).

[0065] [Examples of Mode Transition Conditions] Examples of mode transition conditions are listed below: However, the mode transition conditions in the present invention are not limited to these examples.

[0066] An example of a transition condition is when the user holds the imaging device 100 in a predetermined position or posture. For example, the user holds the imaging device 100 close to the user's head or assumes a posture as if looking into the viewfinder. FIG. 10 is a diagram showing an example of mode transition based on a user's action. Part (a) of FIG. 10 shows a user 2 wearing the goggles 200, and part (b) of the same figure shows a state in which the user 2 holds the imaging device 100 close to his or her head and assumes a posture as if looking into the viewfinder. Such a movement of the user 2 can be detected by the motion sensor 118 or the like. However, if the goggles 200 are equipped with a motion sensor or if the user system 10 includes motion sensors attached to the user's hands or feet, the movement may be detected by the motion sensors.

[0067] When these motion sensors detect movement, the mode control unit 106D and the processor 202 determine that "predetermined conditions have been met," and the imaging device 100 and the goggles 200 transition to a virtual imaging mode in which they accept imaging requests for a virtual image (second image) and settings of imaging parameters for the virtual image (second imaging parameters).

[0068] Other examples of the transition condition include the user performing an action such as shaking or rotating the imaging device 100. Also, other examples include the user performing a predetermined operation (such as operating a specific button or switch) on the operation unit 110 of the imaging device 100 and / or the user performing a predetermined operation on the operation unit 206 of the goggles 200.

[0069] The transition condition is not limited to an operation on the imaging device 100 or the goggles 200. Other examples of the transition condition include (1) the processor 202 of the goggles 200 recognizing a specific phrase (e.g., "switch to virtual imaging mode," "take a photo," etc.) through voice recognition via the microphone 214, or (2) another user in the virtual space uttering a specific phrase (e.g., "let's take a group photo") and the processor 202 recognizing the phrase through voice recognition.

[0070] The transition condition may be a combination of the above-mentioned conditions. For example, if another user in the virtual space asks a predetermined question such as "Let's take a group photo," and the user of the imaging device 100 responds with a predetermined reply such as "Okay," "YES," or "OK," or performs a predetermined action and / or operation, the mode control unit 106D and the processor 202 can determine that "the predetermined condition is met." Similarly, the condition for ending the virtual imaging mode can be determined based on the position and / or orientation of the imaging device 100, an operation, a movement, voice recognition, or a combination thereof.

[0071] [Acquisition and Transmission of Second Imaging Parameters] When the imaging device 100 and the goggles 200 transition to the virtual imaging mode, the processor 106 and the processor 202 acquire the second imaging parameters and transmit them to the virtual space management server 20 (step S180 in FIG. 8). The second imaging parameters include at least one of the focal length, the focusing distance, the shutter speed, the aperture value, and the image quality. Note that the acquired and transmitted second imaging parameters may be initially set parameters or may be parameters that have been changed from the initially set state (see step S200 described later).

[0072] [Switching to First-Person Perspective Image According to Mode Transition] When the image capture device 100 and the goggles 200 transition to the virtual image capture mode, the virtual space management server 20 is notified of the mode transition, and the virtual space management server 20 performs output according to the virtual image capture mode in response to this notification. Specifically, the virtual space management server 20 switches the display on the display 204 of the goggles 200 to a Point of View (a first-person perspective image seen from the avatar of the user of the image capture device 100 or the image capture device in the virtual space).

[0073] [Updating First-Person-Perspective Image (Virtual Live View Image)] Processor 22 (third processor) of virtual space management server 20 generates a first-person-perspective image according to the received second imaging parameters and displays it on display 204 (step S190). Processor 22 also updates this first-person-perspective image at a predetermined rate and transmits the updated first-person-perspective image to user system 10 for display on display 204. The image updated in this manner is similar to a live view image captured in real space, and hereinafter this image is referred to as a "virtual live view image."

[0074] FIG. 11 is a diagram showing a virtual live view image 500 (an image at a specific time) displayed on the display 204. The virtual live view image 500 includes an area 502 and an area 504. The area 502 displays the brand name, model number, and characteristics of the attached optical system (model number and focal length in the example of FIG. 11 ) of the imaging device indicated by the three-dimensional shape data, while the area 504 displays the second imaging parameters set at the time of display. In the example of FIG. 11 , the white balance is AUTO (automatic), the shutter speed is 1 / 60 sec, the aperture is 1.8, and the ISO sensitivity is 400. Avatars 700 of other users (four people) are also displayed in the virtual live view image 500.

[0075] [Updating Virtual Live View Image in Accordance with Changes in Movement and Direction] The processor 106 of the imaging device 100 includes a motion sensor 118 and a measurement unit 106C (first processor, processor), which constitute a measurement unit that measures the imaging position (first imaging position) and imaging direction (first imaging direction) of the imaging device 100 in real space. The measurement unit 106C sets the imaging position (second imaging position) and imaging direction (second imaging direction) of the imaging device in virtual space based on the measured first imaging position and first imaging direction, and transmits information indicating the second imaging position and information indicating the second imaging direction to the virtual space management server 20. The processor 22 (mainly the virtual space management unit 22B and the virtual image generation unit 22C; the third processor) of the virtual space management server 20 can generate a virtual live view image in accordance with the second imaging position and second imaging direction.

[0076] Furthermore, when the measured first imaging position and first imaging direction change, the processor 106 (first processor) transmits information indicating the changed second imaging position and information indicating the changed second imaging direction to the virtual space management server 20. The processor 22 can generate a virtual live view image corresponding to the changed second imaging position and the changed second imaging direction (steps S190 and S210 in FIG. 7 ).

[0077] In this way, in the imaging system 1 according to the present embodiment, when the user moves in real space or changes the orientation of the imaging device, the user's avatar and the imaging device in the virtual space also move accordingly, and the orientation of the imaging device in the virtual space also changes, allowing the user to move to any location in the virtual space and capture images in any direction.

[0078] It is also possible to transmit information about the first imaging position and first imaging direction from the user system 10 to the virtual space management server 20, and calculate and set the second imaging position and second imaging direction in the virtual space management server 20. By calculating and setting the second imaging position and second imaging direction in this way in the virtual space management server 20, the load on the user system 10 can be reduced.

[0079] [Changing the Second Imaging Parameter] The second imaging parameter described above can be changed in response to a user instruction. Specifically, the imaging parameter setting unit 106B (first processor) of the imaging device 100 can set the second imaging parameter in response to an operation (setting instruction) on an imaging parameter setting member. The "imaging parameter setting member" can be a button, dial, or the like (not shown) constituting the operation unit 110. The second imaging parameter can be set and changed using the same members and operations as when setting the first imaging parameter for capturing a real image. By adopting such an embodiment, the operability of capturing a virtual image can be improved. Furthermore, if the display 108 of the imaging device 100 is a touch panel display, the display 108 can be used as the "imaging parameter setting member." The second imaging parameter can also be set and changed via the display 204 of the goggles 200 (see the description of FIG. 12 described later). Furthermore, the second imaging parameter can also be set and changed via a user's gesture or voice input.

[0080] When an operation (setting instruction) to change the second imaging parameter is performed on the imaging parameter setting member (YES in step S200), the imaging parameter setting unit 106B and the communication control unit 106E (first processor) transmit the changed (setting instruction) second imaging parameter to the virtual space management server 20 via the goggles 200 (server connection device). This transmission can be performed using the above-mentioned Web API. The virtual space management server 20 reflects the change in the second imaging parameter in the virtual live view image (described later).

[0081] [Actual Parameters and Virtual Parameters] The second imaging parameters (imaging parameters used to capture a virtual image) described above may be parameters (actual parameters) realized by a real imaging optical system provided in the imaging device 100, or parameters (virtual parameters) realized by a virtual imaging optical system virtually provided in the imaging device 100. An example of setting actual parameters is setting the focal length in the range of 25 mm to 200 mm in accordance with the rotation of the lens barrel when a zoom lens with a focal length of 25 mm to 200 mm is actually attached to the imaging device 100. Note that, unlike when capturing a real image, when capturing a virtual image, an image formed by the imaging optical system 102 of the imaging device 100 is not acquired. The rotation operation of the lens barrel in the above example is an operation for setting the second imaging parameters.

[0082] On the other hand, parameters (virtual parameters) that cannot be obtained by the optical system connected to the image capture device 100 may be set as the second image capture parameters. In the above example, if the focal length of the zoom lens actually attached to the image capture device 100 is 25 mm to 200 mm, the focal length may be changed within a range of 10 mm to 25 mm or 200 mm to 300 mm. Furthermore, even if a lens does not exist in the real world, if lens data corresponding to that lens exists, the focal length may be determined according to that lens data (use of a virtual lens). The same applies to other image capture parameters other than focal length. Setting such virtual parameters as the second image capture parameters improves the convenience of virtual image capture and provides a way to enjoy special lenses unique to virtual space.

[0083] [Setting of Second Imaging Parameters in Virtual Space] A setting instruction (setting or change) of the second imaging parameters related to capturing a virtual image may be made in virtual space. Fig. 12 is a diagram showing how the second imaging parameters are set or changed in virtual space. Part (a) of Fig. 12 shows a state in which the imaging device 100 and the goggles 200 have transitioned to a virtual image capturing mode and a virtual live view image 500 is displayed on the display 204. From this state, the display on the display 204 transitions to a setting screen for the second imaging parameters in response to a user instruction (a predetermined operation, action, voice, etc.).

[0084] Part (b) of FIG. 12 shows a state in which the virtual space management server 20 has displayed a second imaging parameter setting screen (a screen displayed in the virtual space) on the display 204. The second imaging parameter setting screen includes areas 506, 508, and 510, an OK button 512, and a cancel button 514. Area 506 displays icons representing higher-level menu items, while areas 508 and 510 display lower-level (specific) menu items. The example shown in this part shows a state in which focus mode setting and AF-S (single AF) have been selected in the autofocus (button 506A) settings. On this screen, menu selection, confirmation, setting completion, and other processes can be performed in response to user actions (gestures), voice recognition, and the like.

[0085] The setting screen for the second imaging parameters may be a screen identical to or similar to the actual setting screen of the imaging device 100 (the screen displayed on the display 108 when setting the first imaging parameters, which are the imaging conditions for actual imaging), or may be an original screen. When a screen identical to or similar to the actual setting screen is used, information on the imaging parameter setting screen of the imaging device 100 may be transmitted to the virtual space management server 20, and the virtual space management server 20 may display the setting screen in the virtual space based on that information.

[0086] The virtual space management server 20 only needs to show (transmit) the setting screen for the second imaging parameters to the user of the imaging device 100, and does not need to show it to other users. Also, although the case where the virtual space management server 20 displays the setting screen has been described, if the setting screen is not to be shown to other users, the processor 202 of the goggles 200 may generate the setting screen and display it on the display 204.

[0087] When a user is wearing goggles 200, their field of vision may be obstructed, making it difficult to operate operation unit 110 of imaging device 100 or operation unit 206 of goggles 200. However, by performing operations in a virtual space via display 204, the user can easily set the second imaging parameters while wearing goggles 200.

[0088] [Updating the Virtual Live View Image in Accordance with Changes in the Second Imaging Parameter] The processor 22 (mainly the communication control unit 22A, the virtual image generation unit 22C, and the virtual image display control unit 22D; third processor) of the virtual space management server 20 displays a virtual live view image in response to changes in the second imaging parameter on the display 204 of the goggles 200 (step S210). Fig. 13 is a diagram showing an example (virtual live view image 500A) when the focal length is changed (when zooming into area 510 in Fig. 11 ), and Fig. 14 is a diagram showing an example (virtual live view image 500B) when the exposure is changed (when the aperture value is increased). The processor 22 and the display 204 continue to display (update at a determined rate) the virtual live view image until a request to capture a virtual image is received (while step S220 is NO).

[0089] [Acquisition and Output of Virtual Image] If a second image capture instruction is received (YES in step S220), at least a request to capture a virtual image is sent from the user system 10 to the virtual space management server 20. If the latest second image capture parameters have not been sent, they may be sent together with the image capture request. The second image capture instruction may be an operation on the operation unit 110 (such as a release switch used for actual image capture) and / or the operation unit 206, or may be a predetermined user action (such as a gesture) or a voice request (for example, utterance of a specific phrase such as "take virtual image").

[0090] In response to an imaging request, the processor 22 (mainly the virtual image generation unit 22C; third processor) of the virtual space management server 20 generates a virtual image (second image) based on the set second imaging parameters and transmits the generated virtual image to the goggles 200 (server connection device, image output device). The goggles 200 receives image data of the virtual image from the virtual space management server 20 via the network NW (communication line) (acquisition of virtual image) and displays at least the virtual image corresponding to the received image data on the display 204 (output device, image output device) (step S230). The virtual space management server 20 transmits information (attachment information of the virtual image) such as the date and time of imaging of the virtual image, the location in the virtual space, the second imaging parameters, and identification information or ID of the user who participated in the imaging to the goggles 200, and the goggles 200 may output this information together with the virtual image to the display 204 or the like.

[0091] The virtual space management server 20 may generate either a still image or a moving image as the virtual image. The image generated by the virtual space management server 20 can be determined based on a user's instruction (such as an operation of an operating member, such as a release button (not shown), a user's movement, or a voice input). The image capture request or the second image capture parameter may include whether the image is a still image or a moving image. The virtual image may also include audio data. The recording control unit 106G (processor 106; first processor) of the imaging device 100 saves the acquired virtual image to the memory card 116 (output device) in response to a user's instruction (such as an operation of the operation unit 110, a movement, or a voice input). The saved virtual image can be displayed on the display 108. If the imaging device 100 is a camera with a printer, the virtual image may be printed (output) using the printer (image output device). The recording control unit 106G may also save or display the aforementioned supplementary information on a recording medium, such as the memory card 116, or output it by printing it together with the virtual image. When the supplementary information is output, it is preferable to associate it with the virtual image.

[0092] The goggles 200 end the display of the virtual image in response to a user instruction (operation on the operation unit 110 and / or operation unit 206, user movement, voice, etc.) and return to displaying the virtual live view image.

[0093] If the user gives an instruction to end virtual image capture (which may be an instruction via operation, movement, voice, etc.) (YES in step S240), the imaging device 100 and the goggles 200 end the state in which they output first-person perspective images (virtual live view images) and return to step S150 (a state in which the state of the virtual space is output), and if there is no instruction to end virtual image capture (NO in step S240; virtual image capture will continue), they return to step S200.

[0094] [Display of Virtual Images, etc.] A user can display the acquired virtual images in a virtual space. The display may be the image itself, or an object on which the image is printed. FIG. 15 is a diagram showing virtual images displayed in a virtual space. In the example shown in the figure, virtual images 802 and 804 are displayed on the walls of a virtual image exhibition room 800 set up in the virtual space. The appearance of such a virtual image exhibition room 800 is generated by the virtual space management server 20 (processor 22; third processor) in the same manner as described above for step S150 in FIG. 7, and is output to the goggles 200.

[0095] The virtual images displayed in this manner may be made available (downloaded) to desired users. When a user gives an instruction (such as operating the operation unit of the image capture device 100, performing a predetermined action, or speaking a predetermined phrase) regarding the display and acquisition of such a virtual image, the instruction is notified to the virtual space management server 20, and the virtual space management server 20 displays and / or provides (to the user system of the desired user) the virtual image in response to this notification. Regarding the display of images, in addition to virtual images, real images may be uploaded from the image capture device 100 to the virtual space management server 20 and displayed in an exhibition room within the virtual space. Furthermore, documents, diagrams, and the like may be displayed in the virtual space in addition to images.

[0096] As described above, the imaging system 1 of this embodiment not only captures virtual images but also provides users with the enjoyment of exhibiting (displaying) and providing the virtual images and the like.

[0097] As described above, according to the imaging system 1 of this embodiment, real images and virtual images can be captured using the imaging device 100, and data indicating the user's avatar and the three-dimensional shape of the imaging device can also be displayed in a virtual space.

[0098] [Applied Aspects of Virtual Image Capturing] In capturing a virtual image, similar to capturing an image in a real space, it is conceivable that a user would like to capture a virtual image that includes the user (the user's avatar) in a virtual space (such as a so-called "selfie" or "group photo"). An embodiment that takes such circumstances into consideration will be described below.

[0099] [Fixing the Imaging Position and Imaging Direction in Virtual Space] Figure 16 is a diagram showing how the imaging position and imaging direction are fixed in virtual space. Part (a) of the figure shows a state in which the user 2 holds the imaging device 100 near his / her head (or near the goggles 200) and assumes a posture as if looking into the viewfinder. This satisfies the above-mentioned "predetermined condition," causing the imaging device 100 and the goggles 200 to transition to virtual imaging mode. When the user issues a predetermined instruction in this state, the processor 106 (processor, first processor) causes the virtual space management server 20 to fix the imaging position (second imaging position) and imaging direction (second imaging direction) in the virtual space in accordance with the user's instruction to the imaging device 100. Part (b) of Figure 16 shows a state in which the imaging position and imaging direction are fixed. In this state, the imaging position and imaging direction of the virtual live view image output to the display 204 of the goggles 200 and the captured virtual image are fixed. This allows the user to lower the imaging device 100 (without maintaining the same posture).

[0100] The "instruction" to fix the imaging position and imaging direction may be a predetermined operation on the components of the operation unit 110 and / or the operation unit 206, or on the display 108, a specific movement by the user (such as shaking a hand or a leg), or a voice instruction (utterance of specific phrases such as "fix position" or "fix direction"). The user's movement can be detected by the motion sensor 118 and the measurement unit 106C, and the voice instruction can be input via the microphone 214. Two or more of the operation, movement, and voice may be combined to form an "instruction." Note that these instructions can also be used to release the fixation.

[0101] By fixing the imaging position and imaging direction in the virtual space, the user can move to any location and capture a virtual image, and can also capture an image of themselves. Note that when the imaging position and imaging direction are fixed, the user system 10 (the imaging device 100 and the goggles 200) may automatically perform self-timer imaging, as described below.

[0102] [Self-Timer Capture of Virtual Image] The processor 106 (mainly the image acquisition unit 106A and the communication control unit 106E; the first processor) sends a request to capture a second image to the virtual space management server 20 after a predetermined time has elapsed since the second image capture instruction was issued. This "self-timer capture" allows the user to easily capture the self-portrait or a group photo (virtual image) including the user. The "predetermined time (self-timer time)" can be set by operating the operation unit 110 and / or the operation unit 206, by user action, by voice input, or a combination of these. If the image capture device 100 has an operating member such as a switch or lever for self-timer capture in real space, the operating member may also be used in capturing a virtual image.

[0103] Figure 17 is a diagram showing how imaging is performed using the self-timer function. Part (a) of Figure 17 shows a state in which a virtual live view image 500 is displayed on the display 204 (the same as in Figure 11), and in this state, the imaging position and imaging direction are fixed. In this state, the user can move away from the position of the imaging device in the virtual space and enter the field of view of the virtual live view image 500D, as shown in part (b) of Figure 17. Note that in this part, avatar 600 is the avatar of the user of the imaging device 100, and avatar 700 is the avatar of another user.

[0104] FIG. 18 is another diagram illustrating image capture using the self-timer function. Part (a) of FIG. 18 illustrates a state in which the image capture position and image capture direction are fixed in virtual image capture mode, and the user himself / herself is within the field of view (one frame of a virtual live view image). Part (b) of the same figure illustrates a state in which the user has performed an action to switch to self-timer mode (raising his / her right hand and waving it from side to side; one form of image capture instruction). The user's actual action is reflected in the action of the avatar 600. Part (c) of the same figure illustrates a state in which the self-timer mode has been switched to, and part (d) illustrates a state in which the user has performed an action to set the self-timer (raising his / her left hand and waving it from side to side; one form of image capture instruction). In part (d), similar to part (b), the user's actual action is reflected in the action of the avatar 600. Part (e) illustrates a state in which a virtual image is captured after the timer time (a predetermined time) has elapsed since the self-timer was set.

[0105] In the imaging system 1 of this embodiment, such a self-timer function allows the user to enjoy taking self-portraits and group photos with other users. Note that instructions to switch to the self-timer mode or to capture an image may be given not only by the user of the imaging device 100, but also by instructions from other users (such as the movement or voice of an avatar that reflects the movement of the other user).

[0106] [Modification of the User System] In the first embodiment described above, the user system 10 is configured using the goggles 200 as a server connection device, but the configuration of the user system is not limited to this. FIG. 19 is a diagram showing a modification of the user system. In the user system 10A shown in FIG. 19, a smartphone 400 is used as a "server connection device," and the display 410 of the smartphone 400 is used to display virtual images, etc. The smartphone 400 and the imaging device 100 can be connected via wireless communication such as Bluetooth, and the smartphone 400 connects to the router 300 via wireless communication such as Wi-Fi to communicate with the virtual space management server 20. Note that instead of the smartphone 400, a network-connectable device such as a personal computer or tablet terminal may be used as the server connection device.

[0107] In these embodiments, the sense of immersion in the virtual space is reduced, but the user's field of vision is not obstructed, and the user can easily take virtual images, etc. When a device with an in-camera is used as a server connection device, the user's movements may be detected from images acquired by the in-camera of the device, or information such as "facial expressions" and "emotions" may be assigned to the avatar based on information extracted from the images.

[0108] [Second Embodiment] In the first embodiment and the modified example described above, the user system is configured with an imaging device and a server connection device, but the user system may also be configured with an imaging device alone (the imaging device 100A may serve as both the "imaging device" and the "server connection device"). Fig. 20 is a diagram showing the configuration of a user system according to the second embodiment. In the second embodiment, the imaging device 100A alone configures a user system 10B. In the user system 10B, the imaging device 100A connects to the virtual space management server 20 via wireless communication such as Wi-Fi, and sets second imaging parameters, captures virtual images, etc.

[0109] 21 is a diagram showing the configuration of an image capture device 100A according to the second embodiment. As shown in the figure, the image capture device 100A includes a microphone 122 and a speaker 124 in addition to the configuration of the image capture device 100 according to the first embodiment, and these devices can input and output audio (the processor 107 is assumed to have a voice recognition function). The display 108 can also be used for setting second image capture parameters, displaying virtual images, and the like. In this configuration, the sense of immersion in the virtual space is reduced, as in the above-described modification, but the user's field of vision is not obstructed, and the user can easily capture virtual images, etc.

[0110] Although the embodiments and modifications of the present invention have been described above, the present invention is not limited to the above-described aspects and various modifications are possible.

[0111] 1 Imaging system 2 User 10 User system 10A User system 10B User system 20 Virtual space management server 22 Processor 22A Communication control unit 22B Virtual space management unit 22C Virtual image generation unit 22D Virtual image display control unit 28 Database 100 Imaging device 100A Imaging device 102 Imaging optical system 104 Imaging element 106 Processor 106A Image acquisition unit 106B Imaging parameter setting unit 106C Measurement unit 106D Mode control unit 106E Communication control unit 106F Display control unit 106G Recording control unit 107 Processor 108 Display 110 Operation unit 116 Memory card 118 Motion sensor 120 Wireless communication interface 122 Microphone 124 Speaker 200 Goggles 202 Processor 204 Display 206 Operation unit 212 Wireless communication interface 214 Microphone 216 Speaker 218 Microphone 220 Speaker 300 Router 400 Smartphone 410 Display 500 Virtual live view image 500A Virtual live view image 500B Virtual live view image 500D Virtual live view image 502 Area 504 Area 506 Area 506A Button 508 Area 510 Area 512 OK button 514 Cancel button 600 Avatar 610 Three-dimensional shape data 700 Avatar 800 Virtual image exhibition room 802 Virtual image 804 Virtual image S100 to S240 Each step of imaging method

Claims

1. An imaging system having an imaging device and a server connection device, the imaging device includes a first processor; The first processor setting first imaging parameters which are imaging conditions when an object in real space is imaged using an imaging optical system and an imaging element to obtain a first image, and second imaging parameters which are imaging conditions when an object in virtual space is virtually imaged to obtain a second image, acquiring the first image based on the first imaging parameters in response to a first imaging instruction; when a predetermined condition is satisfied, the imaging device and the server connection device transition to a virtual imaging mode in which an imaging request for the second image and settings of the second imaging parameters are accepted; In the virtual imaging mode, the first processor When an instruction to set the second imaging parameters is given, the instruction to set the second imaging parameters is transmitted to a virtual space management server that manages the virtual space via the server connection device; In response to a second image capture instruction, transmitting an image capture request for at least the second image to the virtual space management server via the server connection device; the server connection device includes a second processor and an image output device; In the virtual imaging mode, the second processor transmitting the second image capturing request and the second image capturing parameters to the virtual space management server via a communication line; receiving image data of the second image corresponding to the transmitted second imaging parameters from the virtual space management server via the communication line; causing the image output device to output the second image corresponding to at least the received image data; The server connection device is a goggle-type device worn by a user, and the imaging system includes a display device that displays the second image as the image output device.

2. The first processor setting the second imaging parameter in response to an operation on an imaging parameter setting member; The imaging system according to claim 1 , wherein when an operation to change the second imaging parameter is performed on the imaging parameter setting member, the changed second imaging parameter is transmitted to the virtual space management server.

3. An imaging system as described in claim 1 or 2, wherein the first processor sets the second imaging parameters to actual parameters realized by an actual imaging optical system provided in the imaging device, or virtual parameters realized by a virtual imaging optical system virtually provided in the imaging device.

4. A measurement unit that measures a first imaging position and a first imaging direction of the imaging device in the real space, The first processor setting a second imaging position and a second imaging direction of the imaging device in the virtual space based on the measured first imaging position and first imaging direction; transmitting information indicating the second imaging position and information indicating the second imaging direction to the virtual space management server; The imaging system according to claim 1 or 2, wherein image data of the second image corresponding to the second imaging position and the second imaging direction is received from the virtual space management server.

5. An imaging system as described in Claim 4, wherein the first processor, when the measured first imaging position and first imaging direction change, transmits information indicating the changed second imaging position and information indicating the changed second imaging direction to the virtual space management server.

6. An imaging system as described in Claim 4, wherein the first processor fixes the second imaging position and the second imaging direction in the virtual space using the virtual space management server in accordance with user instructions to the imaging device.

7. An imaging system as described in claim 1 or 2, wherein the first processor sends the imaging request for the second image to the virtual space management server after a predetermined time has elapsed since the second imaging instruction was issued.

8. A first storage unit that stores avatar information indicating an avatar of a user of the imaging device; The imaging system according to claim 1 or 2, wherein the first processor transmits the stored avatar information to the virtual space management server, and causes the virtual space management server to display the avatar of the user in the virtual space.

9. A second storage unit that stores data indicating a three-dimensional shape of the imaging device, The imaging system described in claim 8, wherein the first processor transmits the stored data to the virtual space management server, and the virtual space management server displays the three-dimensional shape of the imaging device in correspondence with the avatar in the vicinity of the avatar in the virtual space.

10. An imaging system as described in claim 1 or 2, wherein the second image is a still image or a moving image.

11. Further comprising the virtual space management server, The virtual space management server generating image data of the second image corresponding to the second imaging parameters in accordance with the imaging request and the imaging conditions of the second image received from the server connection device; transmitting the generated image data to the server connection device; 3. The imaging system according to claim 1.

12. An imaging system as described in Claim 11, wherein the virtual space management server causes the virtual space management server to display the second image in the virtual space in response to operation by a user of the imaging device.

13. The imaging system described in Claim 11, wherein the virtual space management server transmits image data of the second image generated in response to the imaging request received from one of the imaging devices connected to the virtual space management server to another imaging device in response to a request from the other imaging device connected to the virtual space management server.

14. An imaging method executed by an imaging system having an imaging device including a first processor and a server connection device including a second processor and an image output device, The first processor setting first imaging parameters which are imaging conditions when an object in real space is imaged using an imaging optical system and an imaging element to obtain a first image, and second imaging parameters which are imaging conditions when an object in virtual space is virtually imaged to obtain a second image, acquiring the first image based on the first imaging parameters in response to a first imaging instruction; when a predetermined condition is satisfied, the imaging device and the server connection device transition to a virtual imaging mode in which an imaging request for the second image and settings of the second imaging parameters are accepted; In the virtual imaging mode, the first processor When an instruction to set the second imaging parameters is given, the instruction to set the second imaging parameters is transmitted to a virtual space management server that manages the virtual space via the server connection device; transmitting the imaging request for the second image to the virtual space management server via the server connection device in response to the second imaging instruction; In the virtual imaging mode, the second processor transmitting a request to capture the second image to the virtual space management server via a communication line; receiving image data of the second image corresponding to the second imaging parameters from the virtual space management server via the communication line; causing the image output device to output the second image corresponding to at least the received image data; In the imaging system, the server connection device is a goggle-type device worn by a user, and the image output device includes a display device that displays the second image. Imaging method.