Image pickup apparatus, image pickup system, and image pickup method

Abstract

An embodiment of the present invention provides an imaging device, an imaging system, and an imaging method that enable imaging in an actual space and imaging in a virtual space. In an imaging device according to one embodiment of the present invention, a processor sets a first imaging parameter, which is an imaging parameter when acquiring a first image by imaging an object in an actual space, and a second imaging parameter, which is an imaging parameter when acquiring a second image by imaging the object in the actual space. The second imaging parameter is an imaging parameter when a second image is acquired by virtually imaging an object in a virtual space; acquiring a first image based on the first camera shooting parameter according to the first camera shooting instruction operation; transmitting, via a communication line, an imaging request for a virtual image and a second imaging parameter to a virtual space management server that manages a virtual space in the virtual imaging mode; receiving image data of a second image corresponding to the second imaging parameter from the virtual space management server via the communication line; and causing at least an output device to output the second image.

Description

Technical Field

[0001] The present invention relates to a camera device, a camera system and a camera method, and in particular to a camera device, a camera system and a camera method capable of photographing objects in real space and objects in virtual space. Background Art

[0002] Regarding the technology of capturing an object in a virtual space, for example, Patent Documents 1 to 3 are known.

[0003] Previous technical literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2020-129272

[0006] Patent Document 2: Japanese Patent Application Laid-Open No. 2020-140383

[0007] Patent Document 3: Japanese Patent Application Laid-Open No. 2021-101358 Summary of the Invention

[0008] One embodiment of the technology according to the present invention provides an imaging device, an imaging system, and an imaging method capable of imaging in both a real space and a virtual space.

[0009] Means for solving technical problems

[0010] The first embodiment of the present invention includes a camera device comprising a processor, which performs the following processing: setting a first camera parameter and a second camera parameter, wherein the first camera parameter is a camera parameter when a first image is acquired by photographing an object in a real space using a camera optical system and an imaging element, and the second camera parameter is a camera parameter when a second image is acquired by virtually photographing an object in a virtual space; and acquiring the first image based on the first camera parameter according to a first camera instruction. When a predetermined condition is satisfied, the camera device transitions to a virtual camera mode for accepting a camera request for the second image and the setting of the second camera parameter. In the virtual camera mode, the processor performs the following processing: when there is a setting instruction for the second camera parameter, sending the second camera parameter of the setting instruction to a virtual space management server that manages the virtual space via a communication line; sending at least a camera request for the second image via the communication line according to the second camera instruction; receiving image data of the second image corresponding to the sent second camera parameter from the virtual space management server via the communication line; and causing the image output device to output at least the second image corresponding to the image data. In the first embodiment, the devices and the like for realizing the functions of the imaging device may be housed in one housing, or may be housed in a plurality of separate or separable housings.

[0011] In the camera device involved in the second embodiment of the present invention, in the first embodiment, the processor performs the following processing: setting the second camera parameter according to the operation of the camera parameter setting component; and when the operation of changing the second camera parameter is performed on the camera parameter setting component, the changed second camera parameter is sent to the virtual space management server.

[0012] In the imaging device according to the third aspect, in the first or second aspect, the processor sets actual parameters realized by an actual imaging optical system included in the imaging device or virtual parameters realized by a virtual imaging optical system virtually included in the imaging device as the second imaging parameters.

[0013] The camera device involved in the fourth embodiment includes a measuring unit in any one of the first to third embodiments, which measures the first camera position and the first camera direction of the camera device in the actual space, and the processor performs the following processing: based on the measured first camera position and the first camera direction, setting the second camera position and the second camera direction of the camera device in the virtual space; sending information representing the second camera position and the information representing the second camera direction to the virtual space management server; and receiving image data of the second image corresponding to the second camera position and the second camera direction from the virtual space management server.

[0014] The camera device according to the fifth aspect is configured such that, in the fourth aspect, when the measured first camera position and first camera direction change, the processor transmits information indicating the changed second camera position and information indicating the changed second camera direction to the virtual space management server.

[0015] The camera device according to a sixth aspect is characterized in that, in the fourth or fifth aspect, the processor fixes the second camera position and the second camera direction in the virtual space in accordance with a user instruction to the camera device via the virtual space management server.

[0016] In the imaging device according to a seventh aspect, in any one of the first to sixth aspects, the processor transmits a request to capture the second image to the virtual space management server after a predetermined time has elapsed since the second imaging instruction was issued.

[0017] The camera device involved in the 8th mode is any one of the 1st to 7th modes, and the camera device has a first storage unit, which stores avatar information representing the avatar of the user of the camera device, and the processor sends the stored avatar information to the virtual space management server, and displays the user's avatar in the virtual space through the virtual space management server.

[0018] The camera device involved in the 9th method has a second storage unit in the 8th method, and the second storage unit stores data representing the three-dimensional shape of the camera device. The processor sends the stored data to the virtual space management server, and displays the three-dimensional shape of the camera device in the vicinity of the avatar in the virtual space through the virtual space management server so as to correspond to the avatar.

[0019] In the imaging device according to a tenth aspect, in any one of the first to ninth aspects, the second image is a still image or a moving image.

[0020] The camera system involved in the 11th method includes a camera device and a server connection device, wherein the camera device has a first processor, and the first processor performs the following processing: setting a first camera parameter and a second camera parameter, the first camera parameter being a camera condition when a first image is acquired by using a camera optical system and an imaging element to capture an object in a real space, and the second camera parameter being a camera condition when a second image is acquired by virtually capturing an object in a virtual space; and acquiring the first image based on the first camera parameter according to the first camera instruction; the camera device and the server connection device shift to a virtual camera mode for accepting a camera request for the second image and the setting of the second camera parameter when predetermined conditions are met, and the first processor performs the following in the virtual camera mode Processing: In the presence of a setting instruction for a second camera parameter, the second camera parameter of the setting instruction is sent to the virtual space management server that manages the virtual space via the server connection device; and based on the second camera instruction, at least a camera request for the second image is sent to the virtual space management server via the server connection device. The server connection device has a second processor and an image output device, and the second processor performs the following processing in the virtual camera mode: sending the camera request for the second image and the second camera parameter to the virtual space management server via the communication line; receiving image data of the second image corresponding to the sent second camera parameter from the virtual space management server via the communication line; and causing the image output device to output at least the second image corresponding to the received image data.

[0021] In the imaging system according to the 12th aspect, in the 11th aspect, the server connection device is a goggle-type device worn by the user, and includes a display device for displaying the second image as the image output device.

[0022] The camera system involved in the 13th method further includes a virtual space management server in the 11th or 12th method, and the virtual space management server performs the following processing: based on the camera request and camera conditions of the second image received from the server connection device, generates image data of the second image corresponding to the second camera parameters, and sends the generated image data to the server connection device.

[0023] In the imaging system according to the fourteenth aspect, in the thirteenth aspect, the virtual space management server causes the virtual space management server to display the second image in the virtual space in response to an operation performed by a user of the imaging device.

[0024] In the camera system involved in the 15th method, in the 13th or 14th method, the virtual space management server sends image data of the second image generated based on a camera request received from a camera device connected to the virtual space management server to another camera device based on a request from another camera device connected to the virtual space management server.

[0025] The imaging method according to the sixteenth embodiment is executed by an imaging device having a processor, and the processor performs the following processing: setting a first imaging parameter and a second imaging parameter, wherein the first imaging parameter is an imaging parameter when an object in a real space is captured using an imaging optical system and an imaging element to obtain a first image, and the second imaging parameter is an imaging parameter when an object in a virtual space is virtually captured to obtain a second image; and acquiring the first image based on the first imaging parameter according to the first imaging instruction; the imaging device transfers to accepting an imaging request for the second image and when a predetermined condition is satisfied. In a virtual camera mode in which a second camera parameter is set, the processor performs the following processing in the virtual camera mode: if there is an instruction to set the second camera parameter, the processor transmits the second camera parameter indicated by the setting instruction to a virtual space management server that manages the virtual space via a communication line; transmits a capture request for a second image to the virtual space management server via the communication line based on the second camera instruction; receives image data of the second image corresponding to the transmitted second camera parameter from the virtual space management server via the communication line; and causes the image output device to output at least the second image corresponding to the image data. The camera method according to the sixteenth aspect may also have the same structure as the second to tenth aspects.

[0026] The camera method involved in the 17th method is executed by a camera system having a camera device with a first processor and a server connection device with a second processor and an image output device. In the camera method, the first processor performs the following processing: setting a first camera parameter and a second camera parameter, the first camera parameter being a camera condition when a first image is acquired by using a camera optical system and an imaging element to capture an object in a real space, and the second camera parameter being a camera condition when a second image is acquired by virtually capturing an object in a virtual space; and acquiring the first image based on the first camera parameter according to the first camera instruction, and the camera device and the server connection device transferring to accepting a camera request for the second image and the second camera parameter when predetermined conditions are met. In a virtual camera mode with a number of settings, the first processor performs the following processing in the virtual camera mode: in the presence of a setting instruction for a second camera parameter, the second camera parameter of the setting instruction is sent to the virtual space management server that manages the virtual space via the server connection device; and according to the second camera instruction, the camera request for the second image is sent to the virtual space management server via the server connection device, and the second processor performs the following processing in the virtual camera mode: the camera request for the second image is sent to the virtual space management server via the communication line; the image data of the second image corresponding to the second camera parameter is received from the virtual space management server via the communication line; and the image output device is caused to output at least the second image corresponding to the received image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a diagram showing the configuration of an imaging system according to the first embodiment.

[0028] Figure 2 This is a diagram showing the configuration of a user system according to the first embodiment.

[0029] Figure 3 It is a diagram showing the structure of an imaging device.

[0030] Figure 4 This is a diagram showing the functional structure of a processor of an imaging device.

[0031] Figure 5 It is a figure which shows the structure of goggles.

[0032] Figure 6 This is a diagram showing the structure of a virtual space management server.

[0033] Figure 7 This is a flowchart showing the processing of the imaging method according to the first embodiment.

[0034] Figure 8 is a flowchart showing the processing of the imaging method according to the first embodiment ( Figure 7continuation of ).

[0035] Figure 9 This is a diagram showing a state where the three-dimensional shapes of an avatar and a camera device are displayed in a virtual space.

[0036] Figure 10 This is a diagram showing an example of mode transition due to user actions.

[0037] Figure 11 FIG. 1 is a diagram showing a state where a virtual live view image is displayed in the goggles.

[0038] Figure 12 This is a diagram showing the setting or change status of the second imaging parameter in the virtual space.

[0039] Figure 13 FIG. 1 is a diagram showing an example of a virtual live view image when the focal length of the imaging device is changed.

[0040] Figure 14 FIG. 1 is a diagram showing an example of a virtual live view image when the exposure is changed.

[0041] Figure 15 This diagram shows how a virtual image is displayed in a virtual space.

[0042] Figure 16 This is a diagram showing a state where the camera position and camera direction in the virtual space are fixed.

[0043] Figure 17 This diagram shows a situation in which a picture is taken using the Selfie function.

[0044] Figure 18 This is another diagram showing a state of shooting using the Selfie function.

[0045] Figure 19 It is a diagram showing a modified example of the user system.

[0046] Figure 20 This is a diagram showing the configuration of a user system according to the second embodiment.

[0047] Figure 21 It is a diagram showing the configuration of an imaging device according to a second embodiment. DETAILED DESCRIPTION

[0048] An embodiment of an imaging device, an imaging system, and an imaging method according to the present invention is as follows.

[0049] [First embodiment]

[0050] [Structure of the camera system]

[0051] Figure 1 1 is a diagram showing the configuration of an imaging system according to the first embodiment. Figure 1 As shown, the camera system 1 (camera system) includes a user system 10 (camera system) and a virtual space management server 20 (virtual space management server), which are connected via a network NW (communication line) such as the Internet. The user system 10 may be one or more.

[0052] [Structure of user system]

[0053] Figure 2 1 is a diagram showing the structure of a user system according to the first embodiment. Figure 2 As shown, the user system 10 includes an imaging device 100 (imaging device), goggles 200 (server connection device, goggles-type device), and a router 300. The goggles 200 are connected to the network NW via the router 300. The imaging device 100 and the goggles 200 can be connected via wired communication or wireless communication such as Bluetooth (registered trademark), and the goggles 200 and the router 300 can be connected via wired communication or wireless communication such as Wi-Fi (registered trademark).

[0054] [Structure of the imaging device]

[0055] Figure 3 1 is a block diagram showing the structure of the imaging device 100. Figure 3 As shown, the camera device 100 includes an imaging optical system 102 (imaging optical system, actual imaging optical system), an imaging element 104 (imaging element), a processor 106 (processor), a display 108 (display device), an operating unit 110 (a component for setting imaging parameters), a flash ROM (ROM is Read Only Memory: read-only memory; a non-temporary and tangible recording medium), a RAM 114 (RAM is Random Access Memory: random access memory), a storage card 116, a motion sensor 118 (measuring unit), and a wireless communication interface 120.

[0056] The imaging optical system 102 includes optical components such as an aperture in addition to lenses (imaging lens, zoom lens, focusing lens, etc.) not shown in the figure, and forms an optical image of a subject (object) existing in real space on the imaging element 104. As described in detail later, the processor 106 processes the optical image to generate an actual image of the subject (a first image; a still image or a moving image). As the imaging element 104, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal-Oxide Semiconductor) type imaging element can be used. The generated actual image can be displayed on the display 108. The display 108 is composed of a device such as a liquid crystal monitor.

[0057] The operating unit 110 is composed of operating components such as a release button, a cross-shaped button, a menu / enter button, a dial, a switch, and a self-timer setting lever. These components can be used to set both the first imaging parameters (i.e., when capturing an actual image) and the second imaging parameters (i.e., when capturing a virtual image). Setting imaging parameters will be described in detail later. Furthermore, the operating unit 110 can also be used for operations such as imaging instructions (image capture requests) and Selfie settings.

[0058] The configuration and function of the processor 106 , the flash ROM 112 , and the RAM 114 will be described later.

[0059] The memory card 116 is a non-temporary, tangible recording medium that can be attached or removed from the imaging device 100 and inserted. It can record information such as the actual image (first image) or virtual image (second image) captured. The memory card 116 can be configured as a storage element using, for example, flash memory. Furthermore, avatar information representing an avatar of a user of the imaging device and data representing the three-dimensional shape of the imaging device may be stored in the flash ROM 112 (first storage unit, second storage unit) and / or the memory card 116 (first storage unit, second storage unit).

[0060] The motion sensor 118 includes sensors such as an acceleration sensor and an angular velocity sensor, and can output information indicating the imaging position (first imaging position) and imaging direction (first imaging direction) of the imaging device 100 in real space. In addition to these sensors, the imaging device 100 may also be equipped with a GPS receiver (GPS: Global Positioning System).

[0061] The wireless communication interface 120 includes an antenna for wireless communication such as Bluetooth (registered trademark) and is used for communication with the goggles 200 .

[0062] [Camera processor]

[0063] Figure 4 1 is a diagram showing the functional structure of the processor 106 (processor, first processor) of the imaging device 100. Figure 4 As shown, the processor 106 functions as an image acquisition unit 106A, an imaging parameter setting unit 106B, a measurement unit 106C, a mode control unit 106D, a communication control unit 106E, a display control unit 106F, and a recording control unit 106G. If the imaging device 100 includes a GPS receiver, the processor 106 may function as a positioning unit that performs positioning based on GPS signals.

[0064] The processor 106 is composed of various processors or circuits, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), and a PLD (Programmable Logic Device). When these processors or circuits execute software (a program), code readable by a computer executing the software (e.g., the various processors or circuits constituting the processor and / or a combination thereof) is stored in a non-transitory, tangible recording medium such as a flash ROM 112, and the computer references the software. The software stored in the non-transitory, tangible recording medium includes the imaging program involved in the present invention (a program that executes the imaging method involved in the present invention) and data used when executing the program. The code may also be recorded in a non-transitory, tangible recording medium such as a flash ROM or an EEPROM (Electronically Erasable and Programmable Read Only Memory) instead of in the flash ROM 112. The "non-transitory and tangible recording medium" does not include non-tangible recording media such as carrier signals or propagation signals themselves (the same applies to processors 202 and 22 described later). When processing using software, RAM 114 is used as a temporary storage area or work area.

[0065] The processing performed by the processor 106 using the above-described configuration will be described in detail later.

[0066] [Structure of goggles]

[0067] Figure 5 2 is a diagram showing the structure of the goggles 200. Figure 5As shown, the goggles 200 include a processor 202, a display 204, an operating unit 206, a flash ROM 208, a RAM 210, a wireless communication interface 212, a microphone 214, and a speaker 216. These communicate with the imaging device 100 and with the virtual space management server 20 via the router 300 and the network NW. Specifically, the goggles 200 function as a server connection device and an image display device (image output device) for virtual images, and can be configured as a VR goggle-type device (VR: Virtual Reality) worn on the user's head. The goggles 200 may also include a memory card and / or a motion sensor.

[0068] Similar to the above description of the processor 106 of the imaging device 100, the processor 202 (processor, second processor) of the goggles 200 can be configured using a CPU, GPU, or the like, and executes various programs, such as the imaging program of the present invention (a program that executes the imaging method of the present invention). During execution, the program or data stored in the flash ROM 208 can be used, and during processing, the RAM 210 can be used as a temporary storage area or work area.

[0069] The display 204 is used to display various information, such as a virtual image (second image), imaging parameters, an operation menu, and messages to the user. Furthermore, operations such as setting imaging parameters (second imaging parameters), switching imaging modes (shifting), requesting imaging, and displaying images can be performed using both the operation unit 206 of the virtual image goggles 200 and the operation unit 110 of the imaging device 100. Even in situations where the goggles 200 covering the area around the eyes makes it difficult to operate the operation unit 110 of the imaging device 100, simultaneous use of the operation unit 110 and the operation unit 206 improves operability.

[0070] Furthermore, the goggles 200 can operate via voice input via the microphone 218 and speaker 220, listen to voices in the virtual space, and communicate with other users connected (participating) in the virtual space. Messages can also be output to the user via these devices. Furthermore, the processor 202 can perform voice recognition on voice input via the microphone 218.

[0071] The processing using the goggles 200 having the above-described structure will be described in detail later.

[0072] [Structure of the virtual space management server]

[0073] Figure 6 20 (virtual space management server). Figure 6As shown, the virtual space management server 20 includes a processor 22 (processor, third processor), ROM 24, RAM 26, and a database 28, and manages the positions of subjects or avatars in the virtual space, voices, lighting (brightness, saturation, etc.) in the virtual space, and changes thereto.

[0074] Regarding the processor 22, similar to the above-mentioned contents regarding the processor 106 of the camera device 100 and the processor 202 of the goggles 200, it can be constructed using a CPU, a GPU, etc., and execute various programs such as the camera program involved in the present invention (a program for executing the camera method involved in the present invention). When executing these programs, the programs or data stored in the ROM 24 (a non-temporary and tangible recording medium) can be used, and when processing, the RAM 26 can be used as a temporary storage area or a working area. In addition, the database 28 has various optical magnetic recording devices, semiconductor recording devices, etc. (non-temporary and tangible recording media), which can record and output various information required when managing the virtual space (virtual space environment setting information, subject configuration, user account, etc.).

[0075] [Camera in the camera system]

[0076] In the imaging system 1 having the above-described configuration, as will be described below, it is possible to capture a real image (first image) and a virtual image (second image).

[0077] [Acquisition of actual images]

[0078] In the camera device 100, as with a general camera device, an actual image can be acquired. For example, the processor 106 (processor, first processor) of the camera device 100 sets the camera conditions, i.e., the first camera parameters, when capturing an object in the actual space and acquiring a real image (first image) according to the user's operation. The first camera parameters may include at least one of focal length, focus distance, shutter speed, aperture value, and image quality. In addition, they may also include camera modes (photographic scenes), the presence or absence of image processing and its content, the presence or absence of flash light, etc. Furthermore, "image quality" may include at least one of resolution, clarity, ISO sensitivity (ISO: International Organization for Standardization), and white balance, and may also include film simulation, color tone, saturation, contrast, hue, noise, dynamic range, brightness, and other items that can be set by capturing an actual image (the same applies to virtual images described later).

[0079] The user can set the first imaging parameter using a dial, button, or the like (not shown) on the operating unit 110. The setting operation may be a zoom operation and / or a focus operation on the imaging optical system 102 (e.g., rotation of the zoom ring or focus ring on the lens barrel, forward / backward movement, etc.), and if the display 108 is a touch panel type, the operation may also be performed via the touch panel.

[0080] The camera device 100 can obtain an actual image (first image) by using the camera optical system 102 and the imaging element 104 to shoot an object in the actual space based on the first camera parameter in response to an operation (first camera instruction, camera request) such as pressing a release button (not shown) by the user.

[0081] [Photography of virtual images]

[0082] [Basic method of virtual image capture]

[0083] Figure 7 and Figure 8 This is a flowchart showing the steps of a basic method of capturing a virtual image.

[0084] [Pairing of camera and goggles]

[0085] The mode control unit 106D (processor 106) sets the camera 100 to the virtual space connection mode, and the processor 202 sets the goggles 200 to the virtual space connection mode (step S100). The triggers (opportunities) for setting may include, for example, detecting an operation on the operating unit 110, detecting a predetermined action (such as shaking the camera 100) by the motion sensor 118, detecting an operation on the operating unit 206, or recognizing a voice input to the microphone 214 (in this case, the processor 202 of the goggles 200 is deemed to have a voice recognition function).

[0086] If the camera device 100 and the goggles 200 are set to the virtual space connection mode, the mode control unit 106D and the processor 202 pair the camera device 100 with the goggles 200 (step S110). The communication between the camera device 100 and the goggles 200 can be carried out through wireless communication such as Bluetooth (registered trademark) via the wireless communication interface 120 and the wireless communication interface 212, and the pairing process is the same as the pairing of a PC (Personal Computer) or a smartphone with peripheral devices (printers, speakers, headphones, etc.). If the pairing is successful ("Yes" in step S120), proceed to step S130. Thereafter, the camera device 100 acts as a controller of the goggles 200. In addition, in Figure 7In the example, pairing is repeated until it succeeds (until "YES" is obtained in step S120), but if it fails within a predetermined number of times or within a predetermined time, the process can be terminated.

[0087] [Sending avatar information]

[0088] If pairing is successful in step S120, the user system 10 transmits avatar information representing the avatar of the user of the camera device 100 to the virtual space management server 20 (step S130). Specifically, the control unit 106E (processor 106) and the processor 202 transmit the avatar information to the virtual space management server 20 via the goggles 200 (server connection device) and the router 300 (step S130). As described above, avatar information can be stored in the flash ROM 112 (first storage unit) and / or the memory card 116 (first storage unit). Furthermore, the user can create avatar information or download (purchase) avatar information. Furthermore, the avatar can be a human-like avatar, or it can be an animal, plant, or other object. The avatar can have the user's "emotions" or "expressions," or attributes equivalent thereto.

[0089] In the camera system 1, by detecting the user's movements using the motion sensors 118 of the camera device 100, for example, the avatar can reflect these movements and change its position or posture within the virtual space. Furthermore, to accurately reflect the user's movements on the avatar, the user system 10 may also include motion sensors (accelerometers, angular velocity sensors, etc.) attached to the user's hands and feet. In the user system 10, these motion sensors can communicate with the virtual space management server 20 via the goggles 200 (server connection device).

[0090] [Transmission of 3D Shape Data from the Imaging Device]

[0091] [Meaning of 3D shape data]

[0092] In recent years, photographing people and landscapes within virtual spaces (virtual reality spaces, VR spaces) has become increasingly established as an application scenario, leading to an increasing demand for three-dimensional shape data for digital cameras (cameras) that can be used as avatars in virtual spaces. In this context, polygonal models of digital cameras that exist in real space have begun to be provided as items in virtual spaces. However, these models are low-polygon models used as props to decorate avatars in virtual spaces. This trend suggests that avatars in virtual spaces will transition from being assets of specific platform providers to personal assets. Items used to dress up one's avatar will become personal assets, and not only cameras, but also various everyday objects will be circulated rapidly in the form of data in virtual spaces. In view of this situation, in this embodiment, the three-dimensional shape data of the camera device 100 is enabled to be displayed in virtual space.

[0093] The communication control unit 106E (processor 106; first processor) transmits data representing the three-dimensional shape of the camera device 100 to the virtual space management server 20 via the goggles 200 (server connection device), the router 300, and the network NW (communication line) (step S140). The three-dimensional shape data can be stored in the flash ROM 112 (second storage unit) and / or the memory card 116 (second storage unit). The communication control unit 106E can transmit the three-dimensional shape data using, for example, a Web API (API: Application Programming Interface). In addition, the Web API preferably communicates, in addition to the three-dimensional shape data, status information of the camera device 100 (power on / off status, tilt information (posture around three axes), focus and zoom information, release status, etc.) and the second camera parameter.

[0094] The three-dimensional shape data can be provided by the company that manufactures or sells the camera device 100. As described later in the "Composition of Virtual Space" section, this data allows users to access the data within the virtual space. Furthermore, the three-dimensional shape data can accurately represent the shape, size, color, texture, and other aspects of the camera device 100, or it can be data that differs from the actual camera device 100, such as data that simulates or deforms the camera device 100. For example, if the actual camera device 100 is a popular or low-priced model, displaying data for an actual high-end device or data that visualizes a high-end device can satisfy the user's desire to own a high-end camera in the virtual space. Data for a model that does not actually exist (a model that exists only in the virtual space) can also be displayed. Furthermore, the brand name and / or logo of the camera device 100 can be displayed within the three-dimensional shape data. This display can satisfy the user's desire to own the device and enhance the manufacturer or seller's promotional efforts.

[0095] [Composition of Virtual Space]

[0096] In addition, the virtual space may be a single space or may be divided into multiple spaces (e.g., a personal room or conference room, a virtual store such as a retail store or restaurant, a facility such as a site, a city, etc.). Furthermore, the virtual space management server 20 may set up in the virtual space a "virtual store where users can obtain (download, purchase, etc.; the same applies hereinafter) information representing an avatar" or a "virtual store where three-dimensional shape data of a camera device is displayed or exhibited so that users can obtain the data", or a "virtual showroom where a captured virtual image or the virtual print is displayed so that users can obtain the image" (see Figure 18 )wait.

[0097] [Output of virtual space status]

[0098] The communication control unit 22A and virtual space management unit 22B (third processor) of the virtual space management server 20 transmit the virtual space status to the user system 10 via the network NW (communication line) and output it to the goggles 200 (image output device) (step S150). "Virtual space status" may include not only the position, shape, and movement of the background, subject, and avatar in the virtual space, but also the sounds and lighting (brightness, saturation, etc.) within the virtual space and their temporal changes. "Output" may include display on the display 204 (image output device, image display device) and audio output from the speaker 216. The communication control unit 22A and virtual space management unit 22B may display not only the avatar of the user of the camera device 100, but also the avatars of other users participating in the virtual space. The virtual space management server 20 preferably updates this output at a predetermined rate.

[0099] In addition, the virtual space management server 20 can also generate an image (for example, as described later) showing the state of the virtual space from a predetermined position (viewpoint) such as a bird's-eye view or a bird's-eye view, similar to a game played on a smartphone or PC. Figure 15 The virtual space management server 20 may change the viewpoint (position and / or direction) in accordance with the user's operation or movement in the virtual space, and may also change the output image and / or audio according to the change in viewpoint.

[0100] Users participating in the virtual space can engage in conversations (communication with other users) with other users under the control of the virtual space management unit 22B, thereby enabling meetings and the like within the virtual space. Furthermore, conversations within the virtual space can be conducted not only through conventional voice conversations via the microphone 214 and speaker 216, but also through text-based chats with specific users, and through text displays within the virtual space using dialog boxes or the like (which can be configured to allow visual identification of all participants in the virtual space).

[0101] Furthermore, it is preferable that the virtual space management server 20 continuously performs the above-mentioned “output of the virtual space status” at a predetermined rate.

[0102] [Display of 3D Shape Data of Avatar and Camera Device]

[0103] When the user of the camera 100 instructs the user to transmit the three-dimensional shape data, the virtual space management server 20 displays the three-dimensional shape of the camera 100 in association with the avatar near the virtual space (step S150). Specifically, the virtual space management unit 22B and the communication control unit 22A update the virtual space to display the three-dimensional shape of the camera 100 in association with the avatar near the virtual space, and transmit the virtual space information to the user system 10 for display on the display 204 of the goggles 200.

[0104] Here, "displaying nearby" can, for example, involve displaying three-dimensional shape data within a specified distance from the avatar, or displaying it on a specified location such as the avatar's fingers or chest. This display can inform (demonstrate) to other users that the user is holding a camera and can capture virtual images. Furthermore, when displaying an avatar, the virtual space management server 20 can associate the user's ID or nickname with the avatar and display it near the avatar. This allows participants in the virtual space to understand the identity of the user represented by the avatar.

[0105] Figure 9This figure shows a state where an avatar 600 of the user (user of the imaging device 100), three-dimensional shape data 610 of the imaging device 100, and avatars 700 of other users are displayed in a virtual space (virtual live view image 500, described later). In the example of this figure, the three-dimensional shape data 610 is displayed in the hand area of ​​avatar 600. This display is a method of associating the data representing the three-dimensional shape of the imaging device with the avatar and displaying it near the avatar.

[0106] The virtual space management server 20 can turn on or off the display of the three-dimensional shape or ID, etc., of the camera device 100 based on user instructions (operation, action, voice input, etc.). Furthermore, if the connection between the camera device 100 and the goggles 200 is disconnected, the camera device 100 and goggles 200 can attempt to reconnect, or the goggles 200 can notify the virtual space management server 20 of the situation and delete the depiction of the camera device data from the virtual space.

[0107] [Transition to virtual camera mode]

[0108] If the mode control unit 106D (processor 106; first processor) and the processor 202 (second processor) determine that the predetermined conditions are satisfied ("YES" in step S160), the imaging device 100 and the goggles 200 transition to a virtual imaging mode in which a request for capturing a virtual image (second image) and setting second imaging parameters are received (step S170). The second imaging parameters are imaging conditions for capturing a virtual image (second image) by virtually capturing an object in a virtual space.

[0109] [Example of mode transition condition]

[0110] Examples of mode transition conditions are given below, but the mode transition conditions in the present invention are not limited to these examples.

[0111] An example of the transition condition is that the user places the imaging device 100 in a predetermined position and posture, such as holding the imaging device 100 close to the user's head or taking a posture of looking through a viewfinder. Figure 10 This is a diagram showing an example of mode transition due to user actions. Figure 10 Part (a) in FIG. 2 shows the user 2 wearing the goggles 200. Figure 10 Part (b) shows the user 2 holding the camera device 100 close to their head and looking through the viewfinder. This movement of the user 2 can be detected by the motion sensor 118 or the like. However, if the goggles 200 include a motion sensor or the user system 10 includes motion sensors attached to the user's hands and feet, these motion sensors can also be used to detect the movement.

[0112] If these motion sensors detect movement, the mode control unit 106D and the processor 202 determine that "the predetermined conditions are met", and the camera device 100 and the goggles 200 shift to a virtual camera mode that accepts camera requests for a virtual image (the second image) and the setting of camera parameters for the virtual image (the second camera parameters).

[0113] Another example of a transition condition is when the user performs an action such as shaking or rotating the imaging device 100. Alternatively, the user may perform a predetermined operation (such as operation of a specific button or switch) on the operating unit 110 of the imaging device 100 and / or perform a predetermined operation on the operating unit 206 of the goggles 200.

[0114] The transition condition is not limited to operations on the camera device 100 or the goggles 200. Another example of the transition condition is as follows: (1) the processor 202 of the goggles 200 recognizes a specific sentence (e.g., "transfer to virtual camera mode" or "take a photo") via voice recognition of the microphone 214; (2) another user in the virtual space speaks a specific sentence (e.g., "let's take a photo together") and the processor 202 performs voice recognition on the sentence.

[0115] The transition conditions can be a combination of the above conditions. For example, if another user in the virtual space asks a prescribed question, such as "Let's take a photo together," and the user of the camera 100 responds with a prescribed response, such as "OK," "Yes," or "Confirm," or performs a prescribed action and / or operation, the mode control unit 106D and processor 202 can determine that the "predetermined condition has been met." Similarly, the conditions for terminating the virtual camera mode can be based on the position and / or posture of the camera 100, an operation, a movement, voice recognition, or a combination of these.

[0116] [Acquisition and transmission of the second imaging parameter]

[0117] When the camera device 100 and the goggles 200 are switched to the virtual camera mode, the processor 106 and the processor 202 obtain the second camera parameter and send it to the virtual space management server 20 ( Figure 8 The second imaging parameter includes at least one of focal length, focus distance, shutter speed, aperture value, and image quality. The acquired and transmitted second imaging parameter may be an initially set parameter or a parameter that has been modified from the initially set state (see step S200 described later).

[0118] [Switching to the first-person viewpoint image in accordance with the mode transition]

[0119] Furthermore, when the camera device 100 and goggles 200 switch to virtual image capture mode, the virtual space management server 20 is notified of the mode change. Based on this notification, the virtual space management server 20 outputs information appropriate to the virtual image capture mode. Specifically, the virtual space management server 20 switches the display 204 of the goggles 200 to a perspective (either the avatar of the user of the camera device 100 or a first-person viewpoint image viewed from the camera device within the virtual space).

[0120] [Update of first-person viewpoint image (virtual live view image)]

[0121] The processor 22 (third processor) of the virtual space management server 20 generates a first-person viewpoint image corresponding to the received second imaging parameters and displays it on the display 204 (step S190). Furthermore, the processor 22 can update this first-person viewpoint image at a predetermined rate, transmit the updated first-person viewpoint image to the user system 10, and display it on the display 204. The image thus updated is identical to the live view image captured in the real space and is hereinafter referred to as the "virtual live view image."

[0122] Figure 11 2 is a diagram showing a state where a virtual live view image 500 (an image at a specific point in time) is displayed on the display 204. The virtual live view image 500 includes an area 502 and an area 504. In the area 502, the brand name and model of the imaging device indicated by the three-dimensional shape data, the characteristics of the optical system installed (in Figure 11 In the example of , the second imaging parameter set at the time of display is displayed in area 504. Figure 11 In the example shown in FIG, the white balance is set to AUTO, the shutter speed is 1 / 60 sec, the aperture is 1.8, and the ISO sensitivity is 400. In addition, avatars 700 of other users (four persons) are displayed in the virtual live view image 500 .

[0123] [Updating of virtual live view images in response to movement or changes in direction]

[0124] The processor 106 of the camera device 100 includes a motion sensor 118 and a measuring unit 106C (first processor, processor). These constitute a measuring unit that measures the camera position (first camera position) and camera direction (first camera direction) of the camera device 100 in real space. The measuring unit 106C sets the camera position (second camera position) and camera direction (second camera direction) of the camera device in virtual space based on the measured first camera position and first camera direction, and transmits information indicating the second camera position and the second camera direction to the virtual space management server 20. The processor 22 of the virtual space management server 20 (primarily the virtual space management unit 22B, the virtual image generation unit 22C; the third processor) can generate a virtual live view image based on the second camera position and the second camera direction.

[0125] Furthermore, when the measured first camera position and first camera direction are changed, the processor 106 (first processor) transmits information indicating the changed second camera position and information indicating the changed second camera direction to the virtual space management server 20. The processor 22 can generate a virtual live view image ( Figure 7 Steps S190, S210).

[0126] Thus, in the imaging system 1 according to this embodiment, if a user moves or changes the orientation of the imaging device in real space, the user's avatar and the imaging device in virtual space will also move accordingly, and the orientation of the imaging device in virtual space will also change. This allows the user to move to a desired location in virtual space and capture images in a desired orientation.

[0127] Alternatively, the user system 10 may transmit information regarding the first camera position and the first camera direction to the virtual space management server 20, and the virtual space management server 20 may calculate and set the second camera position and the second camera direction. By having the virtual space management server 20 calculate and set the second camera position and the second camera direction in this manner, the burden on the user system 10 can be reduced.

[0128] [Change of the second imaging parameter]

[0129] The above-mentioned second imaging parameter can be changed according to the user's instructions. Specifically, the imaging parameter setting unit 106B (first processor) of the imaging device 100 can set the second imaging parameter according to the operation (setting instruction) of the imaging parameter setting component. As the "imaging parameter setting component", a button or dial not shown in the figure constituting the operating unit 110 can be used, and the second imaging parameter can also be set and changed by the same component and the same operation as the case where the first imaging parameter in the shooting of the actual image can be set. By adopting this method, the operability of virtual image shooting can be improved. In addition, in the case where the display 108 of the imaging device 100 is a touch panel display, the display 108 can be used as a "imaging parameter setting component". In addition, the second imaging parameter can also be set and changed via the display 204 of the goggles 200 (refer to the later-described Figure 12 In addition, the second camera parameter can also be set and changed through user actions (gestures) or voice input.

[0130] When an operation (setting instruction) is performed on the imaging parameter setting component to change the second imaging parameter ("YES" in step S200), the imaging parameter setting unit 106B and the communication control unit 106E (first processor) transmit the changed (setting instruction) second imaging parameter to the virtual space management server 20 via the goggles 200 (server connection device). This transmission can utilize the aforementioned Web API. The virtual space management server 20 reflects the change in the second imaging parameter in the virtual live view image (described later).

[0131] [Actual parameters and virtual parameters]

[0132] The second imaging parameters (imaging parameters for capturing a virtual image) may be parameters realized by the actual imaging optical system included in the imaging device 100 (actual parameters) or parameters realized by a virtual imaging optical system virtually included in the imaging device 100 (virtual parameters). An example of setting the actual parameters is setting the focal length within the range of 25mm-200mm by rotating the lens barrel when a zoom lens with a focal length of 25mm-200mm is actually installed in the imaging device 100. Furthermore, unlike capturing an actual image, capturing a virtual image does not involve capturing an image formed by the imaging optical system 102 of the imaging device 100. The lens barrel rotation operation in the above example serves as the operation for setting the second imaging parameters.

[0133] On the other hand, a parameter (virtual parameter) that cannot be obtained in the optical system connected to the camera device 100 can also be set as the second camera parameter. In the above example, when the focal length of the zoom lens actually installed on the camera device 100 is 25mm to 200mm, the focal length can be changed within the range of 10mm to 25mm or 200mm to 300mm. Moreover, even if it is a lens that does not exist in the real world, if there is lens data corresponding to the lens, the focal length can be determined based on the lens data (use of a virtual lens). The same applies to other camera parameters other than the focal length. By setting such a virtual parameter as the second camera parameter, the convenience of virtual image capture can be improved, and the fun of special lenses unique to the virtual space can be provided.

[0134] [Setting the Second Camera Parameters in Virtual Space]

[0135] The setting instruction (setting or changing) of the second imaging parameter related to the imaging of the virtual image can be performed in the virtual space. Figure 12 This is a diagram showing the setting or change status of the second imaging parameter in the virtual space. Figure 12 Part (a) shows a state in which the imaging device 100 and the goggles 200 have switched to the virtual image capturing mode and a virtual live view image 500 is displayed on the display 204. From this state, the display on the display 204 switches to a second imaging parameter setting screen in response to a user instruction (predetermined operation, action, voice, etc.).

[0136] Figure 12 Part (b) in the figure shows the state in which the virtual space management server 20 displays the second camera parameter setting screen (screen displayed in the virtual space) on the display 204. The second camera parameter setting screen includes areas 506, 508, and 510, an OK button 512, and a Cancel button 514. In area 506, the upper-level items of the menu are displayed as icons, and in areas 508 and 510, the lower-level (specific) items of the menu are displayed. In the example shown in this part, in the setting of the autofocus (button 506A), the focus mode setting and AF-S (single-shot AF) are selected. In such a screen, menu selection or confirmation, setting completion, and other processing can be performed based on user actions (gestures), voice recognition, etc.

[0137] The second imaging parameter setting screen may be the same as or similar to the actual setting screen of the imaging device 100 (the screen displayed on the display 108 when setting the first imaging parameters as the imaging conditions for actual shooting), or may be a separate screen. When using a screen that is the same as or similar to the actual setting screen, information on the imaging parameter setting screen of the imaging device 100 may be transmitted to the virtual space management server 20, and the virtual space management server 20 may display the setting screen in the virtual space based on this information.

[0138] Furthermore, the virtual space management server 20 only needs to display (transmit) the second imaging parameter setting screen to the user of the imaging device 100, and does not need to display it to other users. Furthermore, while the virtual space management server 200 has been described as displaying the setting screen, the processor 202 of the goggles 200 may generate the setting screen and display it on the display 204, as long as the setting screen is not displayed to other users.

[0139] If the user wears the goggles 200, the field of vision may be blocked, making it difficult to operate the operating unit 110 of the camera device 100 or the operating unit 206 of the goggles 200. However, by operating in the virtual space through the display 204, the user can easily set the second camera parameter while wearing the goggles 200.

[0140] [Updating of Virtual Live View Image in Accordance with Change of Second Imaging Parameter]

[0141] The processor 22 of the virtual space management server 20 (mainly the communication control unit 22A, the virtual image generation unit 22C, the virtual image display control unit 22D; the third processor) displays a virtual live view image corresponding to the change in the second camera parameter on the display 204 of the goggles 200 (step S210). Figure 13 Indicates that the focal length has been changed (in Figure 11 FIG. 5 is a diagram showing an example (a case where zooming is performed on the area 510 in FIG. 5 ) (virtual live view image 500A), Figure 14 This figure shows an example (virtual live view image 500B) of a situation where the exposure is changed (the aperture value is increased). The processor 22 and display 204 continue to display the virtual live view image (updated at a predetermined rate) until a request to capture a virtual image is received (while the answer is "No" in step S220).

[0142] [Acquisition and output of virtual images]

[0143] If a second imaging instruction is provided ("YES" in step S220), a request to capture at least a virtual image is transmitted from the user system 10 to the virtual space management server 20. If the latest second imaging parameters have not been transmitted, they may be transmitted along with the imaging request. The second imaging instruction may be an operation of the operating unit 110 (such as a release switch for actual imaging) and / or the operating unit 206, or may be a request based on a predetermined user action (such as a gesture) or voice (for example, utterance of a specific phrase such as "capture virtual image").

[0144] In response to the image capture request, the processor 22 of the virtual space management server 20 (primarily the virtual image generation unit 22C; the third processor) generates a virtual image (second image) based on the set second image capture parameters and transmits it to the goggles 200 (server connection device, image output device). The goggles 200 receives image data of the virtual image from the virtual space management server 20 via the network NW (communication line) (acquisition of the virtual image) and displays at least the virtual image corresponding to the received image data on the display 204 (output device, image output device) (step S230). The virtual space management server 20 may transmit information such as the capture date and time of the virtual image, the location within the virtual space, the second image capture parameters, and identification information or ID of the user who participated in the capture (accompanying information of the virtual image) to the goggles 200, which then outputs this information along with the virtual image to the display 204, etc.

[0145] The virtual space management server 20 can generate either a static image or a dynamic image as a virtual image. The virtual space management server 20 can determine which image to generate based on user instructions (such as operation of an operating element (not shown) such as a release button, user gestures, or voice input). Information on whether to generate a static or dynamic image may also be included in the recording request or second recording parameters. Furthermore, the virtual image may include voice data. The recording control unit 106G (processor 106; first processor) of the camera 100 stores the captured virtual image on the memory card 116 (output device) based on user instructions (such as operation of the operating unit 110, gestures, or voice input). The stored virtual image can be displayed on the display 108. Furthermore, if the camera 100 is a camera with a printer, the virtual image can be printed (output) using its printer (image output device). The recording control unit 106G may also store or display the aforementioned supplementary information on a recording medium such as the memory card 116, or output it by printing it together with the virtual image. When the supplementary information is output, it is preferably associated with the virtual image.

[0146] The goggles 200 end the display of the virtual image in response to a user instruction (operation of the operation unit 110 and / or the operation unit 206 , user's motion, voice, etc.) and return to the display of the virtual live view image.

[0147] When the user issues an instruction to end virtual image capture (which can be an instruction based on operation, action, voice, etc.) ("Yes" in step S240), the camera device 100 and the goggles 200 end outputting the first-person viewpoint image (virtual live view image) and return to step S150 (the state of outputting the state of the virtual space). When there is no instruction to end virtual image capture ("No" in step S240; the virtual image capture continues), the process returns to step S200.

[0148] [Display of virtual images, etc.]

[0149] Users can display the acquired virtual image in a virtual space. This display can be the image itself or a printed object with the image. Figure 15 This figure shows how virtual images are displayed in a virtual space. In this figure, virtual images 802 and 804 are displayed on the walls of a virtual image display room 800 set up in a virtual space. Figure 7 The content described in step S150 is similarly generated by the virtual space management server 20 (processor 22; third processor) and output to the goggles 200.

[0150] The virtual image thus displayed can be obtained (downloaded) by the desired user. If the user instructs the display or acquisition of such a virtual image (e.g., by operating the operating unit of the camera device 100, performing a predetermined action, or uttering a predetermined phrase), the instruction is notified to the virtual space management server 20, which then displays and / or provides the virtual image (to the user system of the desired user) based on the notification. Furthermore, in addition to virtual images, actual images can be uploaded from the camera device 100 to the virtual space management server 20 and displayed in the virtual space's exhibition room. Furthermore, the display of images is not limited to images; documents, charts, and the like can also be displayed in the virtual space.

[0151] As described above, according to the imaging system 1 of the present embodiment, it is possible not only to capture a virtual image but also to provide the user with the pleasure of presenting (displaying) or providing the virtual image or the like.

[0152] As described above, according to the imaging system 1 of this embodiment, real images and virtual images can be captured using the imaging device 100 , and data representing the three-dimensional shape of the user's avatar or the imaging device can be displayed in a virtual space.

[0153] [Application of virtual image photography]

[0154] When capturing virtual images, as with capturing images in real space, it is considered that a user may "want to capture a virtual image that includes themselves (their avatar) in a virtual space (want to take a so-called 'selfie' or 'group photo')." The following describes an embodiment that takes this situation into consideration.

[0155] [Fixing the camera position and direction in virtual space]

[0156] Figure 16 This figure illustrates a situation where the camera position and camera direction in a virtual space are fixed. Part (a) of the figure shows a situation where user 2 is looking through the viewfinder with the camera device 100 positioned near their head (or near the goggles 200). This satisfies the aforementioned "predetermined condition," and the camera device 100 and goggles 200 transition to virtual camera mode. In this state, if the user issues a specified instruction, the processor 106 (processor, first processor) fixes the camera position (second camera position) and camera direction (second camera direction) in the virtual space based on the user's instruction to the camera device 100 via the virtual space management server 20. Figure 16 Part (b) shows a state where the imaging position and imaging direction are fixed. In this state, the imaging position and imaging direction of the virtual live view image and the captured virtual image output to the display 204 of the goggles 200 are fixed. Therefore, the user can move the imaging device 100 downward (without having to maintain the same posture).

[0157] The "instruction" to fix the camera position and camera direction can be a predetermined operation on the operating unit 110 and / or the components of the operating unit 206 or the display 108, a specific action performed by the user (such as shaking a hand or a foot), or an instruction based on voice (saying a specific phrase such as "position fixed" or "direction fixed"). The user's action can be detected by the motion sensor 118 and the measurement unit 106C, and the instruction based on voice can be input via the microphone 214. Two or more of the operation, action, and voice can also be combined as an "instruction". In addition, these instructions can also be given in the case of releasing the fixation.

[0158] If the camera position and camera direction in the virtual space are fixed, the user can move to any position to capture a virtual image, or even capture themselves. Furthermore, when the camera position and camera direction are fixed, the user system 10 (camera 100 and goggles 200) can automatically perform Selfie photography as described below.

[0159] [Selfie video recording of virtual images]

[0160] After a predetermined time has passed since the second image capture instruction was issued, the processor 106 (primarily the image acquisition unit 106A and the communication control unit 106E; the first processor) transmits a capture request for the second image to the virtual space management server 20. This "self-timer capture" makes it easy to capture the aforementioned selfie or group photo (virtual image) of oneself. Furthermore, the "predetermined time (self-timer time)" can be set by operating the operating unit 110 and / or the operating unit 206, by user actions, by voice input, or by a combination of these. If the camera device 100 includes an operating component such as a switch or lever for Selfie capture in real space, this operating component can also be used when capturing virtual images.

[0161] Figure 17 This diagram shows a situation where a picture is taken using the Selfie function. Figure 17 Part (a) shows a state where a virtual live view image 500 is displayed on the display 204 (similar to the state shown in FIG. Figure 11 In this state, the camera position and camera direction are fixed. In this state, the user can move away from the position of the camera in the virtual space, such as Figure 17 As shown in part (b) of FIG. 5 , the field of view of the virtual live view image 500D can be entered. In this part, the avatar 600 is the avatar of the user of the camera 100 , and the avatar 700 is the avatar of another user.

[0162] Figure 18 This is another diagram showing a state of shooting using the Selfie function. Figure 18Part (a) in the figure shows a state in which the camera position and camera direction are fixed in the virtual image camera mode, and the user himself enters the field of view (1 frame of the virtual live view image). Part (b) in the figure shows a state in which the user performs an action to switch to selfie mode (an action of raising the right hand and shaking it left and right; a way of indicating the camera). The user's actual action is reflected in the action of avatar 600. Part (c) in the figure shows a state in which the user switches to selfie mode, and part (d) shows a state in which the user performs an action to set the selfie timer (an action of raising the left hand and shaking it left and right; a way of indicating the camera). In part (d), as in part (b), the user's actual action is reflected in the action of avatar 600. Part (e) shows a state in which a virtual image is captured after the timer time (predetermined time) has passed since the selfie timer was set.

[0163] In the camera system 1 of this embodiment, the Selfie function allows the user to enjoy taking selfies or photos with other users. Furthermore, the switch to the self-timer mode or the instruction to take a photo can be not only from the user of the camera device 100 but also from another user (e.g., an avatar's movement or voice reflecting the movement of the other user).

[0164] [Modification of User System]

[0165] In the first embodiment described above, the user system 10 is configured using the goggles 200 as the server connection device, but the configuration of the user system is not limited to this embodiment. Figure 19 FIG is a diagram showing a modified example of a user system. Figure 19 In the illustrated user system 10A, a smartphone 400 is used as a "server connection device," and a display 410 of the smartphone 400 is used for displaying virtual images, etc. The smartphone 400 and the camera 100 can be connected via wireless communication such as Bluetooth. The smartphone 400 is connected to the router 300 via wireless communication such as Wi-Fi, and communicates with the virtual space management server 20. Alternatively, a network-connectable device such as a personal computer or tablet terminal can be used as the server connection device in place of the smartphone 400.

[0166] While these methods reduce the sense of immersion in the virtual space, they do not obstruct the user's field of view, allowing users to easily capture virtual images. Furthermore, when using a device with a built-in camera as a server connection device, it is possible to detect user movements from images captured by the device's built-in camera, and to assign information such as "expressions" or "emotions" to the avatar based on information extracted from the images.

[0167] [Second embodiment]

[0168] In the first embodiment and the modified example, the user system is composed of the imaging device and the server connection device. However, the user system may be composed of the imaging device alone (the imaging device 100A may serve as both the "imaging device" and the "server connection device"). Figure 20 This diagram shows the configuration of a user system according to the second embodiment. In the second embodiment, an imaging device 100A independently constitutes a user system 10B. In user system 10B, imaging device 100A connects to a virtual space management server 20 via wireless communication such as Wi-Fi, and performs tasks such as setting second imaging parameters and capturing virtual images.

[0169] Figure 21 1 is a diagram showing the structure of an imaging device 100A according to the second embodiment. Figure 7 As shown, in addition to the configuration of the imaging device 100 according to the first embodiment, the imaging device 100A further includes a microphone 122 and a speaker 124, which enable voice input and output (the processor 107 has a voice recognition function). Furthermore, the display 108 can be used to set the second imaging parameter or display virtual images. In this embodiment, as in the aforementioned modified example, although the immersion in the virtual space is reduced, the user's vision is not obstructed, and the user can easily capture virtual images, etc.

[0170] As mentioned above, although embodiment and modification of this invention were described, this invention is not limited to the said form, Various modifications are possible.

[0171] Description of Reference Numerals

[0172] 1-Camera system, 2-User, 10-User system, 10A-User system, 10B-User system, 20-Virtual space management server, 22-Processor, 22A-Communication control unit, 22B-Virtual space management unit, 22C-Virtual image generation unit, 22D-Virtual image display control unit, 28-Database, 100-Camera device, 100A-Camera device, 102-Camera optical system, 104-Imaging element, 106-Processor, 106A-Image acquisition unit, 106B-Camera parameter setting unit, 106C-Measurement unit, 106D-Mode control unit, 106E-Communication control unit, 106F-Display control unit, 106G-Recording control unit, 107-Processor, 108-Display, 110-Operation unit, 116-Memory card, 118-Motion sensor, 120-Wireless communication interface, 122-Microphone, 124-speaker, 200-goggles, 202-processor, 204-display, 206-operating unit, 212-wireless communication interface, 214-microphone, 216-speaker, 218-microphone, 220-speaker, 300-router, 400-smartphone, 410-display, 500-virtual live view image, 500A-virtual live view image, 500B-virtual live view image, 500D-virtual live view image, 502-area, 504-area, 506-area, 506A-button, 508-area, 510-area, 512-confirm button, 514-cancel button, 600-avatar, 610-three-dimensional shape data, 700-avatar, 800-virtual image exhibition room, 802-virtual image, 804-virtual image, S100-S240-steps of the imaging method.

Claims

1. A camera device comprising a processor, The processor performs the following processing: Setting first imaging parameters and second imaging parameters, wherein the first imaging parameters are imaging parameters for acquiring a first image by imaging an object in a real space using an imaging optical system and an imaging element, and the second imaging parameters are imaging parameters for acquiring a second image by virtually imaging the object in a virtual space; and According to the first imaging instruction, the first image is acquired based on the first imaging parameter. When a predetermined condition is satisfied, the imaging device transitions to a virtual imaging mode for accepting an imaging request for the second image and setting the second imaging parameter. The processor performs the following processing in the virtual camera mode: When there is a setting instruction for the second imaging parameter, transmitting the second imaging parameter of the setting instruction to a virtual space management server that manages the virtual space via a communication line; transmitting, in response to a second imaging instruction, at least the imaging request for the second image to the virtual space management server via the communication line; receiving image data of the second image corresponding to the transmitted second imaging parameter from the virtual space management server via the communication line; and The image output device is caused to output at least the second image corresponding to the image data.

2. The imaging device according to claim 1, wherein The processor performs the following processing: Setting the second imaging parameter according to an operation on the imaging parameter setting component; and When an operation to change the second imaging parameter is performed on the imaging parameter setting unit, the changed second imaging parameter is transmitted to the virtual space management server.

3. The imaging device according to claim 1 or 2, wherein: The processor sets, as the second imaging parameter, an actual parameter realized by an actual imaging optical system included in the imaging device or a virtual parameter realized by a virtual imaging optical system virtually included in the imaging device.

4. The imaging device according to claim 1 or 2, wherein: The imaging device includes a measuring unit that measures a first imaging position and a first imaging direction of the imaging device in the real space. The processor performs the following processing: setting a second imaging position and a second imaging direction of the imaging device in the virtual space based on the measured first imaging position and the first imaging direction; sending information indicating the second camera position and information indicating the second camera direction to the virtual space management server; and Image data of the second image corresponding to the second imaging position and the second imaging direction is received from the virtual space management server. The imaging device according to claim 4 , wherein: When the measured first imaging position and the measured first imaging direction are changed, the processor transmits information indicating the changed second imaging position and information indicating the changed second imaging direction to the virtual space management server. The imaging device according to claim 4 , wherein: The processor fixes the second imaging position and the second imaging direction in the virtual space according to a user instruction to the imaging device through the virtual space management server.

7. The imaging device according to claim 1 or 2, wherein: The processor transmits the capture request of the second image to the virtual space management server after a predetermined time has elapsed since the second capture instruction was issued.

8. The imaging device according to claim 1 or 2, wherein: The imaging device includes a first storage unit that stores avatar information representing an avatar of a user of the imaging device. The processor transmits the stored avatar information to the virtual space management server, and causes the avatar of the user to be displayed in the virtual space through the virtual space management server.

9. The imaging device according to claim 8, wherein: The imaging device includes a second storage unit that stores data representing a three-dimensional shape of the imaging device. The processor transmits the stored data to the virtual space management server, and causes the virtual space management server to display the three-dimensional shape of the camera device in correspondence with the avatar near the avatar in the virtual space.

10. The imaging device according to claim 1 or 2, wherein: The second image is a static image or a dynamic image.

11. A camera system comprising a camera device and a server connection device, wherein: The imaging device includes a first processor, and the first processor performs the following processing: Setting first imaging parameters and second imaging parameters, wherein the first imaging parameters are imaging conditions for acquiring a first image by capturing an object in a real space using an imaging optical system and an imaging element, and the second imaging parameters are imaging conditions for acquiring a second image by virtually capturing the object in a virtual space; and According to the first imaging instruction, the first image is acquired based on the first imaging parameter. When predetermined conditions are satisfied, the imaging device and the server connection device transition to a virtual imaging mode for accepting an imaging request for the second image and setting the second imaging parameters. The first processor performs the following processing in the virtual camera mode: When there is a setting instruction for the second imaging parameter, transmitting the second imaging parameter of the setting instruction to a virtual space management server that manages the virtual space via the server connection device; and In response to a second imaging instruction, at least the imaging request for the second image is transmitted to the virtual space management server via the server connection device. The server connection device includes a second processor and an image output device, and the second processor performs the following processing in the virtual camera mode: sending a capture request for the second image and the second capture parameters to the virtual space management server via a communication line; receiving image data of the second image corresponding to the transmitted second imaging parameter from the virtual space management server via the communication line; and The image output device is caused to output at least the second image corresponding to the received image data.

12. The imaging system according to claim 11, wherein: The server connection device is a goggle-type device worn by a user and includes a display device for displaying the second image as the image output device.

13. The imaging system according to claim 11 or 12, wherein: The camera system further includes the virtual space management server. The virtual space management server performs the following processing: generating image data of the second image corresponding to the second imaging parameters based on the imaging request for the second image and the imaging conditions received from the server connection device; and The generated image data is transmitted to the server connection device.

14. The imaging system according to claim 13, wherein: The virtual space management server causes the virtual space management server to display the second image in the virtual space in response to an operation performed by a user of the imaging device.

15. The imaging system according to claim 13, wherein: The virtual space management server transmits the image data of the second image generated in response to the imaging request received from one of the imaging devices connected to the virtual space management server to another imaging device in response to a request from the other imaging device connected to the virtual space management server.

16. A camera method, which is performed by a camera device having a processor, wherein: The processor performs the following processing: Setting first imaging parameters and second imaging parameters, wherein the first imaging parameters are imaging parameters for acquiring a first image by imaging an object in a real space using an imaging optical system and an imaging element, and the second imaging parameters are imaging parameters for acquiring a second image by virtually imaging the object in a virtual space; and According to the first imaging instruction, the first image is acquired based on the first imaging parameter. When a predetermined condition is satisfied, the imaging device transitions to a virtual imaging mode for accepting an imaging request for the second image and setting the second imaging parameter. The processor performs the following processing in the virtual camera mode: When there is a setting instruction for the second imaging parameter, transmitting the second imaging parameter of the setting instruction to a virtual space management server that manages the virtual space via a communication line; transmitting the capture request of the second image to the virtual space management server via the communication line in response to a second capture instruction; receiving image data of the second image corresponding to the transmitted second imaging parameter from the virtual space management server via the communication line; and The image output device is caused to output at least the second image corresponding to the image data.

17. A camera method, the method being performed by a camera system comprising a camera device and a server connection device, the camera device comprising a first processor, the server connection device comprising a second processor and an image output device, In the imaging method, the first processor performs the following processing: Setting first imaging parameters and second imaging parameters, wherein the first imaging parameters are imaging conditions for acquiring a first image by capturing an object in a real space using an imaging optical system and an imaging element, and the second imaging parameters are imaging conditions for acquiring a second image by virtually capturing the object in a virtual space; and According to the first imaging instruction, the first image is acquired based on the first imaging parameter. When predetermined conditions are satisfied, the imaging device and the server connection device transition to a virtual imaging mode for accepting an imaging request for the second image and setting the second imaging parameters. The first processor performs the following processing in the virtual camera mode: When there is a setting instruction for the second imaging parameter, transmitting the second imaging parameter of the setting instruction to a virtual space management server that manages the virtual space via the server connection device; and According to the second imaging instruction, the imaging request of the second image is transmitted to the virtual space management server via the server connection device, The second processor performs the following processing in the virtual camera mode: sending a capture request for the second image to the virtual space management server via a communication line; receiving image data of the second image corresponding to the second imaging parameter from the virtual space management server via the communication line; and The image output device is caused to output at least the second image corresponding to the received image data.

Images