Data acquisition device, image acquisition device, data structure, data acquisition method, image acquisition method, program, and recording medium

Abstract

Provided are a data acquisition device, an image acquisition device, a data structure, a data acquisition method, an image acquisition method, a program, and a recording medium with which it is possible to acquire an image close to that in real space through imaging in a virtual space.　A data acquisition device according to one embodiment of the present invention is equipped with a processor which acquires real photographic subject data corresponding to virtual object data for displaying a virtual object in a virtual space, on the basis of the virtual object data and a display condition for displaying a virtual space related to the virtual object.

Description

Data acquisition device, image acquisition device, data structure, data acquisition method, image acquisition method, program, and recording medium

[0001] One embodiment of the present invention relates to a data acquisition device, an image acquisition device, a data structure, a data acquisition method, an image acquisition method, a program, and a recording medium for acquiring real subject data corresponding to virtual object data.

[0002] A user can set an avatar (a character that resembles the user) in a virtual space (metaverse) that replicates the real world, and can perform various activities in the virtual space through the avatar. As an example, a user can operate an avatar in the virtual space and take pictures of the virtual space using a virtual camera that resembles a real camera (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2009-176025

[0004] Images captured in virtual spaces are merely CG (Computer Graphics) or animations. By making these images appear as if they were captured in real space, users can experience the experience of visiting the location in real space and taking the photos. In other words, capturing images that resemble the real world through virtual space photography can provide users with a more valuable experience.

[0005] One embodiment of the present invention has been made in consideration of the above circumstances, and aims to provide a data acquisition device, an image acquisition device, a data structure, a data acquisition method, an image acquisition method, a program, and a recording medium that are capable of acquiring images that are close to those in real space through shooting in a virtual space.

[0006] To solve this problem, the present invention has the following configuration. [1] A data acquisition device including a processor, wherein the processor acquires real subject data corresponding to the virtual object data based on virtual object data for displaying a virtual object in a virtual space and display conditions for displaying a virtual space related to the virtual object. [2] The data acquisition device according to [1], wherein the processor associates the acquired real subject data with virtual object data. [3] The data acquisition device according to [1] or [2], wherein the real subject data is data for displaying a real subject corresponding to a virtual object. [4] The data acquisition device according to any of [1] to [3], wherein the real subject data is an image obtained by photographing a real subject corresponding to a virtual object in real space. [5] The data acquisition device according to any of [1] to [4], wherein the real subject data is a three-dimensional image of a three-dimensional shape of a real subject corresponding to a virtual object. [6] An image acquisition device including a processor, wherein the processor accepts image acquisition conditions for acquiring a virtual image displaying a virtual space related to a virtual object, and acquires a real subject image of a real subject corresponding to the virtual object based on the virtual image and the image acquisition conditions. [7] The image acquisition device according to [6], wherein the processor generates a composite image corresponding to the virtual image by combining multiple acquired real object images. [8] The image acquisition device according to [6] or [7], wherein the virtual image includes an avatar and a background as virtual objects. [9] The image acquisition device according to any of [6] to [8], wherein the image acquisition conditions include shooting conditions for shooting with a virtual camera that simulates a real camera in a virtual space.

[10] The image acquisition device according to any of [6] to [9], wherein the image acquisition conditions include environmental conditions related to a virtual environment that simulates a real environment in the virtual space.

[11] The image acquisition device according to any of [6] to

[10] , wherein the image acquisition conditions include conditions related to the background in the virtual space.

[12] The image acquisition device according to any of [6] to

[11] , wherein the processor corrects the real object image based on the image acquisition conditions.

[13] The image acquisition device according to any one of [6] to

[12] , wherein the processor corrects the real subject image according to the state of the virtual object displayed in the virtual image.

[14] The image acquisition device according to any one of [7] to

[13] , wherein the processor associates image acquisition conditions with the acquired composite image and stores them.

[15] A data structure having virtual object data used in a process of displaying a virtual object in a virtual space, and display conditions that are used together with the virtual object data in a process of acquiring real subject data corresponding to the virtual object data and that display a virtual space related to the virtual object.

[16] The data structure according to

[15] , further having the acquired real subject data.

[17] A data acquisition method in which a processor performs a process of acquiring real subject data corresponding to the virtual object data based on virtual object data for displaying the virtual object in a virtual space and display conditions that display a virtual space related to the virtual object.

[18] An image acquisition method in which a processor performs the following processes: a process of accepting image acquisition conditions for acquiring a virtual image displaying a virtual space related to a virtual object; and a process of acquiring a real subject image of a real subject corresponding to the virtual object based on the virtual image and the image acquisition conditions.

[19] A program for causing a computer to execute each process included in the data acquisition method described in

[17] .

[20] A program for causing a computer to execute each process included in the image acquisition method described in

[18] .

[21] A computer-readable recording medium, on which a program for causing a computer to execute each process included in the data acquisition method described in

[17] is recorded.

[22] A computer-readable recording medium, on which a program for causing a computer to execute each process included in the image acquisition method described in

[18] is recorded.

[0007] According to one embodiment of the present invention, it is possible to provide a data acquisition device, an image acquisition device, a data structure, a data acquisition method, an image acquisition method, a program, and a recording medium that are capable of acquiring images that are close to real space through shooting in a virtual space.

[0008] FIG. 1 is a diagram illustrating an overview of image acquisition according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of the configuration of an image acquisition system including an image acquisition device according to an embodiment of the present invention. FIG. 3 is a diagram conceptually illustrating a management database that manages the association between virtual object data and real subject data. FIG. 4 is an explanatory diagram of functions of an image acquisition device according to an embodiment of the present invention. FIG. 5 is a diagram conceptually illustrating image acquisition processing in an image acquisition device according to an embodiment of the present invention. FIG. 6 is a diagram illustrating an example of a composite image. FIG. 7 is a diagram illustrating a procedure of an image acquisition flow. FIG. 8 is a diagram illustrating a composite image obtained by correcting a real subject image. FIG. 9 is a diagram illustrating a composite image obtained by correcting a real subject image.

[0009] A specific embodiment of the present invention will be described with reference to the drawings. However, the embodiment described below is merely an example given to facilitate understanding of the present invention and is not intended to limit the present invention. Furthermore, the present invention may be modified or improved from the following embodiment without departing from the spirit of the present invention. Furthermore, the present invention includes equivalents thereof.

[0010] In this specification, the concept of "device" includes a single device that performs a specific function by itself, as well as a combination of multiple devices that exist independently and in a distributed manner but work together (in cooperation) to perform a specific function.

[0011] In addition, in this specification, "image" refers to "image data" unless otherwise specified. Image data may include, for example, lossy compressed image data such as JPEG (Joint Photographic Experts Group) format, lossless compressed image data such as GIF (Graphics Interchange Format) or PNG (Portable Network Graphics) format, 3D image data used on a web browser such as VRML (Virtual Reality Modeling Language) format, and video data composed of multiple frame images. Image data may also include additional information indicating the file name, shooting date and time, shooting location, and other information.

[0012] Furthermore, one aspect of the data acquisition device according to the present invention is an image acquisition device according to the present invention, and the following description will be given based on an image acquisition device according to one embodiment of the present invention.

[0013] In this specification, the term "user" refers to a user who uses the image acquisition device according to the present invention. Using the image acquisition device means using the functions of the image acquisition device, and includes not only directly operating the image acquisition device but also using the functions of the image acquisition device through a device (e.g., a user terminal) that can communicate with the image acquisition device.

[0014] <<Outline of Main Image Acquisition>> Image acquisition (hereinafter, main image acquisition) performed using an image acquisition device and an image acquisition method according to an embodiment of the present invention will be described with reference to FIG.

[0015] The image finally acquired by this image acquisition is a composite image that imitates a real image captured in real space. The right diagram of FIG. 1 is an example of a composite image. A composite image is acquired by combining multiple images of a real subject captured in real space to recreate a scene in real space. As will be described later, a "real subject" refers to a person, object, background, etc. that exists in real space. As a result, the composite image becomes a pseudo-real image that appears as if it were captured in real space. In this image acquisition, a user first controls their own avatar in the virtual space and captures an image of the virtual space with a virtual camera that imitates a real camera at a desired capture location in the virtual space. As a result, the image acquisition device acquires a virtual image that displays a portion or the entire virtual space. The left diagram of FIG. 1 is an example of a live view screen of the virtual camera, and the center diagram of FIG. 1 is an example of a virtual image acquired by capturing an image with the virtual camera. In the following description, capturing an image with a virtual camera includes capturing a screenshot.

[0016] Note that specific data processing and the like when photographing a virtual space with a virtual camera are publicly known, as described in Patent Document 1, etc., and therefore will not be described in this specification. Furthermore, since objects, places, environments, etc. of the real space are reproduced in the virtual space, the virtual camera can be handled in the same way as a real camera, and specifically, photographing conditions can be set in the same way as for photographing in the real space.

[0017] A "virtual space" is, for example, a virtual space (metaverse) that reproduces real space, in which virtual objects (described later) are placed. Places that exist in real space, such as a city in Tokyo, are reproduced within the virtual space. Even if a user has never actually visited a place, the user can experience the sensation of visiting that place by operating their own avatar within the virtual space that reproduces that place. The user takes photos through their own avatar within the virtual space. The virtual camera allows the user to set shooting conditions similar to those of a real camera. Specifically, the user can set virtual aperture, focal length, shutter speed, exposure, ISO sensitivity, white balance, focus position, shooting direction, and angle of view. This allows the user to use the virtual camera to take photos under shooting conditions set to mimic those of a real camera.

[0018] The acquired virtual image, i.e., the captured image of the virtual space, is displayed on the screen of a user terminal (such as a smart device, tablet terminal, or notebook PC (Personal Computer)) used by the user. The image acquisition device then generates a composite image (the right diagram in FIG. 1 ) corresponding to the virtual image based on the virtual image and the image acquisition conditions (display conditions) under which the virtual image was acquired, and acquires the composite image. The acquired composite image is displayed on the screen of the user terminal.

[0019] To explain this image acquisition in more detail, a user who uses this image acquisition first launches an image acquisition application program (hereinafter referred to as the image acquisition app) that has been pre-installed on the user terminal. After the screen of the user terminal transitions to a screen for entering user information, the user enters their own user information (user ID, password, etc.). Once the user is authenticated, the screen of the user terminal transitions to a virtual space.

[0020] Through a user terminal with an image acquisition application running, the user operates the avatar in the virtual space in which the avatar is placed, determines the desired shooting location, and then prepares to shoot with the virtual camera. When the virtual camera starts, the screen of the user terminal transitions to a live view screen (left diagram in Figure 1) of the virtual camera. The user sets shooting conditions while viewing the live view screen. After setting various conditions, the user clicks, for example, a "shoot" icon (not shown) displayed on the screen of the user terminal. This causes the image acquisition device to acquire a virtual image. The acquired virtual image is displayed on the screen of the user terminal. The user may repeat shooting with the virtual camera while checking the displayed virtual image until the desired virtual image is obtained.

[0021] After obtaining the desired virtual image, the user clicks, for example, a "pseudo-realistic image" icon (not shown) displayed on the user screen. This causes the image acquisition device to generate and acquire a composite image corresponding to the virtual image based on the virtual image and image acquisition conditions, including the shooting conditions. The acquired composite image is displayed on the screen of the user terminal.

[0022] In the above example, the virtual image is displayed on the screen of the user terminal, but this is not limiting. For example, after capturing an image with the virtual camera, the step of displaying the virtual image may be skipped and the composite image may be directly displayed on the screen of the user terminal. In this case, the user may repeat capturing an image with the virtual camera while checking the displayed composite image until the desired composite image is obtained.

[0023] As explained above, this image acquisition method allows a synthetic image (pseudo-real image) corresponding to a virtual image to be acquired by taking a picture with a virtual camera. In other words, an image close to the real space can be acquired by taking a picture in a virtual space, and as a result, a more valuable experience can be provided to the user.

[0024] <<Configuration Example of an Image Acquisition System According to an Embodiment of the Present Invention>> The configuration of an image acquisition system (hereinafter referred to as "image acquisition system S") including an image acquisition device according to an embodiment of the present invention will be described with reference to Fig. 2. As shown in Fig. 2, the image acquisition system S is configured by an image acquisition device 10, multiple user terminals 100, and a database server 11. Note that the database server 11 may be configured as a part of the image acquisition device 10.

[0025] The image acquisition device 10 is composed of a computer, such as a personal computer (PC), a workstation, or a server computer. The image acquisition device 10 may be composed of a single computer or multiple computers distributed in parallel. Furthermore, if the computer constituting the image acquisition device 10 is a server computer, it may be a server computer for an ASP (Application Service Provider), SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service). In this case, when necessary information is input into the user terminal 100, the server computer performs various processes (calculations) based on the input information, and the calculation results are output on the user terminal 100. In other words, the functions of the server computer, which is the image acquisition device 10, can be used on the user terminal.

[0026] As shown in FIG. 2, the computer that constitutes the image acquisition device 10 includes a processor 21, a memory 22, a communication interface 23, and a storage device 24.

[0027] The processor 21 is configured by, for example, a central processing unit (CPU), a micro-processing unit (MPU), a microcontroller unit (MCU), a graphics processing unit (GPU), a digital signal processor (DSP), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), etc. The memory 22 is configured by, for example, semiconductor memory such as a read only memory (ROM) and a random access memory (RAM).

[0028] The communication interface 23 may be configured by, for example, a network interface card, a communication interface board, etc. The computer configuring the image acquisition device 10 can communicate with other devices connected to the Internet, a mobile communication line, etc. via the communication interface 23.

[0029] The storage device 24 may be configured, for example, by a flash memory, a hard disc drive (HDD), a solid state drive (SSD), a flexible disc (FD), a magneto-optical disc (MO disc), a compact disc (CD), a digital versatile disc (DVD), a secure digital card (SD card), or a universal serial bus memory (USB memory). The storage device 24 may be built into the computer main body that constitutes the image acquisition device 10, or may be attached to the computer main body in an external format.

[0030] The user terminal 100 is a computer used by a user, and specifically comprises a smart device such as a smartphone, a tablet terminal, or a notebook PC (Personal Computer). The user terminal 100 is equipped with a processor, memory, and a communication interface. The user terminal 100 stores an image acquisition application for using the image acquisition system S. When using the image acquisition system S, the user downloads the image acquisition application from a predetermined website and installs it on the user terminal 100.

[0031] <About the Database> The database server 11 is a cloud-based server provided by, for example, a company that operates a service using the image acquisition device 10, specifically, a company that provides the image acquisition service. The database server 11 stores various data necessary for the image acquisition, and the various data have a data structure. The image acquisition device 10 is connected to the database server 11 via the network N so that it can communicate with the database server 11, and can freely read out various data.

[0032] The various data that make up the data structure include data about the user, data about the virtual space, virtual object data about virtual objects, real subject data about real subjects, image acquisition conditions (display conditions), virtual images, and synthetic images.

[0033] Here, the term "virtual object" is broadly interpreted and is not limited to a specific tangible object, but includes people (avatars), objects, backgrounds, etc. that exist in a virtual space. Information about "people (avatars)" as virtual objects is stored in the database server 11 in association with data related to the user. "Objects" as virtual objects are objects other than people, and broadly include, for example, real estate (buildings, mountains, etc.) and movable property (automobiles, animals, etc.). "Backgrounds" as virtual objects include, for example, the sky in the case of outdoors, and the ceiling, walls, floor, etc. in the case of indoors. Furthermore, the term "real subject" is broadly interpreted and is not limited to a specific tangible object, but includes people (or human faces), objects, backgrounds, etc. that exist in real space. The real subjects referred to here correspond to virtual objects displayed in a virtual space, and each virtual object is assumed to have a corresponding real subject.

[0034] (Regarding user data) "User data" includes personal information such as a user's ID and password, and is associated with information about a person (avatar) as a virtual object. The user's ID and password are used, for example, when logging in to an image acquisition app. The information about the avatar is used by the user to identify their own avatar after the user is permitted to log in.

[0035] (Data Related to Virtual Space) "Data related to virtual space" includes, for example, information related to the virtual space in which virtual objects are placed, specifically information for specifying the size of the space, the internal environment, and the like.

[0036] (Virtual Object Data) "Virtual object data" is used in the process of displaying a virtual object in a virtual space, and is data for displaying the virtual object in the virtual space.

[0037] Virtual object data exists for each virtual object placed in the virtual space, and more specifically, exists according to the number of people (avatars), objects, and backgrounds that exist in the virtual space.

[0038] Furthermore, in this example, virtual object data exists for each feature of the virtual object. More specifically, for a person (avatar) as a virtual object, the features (display content) vary depending on the avatar's own posture and facial expression, as well as the time, weather, season, indoors / outdoors, display position, display direction, etc. in the virtual space. Therefore, for example, for one avatar, multiple sets of virtual object data exist depending on the features. As with people (avatars), multiple sets of virtual object data exist for objects and backgrounds as virtual objects depending on their respective features (display content).

[0039] The virtual object data is stored in a file for each virtual object, more specifically, for each feature of the virtual object, and each file is assigned an identification number (also referred to as an ID), a file name, etc. Similarly, the real subject data described below is also stored in a file for each real subject, more specifically, for each feature of the real subject, and each file is assigned an identification number, a file name, etc. The virtual object data and the real subject data are stored in separate files. Note that the case is not limited to where the virtual object data and the real subject data are stored in separate files. For example, real subject data acquired by executing the "process of acquiring real subject data corresponding to virtual object data" described below may be saved in a data file that stores virtual object data. In other words, the virtual object data and real subject data that correspond to each other may be saved in a form included in the same data file.

[0040] The database server 11 may also have a management database for managing the correspondence between the real subject data obtained by executing the "process of acquiring real subject data corresponding to virtual object data" and the virtual object data corresponding to the real subject data. FIG. 3 is a conceptual diagram showing a management database for managing the correspondence between virtual object data and real subject data. The management database shows the correspondence between each file in which virtual object data is saved and each file in which the corresponding real subject data is saved. More specifically, the management database may manage the file ID, virtual object content, and image file name of the virtual object image for the virtual object data, and the file ID, real subject content, and image file name of the real subject image for the real subject data.

[0041] The virtual object data includes information about a virtual object image and real subject data. A "virtual object image" includes a main image showing the gradation of each pixel and additional information. The main image includes a three-dimensional image that displays the three-dimensional shape of the virtual object, and is displayed in the virtual space as a two-dimensional image according to the display angle. The additional information includes positional information of the virtual object in the virtual space. The positional relationship of each virtual object in the virtual space is determined by the positional information of each virtual object.

[0042] The "information about real subject data" is information about real subject data corresponding to virtual object data, specifically information about the ID, file name, and path to the file in which the corresponding real subject data is stored. This allows the corresponding real subject data to be referenced from the virtual object data. That is, the virtual object data and real subject data acquired by executing the "process of acquiring real subject data corresponding to virtual object data" are associated and stored in the database server 11. Furthermore, the associated virtual object data and real subject data are associated with image acquisition conditions, which will be described later, and stored in the database server 11.

[0043] (Regarding Real Subject Data) "Real Subject Data" is used in the process of displaying a real subject corresponding to a virtual object, and is data for displaying a real subject corresponding to a virtual object. Real subject data exists for each real subject. Real subjects exist according to the number of virtual objects. Furthermore, like virtual object data, in this example, real subject data exists for each feature of the real subject. More specifically, for a person as a real subject, the features differ depending on the person's posture and facial expression, as well as the time, weather, time of year (season), whether indoors or outdoors, display position, display direction, etc. in the real space in which the real subject exists. Therefore, for example, for a specific person, multiple real subject data exist according to the features. Like people, multiple real subject data exist for objects and backgrounds as real subjects according to their respective features.

[0044] The real subject data includes information about the real subject image, the virtual object data, and information for obtaining the real subject image afterwards.

[0045] The "real subject image" includes a main image showing the gradation of each pixel and additional information. The additional information is position information of the real subject in real space and shooting conditions such as the shooting direction, and includes, for example, tag information in the Exchangeable Image File Format (Exif) format. The position information is acquired by a Global Positioning System (GPS) or the like. The position information of the real subject in real space corresponds to the position information of the corresponding virtual object in virtual space.

[0046] The real subject image includes an image (two-dimensional image) obtained by capturing a real subject corresponding to a virtual object in real space, or a three-dimensional image of the three-dimensional shape of the real subject corresponding to the virtual object, etc. The three-dimensional image is obtained, for example, by three-dimensionally scanning the three-dimensional shape of the real subject (e.g., a person).

[0047] The "information about virtual object data" is information about virtual object data corresponding to real subject data, and specifically, information about the ID, file name, and path to the file in which the virtual object data is stored. This allows the corresponding virtual object data to be referenced from the real subject data. That is, in the database server 11, the real subject data and virtual object data acquired by executing the "process of acquiring real subject data corresponding to virtual object data" are associated and stored. As described above, the associated virtual object data and real subject data are associated with image acquisition conditions, which will be described later, and stored in the database server 11.

[0048] "Information for obtaining a real subject image after the fact" is data that enables the real subject image to be obtained after the fact even if the real subject image is not included in the real subject data, such as the name of the real subject and location information of the real subject. The name of the real subject is, for example, a keyword that can identify the real subject by searching on the Internet. The location information of the real subject is coordinate information of the real subject in real space, specifically GPS information indicating longitude and latitude. Examples of means for obtaining a real subject image after the fact include obtaining (downloading) a real subject image found through an Internet search, or taking a photo using a drone.

[0049] (Regarding image acquisition conditions) "Image acquisition conditions" are display conditions for displaying a virtual space related to a virtual object, i.e., a virtual space in which a virtual object exists. In other words, image acquisition conditions are conditions for acquiring a virtual image that displays a virtual space related to a virtual object. The image acquisition conditions are used together with virtual object data in the "process of acquiring real subject data corresponding to virtual object data."

[0050] More specifically, the image acquisition conditions include shooting conditions for taking pictures with a virtual camera that imitates a real camera in the virtual space, environmental conditions related to the virtual environment that imitates a real environment in the virtual space, and conditions related to the background in the virtual space (also referred to as "background conditions"), etc. In addition, in the case of moving images, the image acquisition conditions include the movement path of the virtual object, etc.

[0051] "Photographing conditions" are settings for photographing with a virtual camera, and are photographing conditions set to mimic those of a real camera. Specific photographing conditions include the aperture of a virtual lens, the focal length of the lens, the shutter speed, the exposure amount, the ISO sensitivity, the white balance, the focus position, the photographing direction, and the angle of view. Photographing conditions are set by having an avatar operate a virtual camera in a virtual space. "Environmental conditions" include light source conditions related to a virtual light source that mimics a real light source. "Light source conditions" are conditions such as the position and orientation of a light source relative to a virtual object in a virtual space. For example, light sources in a virtual space include natural light such as sunlight and artificial light such as spotlights. "Background conditions" refer to conditions in the virtual space such as the time of day, weather, time of year (season), and whether indoors or outdoors.

[0052] (Regarding Virtual Images) As described above, a "virtual image" is an image that displays a part or the whole of a virtual space related to a virtual object. The virtual objects displayed in the virtual image include, for example, at least one of an avatar, an object, and a background that exist in the virtual space. The virtual image is stored in the database server 11 in association with the virtual object data of the virtual object displayed in the virtual image and the image acquisition conditions used when the virtual image was acquired (photographed).

[0053] (Regarding Composite Images) As described above, a "composite image" is a pseudo-real image corresponding to a virtual image. The real subjects displayed in the composite image include at least one of a person, an object, and a background as real subjects corresponding to virtual objects. The composite image is stored in the database server 11 in association with real subject data of the real subjects displayed in the composite image, virtual object data corresponding to the real subject data, and the image acquisition conditions used when the virtual image was acquired (photographed).

[0054] <<Functions of an Image Acquisition Device According to an Embodiment of the Present Invention>> Next, the configuration of an image acquisition device 10 according to an embodiment of the present invention will be described again from a functional perspective. As shown in FIG. 4 , the image acquisition device 10 has a user information accepting unit 31, a data transmitting / receiving unit 32, an image acquisition condition accepting unit 33, a virtual image generating unit 34, a virtual image transmitting unit 35, an instruction accepting unit 36, a real subject image acquiring unit 37, a real subject image synthesis unit 38, and a synthesized image transmitting unit 39. These functional units are realized by cooperation between hardware devices included in a computer constituting the image acquisition device 10 and a program (i.e., software) installed on the computer. Each functional unit will be described below.

[0055] The user information accepting unit 31 accepts user information (user ID, password, etc.) input from the screen of the user terminal 100. Thereafter, the user information accepting unit 31 executes user authentication to determine whether or not login is permitted.

[0056] The data transmitter / receiver 32 transmits data (more specifically, data related to the virtual space and virtual object data, etc.) to the user terminal 100 for which login is permitted, and displays on the screen of the user terminal 100 a state in which a virtual object is placed at a predetermined position in the virtual space. The virtual objects present in the virtual space include at least one of an avatar (person), an object, and a background. When a user operates the user terminal 100 to move a virtual object such as an avatar in the virtual space, the data transmitter / receiver 32 receives changes in the position information, updates the data according to the changed position information, and transmits the data to the user terminal 100.

[0057] The image acquisition condition receiving unit 33 receives image acquisition conditions. As described above, the image acquisition conditions are conditions for acquiring a virtual image, and include shooting conditions, environmental conditions, background conditions, and the like.

[0058] The virtual image generation unit 34 generates a virtual image based on the virtual object data, for example, by using a known image generation technique. In the example shown in FIG. 1 , the virtual image is the central diagram of FIG. 1 , and the virtual image includes virtual objects such as an avatar (person), a building (object), and a sky pattern (background). In this example, the virtual image generation unit 34 generates a virtual image based on virtual object images included in the virtual object data of each virtual object displayed on the live view screen (left diagram of FIG. 1 ). The virtual image generation unit 34 associates the virtual object data of the virtual objects displayed in the virtual image with image acquisition conditions for acquiring the virtual image, and stores the associated virtual image in the database server 11.

[0059] The virtual image sending unit 35 sends the generated virtual image to the user terminal 100. The sent virtual image is displayed on the user terminal 100. The instruction receiving unit 36 ​​receives an instruction from the user terminal 100 to generate a composite image corresponding to the virtual image.

[0060] When the real subject image acquisition unit 37 receives an instruction to generate a composite image, it acquires real subject data corresponding to the virtual object data based on the virtual object data and the image acquisition conditions. That is, the real subject image acquisition unit 37 executes a "process of acquiring real subject data corresponding to virtual object data."

[0061] The "process of acquiring real subject data corresponding to virtual object data" will be described in more detail with reference to FIG. 5 as an example. The left diagram in FIG. 5 is a virtual image, and multiple virtual objects (avatars, buildings, and sky patterns) exist in this virtual image. The real subject image acquisition unit 37 analyzes the virtual image and acquires virtual object data of virtual objects existing in the virtual image. That is, in the example shown in FIG. 5, virtual object data for each of the virtual objects, i.e., the avatar, the building, and the sky pattern, is acquired. As shown in the center diagram in FIG. 5, the real subject image acquisition unit 37 acquires real subject data (real subject images) for real subjects corresponding to each virtual object based on the virtual image, more specifically, on the virtual object data acquired from the virtual image and the image acquisition conditions used when the virtual image was acquired. That is, the real subject image acquisition unit 37 acquires real subject images for real subjects corresponding to the virtual objects based on the virtual image and the image acquisition conditions. The real subject image acquisition unit 37 stores the real subject data acquired by the "process of acquiring real subject data corresponding to virtual object data" in the database server 11 in association with the virtual object data and image acquisition conditions.

[0062] The real subject image synthesis unit 38 synthesizes the acquired multiple real subject images to generate a composite image corresponding to the virtual image. In the example shown in FIG. 5 , the real subject image synthesis unit 38 synthesizes the acquired multiple real subject images, specifically, the real subject images of people, objects, and backgrounds. The real subject image synthesis unit 38 associates various data used in the "process of acquiring real subject data" with the acquired composite image and stores the resulting image in the database server 11. More specifically, the real subject image synthesis unit 38 associates real subject data of the real subjects displayed in the composite image, virtual object data corresponding to the real subject data, and image acquisition conditions used when the virtual image was acquired with the acquired composite image and stores the resulting image in the database server 11.

[0063] Furthermore, when the shooting conditions as the image acquisition conditions are set to open the lens aperture, the real subject image synthesis unit 38 generates a composite image in which, for example, a person of the real subject closest to the virtual camera is in focus and buildings (objects) other than the person and the sky (background) farther from the virtual camera are out of focus, thereby creating a suitable blur, as shown in Fig. 6. When the shooting conditions as the image acquisition conditions are set to set the lens focal length to a telephoto distance, the real subject image synthesis unit 38 generates a composite image in which, for example, the angle of view is narrow and a compression effect is created, as shown in Fig. 7. When the shooting conditions as the image acquisition conditions are set to set the ISO sensitivity to high sensitivity, the real subject image synthesis unit 38 generates a composite image in which, for example, a night view is captured in a suitable manner.

[0064] The composite image transmission unit 39 transmits the generated composite image to the user terminal 100. The transmitted composite image is displayed on the user terminal 100.

[0065] <<Image Acquisition Flow According to One Embodiment of the Present Invention>> Next, the image acquisition flow, which is a data processing flow using the image acquisition device 10 described above, will be described. The image acquisition flow employs the image acquisition method (data acquisition method) of the present invention and proceeds according to the flow shown in FIG. 8. In other words, each step in the flow shown in FIG. 8 corresponds to each element constituting the image acquisition method of the present invention. Note that the flow shown in FIG. 8 is merely an example, and unnecessary steps may be deleted, new steps may be added, or the order of steps may be changed without departing from the spirit of the present invention.

[0066] This image acquisition flow is started when an image acquisition application installed on the user terminal 100 is started.

[0067] In this image acquisition flow, first, the user terminal 100 transmits user information (such as an identification ID and password) entered by the user to the image acquisition device 10 (S001). When the processor 21 of the image acquisition device 10 receives the user information (S002), it performs user authentication and determines whether or not the user is permitted to log in. The processor 21 then transmits data (specifically, data related to the virtual space and virtual object data) to the user terminal 100 for which login is permitted (S003). When the user terminal 100 receives the data, it displays a virtual space in which virtual objects exist on the screen of the user terminal 100 (S004). This allows the user to control their own avatar, etc., within the virtual space.

[0068] The user then operates the avatar in the virtual space, determines the desired shooting location, and prepares to shoot using the virtual camera. The user terminal 100 activates the virtual camera in the virtual space in accordance with the user's instructions and transitions the screen of the user terminal 100 to a live view screen. The user terminal 100 transmits the shooting conditions set by the user while viewing the live view screen to the image capture device 10 (S005).

[0069] The processor 21 of the image acquisition device 10 accepts the image acquisition conditions acquired by the user terminal 100 (S006) and generates a virtual image based on the virtual object data (S007). The processor 21 associates the generated virtual image, the virtual object data of the virtual objects displayed in the virtual image, and the image acquisition conditions used to acquire the virtual image, and stores them in the database server 11. The processor 21 transmits the generated virtual image to the user terminal 100 (S008).

[0070] The user terminal 100 displays the received virtual image on the screen of the user terminal 100 (S009). For example, when the "pseudo-realistic image" icon displayed on the screen of the user terminal 100 is clicked, the user terminal 100 instructs the image acquisition device 10 to generate a synthetic image (S010). When the processor 21 of the image acquisition device 10 receives the instruction to generate a synthetic image (S011), it acquires a real subject image of a real subject corresponding to the virtual object based on the virtual image and the image acquisition conditions (S013). The processor 21 associates real subject data including the acquired real subject image, virtual object data including the virtual object image, and the image acquisition conditions, and stores them in the database server 11.

[0071] The processor 21 generates a composite image corresponding to the virtual image by combining the acquired real subject images (S014). In the example shown in Fig. 5, the acquired real subject images, specifically, the real subject images of people, objects, and backgrounds, are combined. The processor 21 associates the generated composite image with real subject data of the real subjects displayed in the composite image, virtual object data corresponding to the real subject data, and image acquisition conditions for acquiring the virtual image, and stores the associated composite image in the database server 11.

[0072] The processor 21 transmits the generated composite image to the user terminal 100 (S016). The user terminal 100 displays the received composite image on the user terminal 100 (S015).

[0073] The image acquisition flow ends when the above series of processes are completed. Furthermore, since the image acquisition is performed arbitrarily in the virtual space, the image acquisition flow is repeated each time the image is acquired.

[0074] <<Effectiveness of One Embodiment of the Present Invention>> As described above, the image acquisition device 10 (data acquisition device 1) acquires real subject data corresponding to virtual object data based on virtual object data and image acquisition conditions (display conditions). That is, the image acquisition device 10 acquires a real subject image of a real subject corresponding to a virtual object based on a virtual image and image acquisition conditions. This makes it possible to acquire a composite image, i.e., a pseudo-real image, corresponding to the virtual image using the real subject image included in the acquired real subject data. As a result, by viewing the composite image, for example, a user can have the experience of visiting the location in real space and taking a photo. In this way, the image acquisition device 10 can acquire images that are close to the real space by capturing images in a virtual space, and the acquired images can provide the user with a more valuable experience.

[0075] Furthermore, since the processor 21 associates the acquired real subject data with the virtual object data, the real subject data can be easily acquired, thereby enabling an image close to the real space to be appropriately acquired through shooting in the virtual space.

[0076] The real subject data is data for displaying a real subject corresponding to a virtual object. Specifically, the real subject data is a two-dimensional image obtained by capturing an image of the real subject corresponding to the virtual object in real space, or a three-dimensional image obtained by scanning the three-dimensional shape of the real subject corresponding to the virtual object. This allows the image acquisition device 10 to acquire an image that is closer to the real space through capturing an image in virtual space.

[0077] The processor also generates a composite image corresponding to the virtual image by combining the acquired images of the real subject, thereby enabling an image that is closer to an image in real space and that displays the multiple real subjects to be appropriately acquired through shooting in the virtual space.

[0078] The virtual image also includes an avatar and a background as virtual objects, which allows a composite image to be appropriately acquired in which a person and background as real subjects corresponding to the virtual objects are displayed.

[0079] The image acquisition conditions include shooting conditions for taking an image with a virtual camera that imitates a real camera in the virtual space, environmental conditions related to the virtual environment that imitates a real environment in the virtual space, or conditions related to the background in the virtual space. In this way, by matching the image acquisition conditions when acquiring a virtual image with the acquisition conditions when acquiring a real subject image in the real space, it is possible to acquire an appropriate real subject image from the virtual image.

[0080] Other Embodiments In the above embodiment, the real subject image acquisition unit 37 acquires real subject data corresponding to the virtual object data based on the virtual object data and the image acquisition conditions. However, this is not limited to this, and the image acquisition device may also have a correction unit that corrects the acquired real subject image.

[0081] More specifically, the correction unit may correct the real subject image based on the image acquisition conditions. For example, in the example shown in FIG. 9 , sunlight is set to the left of the person as the real subject as a light source condition among the image acquisition conditions. Therefore, in the case of an image (photograph) acquired in real space, the left side of the person as the real subject is illuminated by light and the right side is in shadow, so that a shadow should be formed on the person as the real subject. However, if the real subject data acquired by the real subject image acquisition unit does not include real subject data that reflects the shadow due to the light source condition, the correction unit may correct the shadow of the real subject image included in the real subject data.

[0082] More specifically, the correction unit corrects the color (hue, saturation, brightness, etc.) of the real subject image of the person based on information about the position of sunlight in the virtual space. In the example shown in Figure 9, for example, the brightness of the area surrounded by the dashed line on the left side of the real subject image of the person is corrected to be brighter, and the brightness of the other parts is corrected to be darker. As a result, the image acquisition device can generate a composite image that corresponds to the direction of the sun in real space.

[0083] The correction unit may also correct the real subject image according to the state of the virtual object displayed in the virtual image. The state of the virtual object includes, for example, the posture and facial expression of the avatar. In the example shown in FIG. 10 , it is assumed that the posture of the avatar as a virtual object displayed in the virtual image is sideways. Meanwhile, the real subject data of a person acquired by the real subject image acquisition unit is a real subject image of a person facing forward. In this case, the correction unit may, for example, use a known image correction technique to correct the face of the person shown in the real subject image from facing forward to facing sideways.

[0084] Furthermore, when the correction range is wide, as described above, a real subject image of a person facing sideways may be acquired later based on the "information for later acquisition of a real subject image" stored in the real subject data. As a result, the image acquisition device can generate a more realistic composite image using the corrected or later acquired real subject image of the person.

[0085] The image acquisition device may also have a correction unit that receives instructions from the user terminal 100 and corrects image acquisition conditions, such as position information of a virtual object, as well as the time, season, past, and weather in the virtual space. More specifically, for example, suppose a user who has reviewed an acquired virtual image references the position of a virtual object in the virtual image and attempts to change the position of a real subject corresponding to the virtual object to generate a composite image. In this case, an instruction to change the image acquisition conditions is transmitted from the user terminal 100 to the image acquisition device. The correction unit changes the position of the real subject to the desired position in accordance with the instruction. As a result, the image acquisition device can generate a more realistic composite image that meets the user's needs based on the corrected image acquisition conditions.

[0086] The processors included in the data acquisition device and image acquisition device of the present invention include various types of processors. These include, for example, a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units. The various processors also include a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array). Furthermore, the various processors also include dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for performing specific processing.

[0087] Furthermore, a single processing unit included in the data acquisition device and image acquisition device of the present invention may be configured using one of the various processors described above, or may be configured using a combination of two or more processors of the same or different types, such as a combination of multiple FPGAs, or a combination of an FPGA and a CPU. Furthermore, a single functional unit included in the image acquisition device of the present invention may be configured using one of the various processors, or two or more of the multiple functional units may be combined into a single processor. Furthermore, as in the above-described embodiment, a single processor may be configured using a combination of one or more CPUs and software, and this processor may function as a single functional unit.

[0088] Furthermore, a processor may be used that realizes the functions of the entire system including the multiple functional units in the data acquisition device and image acquisition device of the present invention on a single IC (Integrated Circuit) chip, as typified by, for example, SoC (System on Chip), etc. Furthermore, the hardware configuration of the various processors described above may be an electric circuit (Circuitry) that combines circuit elements such as semiconductor elements.

[0089] REFERENCE SIGNS LIST 1 Data acquisition device 10 Image acquisition device 11 Database server 21 Processor 22 Memory 23 Communication interface 24 Storage device 31 User information acceptance unit 32 Data transmission / reception unit 33 Image acquisition condition acceptance unit 34 Virtual image generation unit 35 Virtual image transmission unit 36 ​​Instruction acceptance unit 37 Real subject image acquisition unit 38 Real subject image synthesis unit 39 Synthesized image transmission unit 100 User terminal N Network S Image acquisition system

Claims

1. 1. A data acquisition device comprising a processor, The processor: A data acquisition device that acquires real subject data corresponding to virtual object data based on virtual object data for displaying a virtual object in a virtual space and display conditions for displaying the virtual space related to the virtual object.

2. The data acquisition device according to claim 1 , wherein the processor associates the acquired real subject data with the virtual object data.

3. The data acquisition device according to claim 1 , wherein the real subject data is data for displaying a real subject corresponding to the virtual object.

4. The data acquisition device according to claim 3 , wherein the real subject data is an image obtained by photographing the real subject corresponding to the virtual object in real space.

5. The data acquisition device according to claim 3 , wherein the real subject data is a three-dimensional image of a three-dimensional shape of the real subject corresponding to the virtual object.

6. 1. An image acquisition device comprising a processor, The processor: accepting an image acquisition condition for acquiring a virtual image displaying a virtual space related to a virtual object; an image acquisition device that acquires a real subject image of a real subject corresponding to the virtual object based on the virtual image and the image acquisition conditions;

7. The image acquisition device according to claim 6 , wherein the processor generates a composite image corresponding to the virtual image by combining the acquired plurality of images of the real object.

8. The image acquisition device according to claim 6 , wherein the virtual image includes an avatar as the virtual object and a background.

9. The image acquisition device according to claim 6 , wherein the image acquisition conditions include imaging conditions for capturing an image in the virtual space with a virtual camera that imitates a real camera.

10. The image acquisition device according to claim 6 , wherein the image acquisition conditions include environmental conditions relating to a virtual environment that imitates a real environment in the virtual space.

11. The image acquisition device according to claim 6 , wherein the image acquisition conditions include a condition related to a background in the virtual space.

12. The image acquisition device according to claim 6 , wherein the processor corrects the image of the actual subject based on the image acquisition conditions.

13. The image acquisition device according to claim 6 , wherein the processor corrects the real subject image in accordance with a state of the virtual object displayed in the virtual image.

14. The image acquisition device according to claim 7 , wherein the processor stores the image acquisition conditions in association with the acquired composite image.

15. virtual object data used in processing to display a virtual object in a virtual space; a display condition that is used together with the virtual object data in a process of acquiring real subject data corresponding to the virtual object data, and that displays the virtual space related to the virtual object.

16. The data structure of claim 15 , further comprising the acquired real object data.

17. A data acquisition method in which a processor performs a process of acquiring real subject data corresponding to virtual object data based on virtual object data for displaying a virtual object in a virtual space and display conditions for displaying the virtual space related to the virtual object.

18. A process of receiving an image acquisition condition for acquiring a virtual image displaying a virtual space related to a virtual object; a process of acquiring a real subject image of a real subject corresponding to the virtual object based on the virtual image and the image acquisition conditions; The image acquisition method is performed by a processor.

19. A program for causing a computer to execute the processing included in the data acquisition method according to claim 17.

20. A program for causing a computer to execute each process included in the image acquisition method according to claim 18.

21. A computer-readable recording medium, A recording medium on which a program for causing a computer to execute the processing included in the data acquisition method according to claim 17 is recorded.

22. A computer-readable recording medium, A recording medium on which a program for causing a computer to execute each process included in the image acquisition method according to claim 18 is recorded.