Information processing device, information processing method and program

Abstract

To provide an information processing device capable of drawing a virtual space image on the basis of a two-dimensional image.SOLUTION: An image information device includes an acquisition part for acquiring two-dimensional image data obtained by capturing an imaging object or a landscape, a data creation part for creating data of a three-dimensional virtual space including the imaging object or the landscape on the basis of the two-dimensional image data, and a drawing part for drawing a virtual space image viewed from a virtual camera on the basis of the data of the three-dimensional virtual space. The data creation part assigns attributes to elements included in the two-dimensional image data, and reflects the attributes in the data of the three-dimensional virtual space, where the attributes represent whether to include the elements in the data of the three-dimensional virtual space.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses an image generation system that can make a display object corresponding to a target object appear in a virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-107252 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 does not specifically disclose a method for generating a virtual space, and in particular does not disclose a technique for rendering a virtual space image based on a two-dimensional image such as a photograph.

[0005] Therefore, in one aspect, the present invention has an object to provide an information processing device capable of rendering a virtual space image based on a two-dimensional image. [Means for solving the problem]

[0006] In one embodiment, An acquisition unit that acquires two-dimensional image data capturing an object or a scene to be imaged; a data creation unit that creates data of a three-dimensional virtual space including the captured object or scenery based on the two-dimensional image data; a rendering processing unit that renders a virtual space image viewed from a virtual camera based on the data of the three-dimensional virtual space, the data creation unit assigns attributes to elements included in the two-dimensional image data and reflects the attributes in the three-dimensional virtual space data; An information processing device is provided, wherein the attribute indicates whether or not the element is to be included in data of the three-dimensional virtual space. Effect of the Invention

[0007] In one aspect, according to the present invention, a virtual space image can be rendered based on a two-dimensional image. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating a configuration of an information processing system including an information processing apparatus according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram illustrating the position of a camera when photogrammetry is applied. [Figure 2A] FIG. 1 is a diagram illustrating the position of a camera when photogrammetry is applied. [Diagram 3] 1A to 1C are diagrams illustrating examples of screens of characters and the like rendered in a three-dimensional virtual space. [Figure 4] FIG. 13 is a diagram showing how the position of the virtual camera changes in accordance with the movement of the character. [Diagram 5] FIG. 1 is a diagram illustrating an example of a 360-degree panoramic photograph. [Figure 6] FIG. 13 is a diagram showing an example of the interior of a clothing store depicted as a virtual space. [Figure 6A] FIG. 13 is a diagram showing an example of the interior of a clothing store depicted as a virtual space. [Figure 6B] FIG. 13 is a diagram showing an example of the interior of a clothing store depicted as a virtual space. [Figure 7] FIG. 13 is a diagram showing an example of a city depicted as a virtual space. [Figure 7A] FIG. 13 is a diagram showing an example of a city depicted as a virtual space. [Figure 8] FIG. 13 is a diagram showing an example of generating a 3D model in which a human figure (person) is removed from a captured image. [Figure 8A] FIG. 13 is a diagram showing an example of generating a 3D model in which a human figure (person) is removed from a captured image. [Figure 8B] FIG. 13 is a diagram showing an example of generating a 3D model in which a human figure (person) is removed from a captured image. [Figure 8C] FIG. 13 is a diagram showing an example of generating a 3D model in which a human figure (person) is removed from a captured image. [Figure 9] FIG. 13 is a diagram showing an example of a representation of avatar footprints. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0010] (Example of configuration of information processing device) FIG. 1 is a diagram showing the configuration of an information processing system including an information processing apparatus according to this embodiment.

[0011] The information processing system of the present invention is characterized in that it renders a virtual space image seen from a virtual camera based on two-dimensional image data such as photographs, videos, etc., capturing captured objects or scenery. By rendering captured objects or scenery that actually exist as virtual space images, it becomes possible to use the virtual space images in various ways.

[0012] As shown in Fig. 1, the information processing device 10 includes an acquisition unit 11 that acquires two-dimensional image data capturing an object to be captured or a landscape, a data creation unit 12 that creates data of a three-dimensional virtual space including the object to be captured or a landscape based on the two-dimensional image data, a drawing processing unit 13 that draws a virtual space image viewed from a virtual camera based on the three-dimensional virtual space data, and an image data storage unit 14 that stores an image captured by a camera 20, image data being processed and handled by the acquisition unit 11, the data creation unit 12, and the drawing processing unit 13, etc. The photograph (two-dimensional image) or video captured by the camera 20 is stored in the image data storage unit 14 as needed. In addition, the image data being processed and handled by the acquisition unit 11, the data creation unit 12, and the drawing processing unit 13 are stored in the image data storage unit 14 as needed, and are used by the acquisition unit 11, the data creation unit 12, and the drawing processing unit 13 as needed.

[0013] The hardware configuration of the information processing device 10 is arbitrary. For example, the information processing device 10 can be configured by one or more mobile terminals and one or more computers, or can be configured by combining one or more mobile terminals with one or more computers. A part of the information processing device 10 may be configured by a mobile terminal, or one or more servers (computers) connectable to other computers. The image data storage unit 14 can be configured using a storage medium provided in a mobile terminal, a storage medium provided in a computer including a server, a memory card, or other external storage devices, etc.

[0014] The functions of the acquisition unit 11, the data creation unit 12, and the drawing processing unit 13 can be realized by, for example, application software (programs) installed in a mobile terminal or a computer.

[0015] Two-dimensional image data capturing an object or scenery can be created based on image data obtained by photographing the object or scenery with camera 20. Camera 20 may be any device capable of photographing still images or moving images, and the image data includes still image data such as photographs, and moving image data captured by a video camera serving as moving image camera 20. Acquisition unit 11 acquires the image data obtained by photographing with camera 20, and creates two-dimensional image data to be passed to data creation unit 12 by executing various processes described below.

[0016] Any object or scenery may be captured by the camera 20, and may include, for example, a park, a store, a tourist spot, a theme park, etc. By capturing images of the object or scenery in the three-dimensional virtual space from various angles, the reality of the three-dimensional virtual space can be improved. In particular, in order to reproduce the three-dimensional shape of an object in the three-dimensional virtual space by photogrammetry or other methods as described below, it is necessary to capture images of the same object from multiple different directions.

[0017] Any type of camera 20 may be used for shooting, including a camera included in a mobile terminal, a digital single-lens camera, a spherical camera (360-degree camera), etc. It is desirable to use a tripod, a spirit level, etc., to fix the camera 20 correctly and stably.

[0018] The acquisition unit 11 generates two-dimensional image data based on images (photos, videos, etc.) captured by the camera 20. The functions of the acquisition unit 11 can be implemented by application software installed in a mobile terminal or a computer (personal computer).

[0019] The acquisition unit 11 performs processing on the image obtained by shooting with the camera 20, which is necessary to generate image data to be handed over to the data creation unit 12. For example, in order to provide appropriate gradation to two-dimensional image data, shooting may be performed multiple times at the same shooting position and shooting direction while switching the exposure amount (shutter speed) in multiple stages, and multiple image data with different exposure amounts may be obtained. In this case, the function of the acquisition unit 11 allows the multiple image data to be synthesized into one image data (corresponding to one photograph), thereby obtaining gradation corresponding to a wide range of brightness, from dark areas such as in the shade to bright areas where direct sunlight hits.

[0020] The acquisition unit 11 may also execute a process for adjusting the color tone of the two-dimensional image data. In this case, it is desirable to use a color chart to obtain an appropriate color tone. For example, it is desirable to shoot in a state in which a color chart is captured for each shooting situation, such as the type of camera 20 used for shooting and the lighting light (for example, sunlight, incandescent light, etc.). This allows the function of the acquisition unit 11 to execute color correction so that the color of the color chart can be reproduced according to the type of camera 20 and the shooting situation, thereby improving color reproducibility. It is desirable to shoot the color chart in a state in which the entire color chart is exposed to uniform light, for example, direct sunlight.

[0021] A fisheye photograph obtained using a spherical camera (360-degree camera) can be used as two-dimensional image data by converting it into a so-called panoramic photograph. In this case, too, the acquisition unit 11 can be endowed with a function for converting a fisheye photograph into a panoramic photograph by implementing application software.

[0022] In addition, the acquisition unit 11 may appropriately perform correction processing using color temperature, color cast correction, and exposure amount on the images (photos, videos, etc.) captured by the camera 20 and the images at each stage during processing. Such processing is not limited to the case of optimizing the final two-dimensional image data, and may be performed in consideration of the convenience of the user's work.

[0023] The data creation unit 12 creates data of a three-dimensional virtual space including an imaged object or scenery based on the two-dimensional image data acquired by the acquisition unit 11.

[0024] For example, the data creation unit 12 executes a process of associating a panoramic photograph obtained using a spherical camera (360-degree camera) with the position of a virtual camera that is the same as the photographing position of the camera 20. That is, in this embodiment, the data of the three-dimensional virtual space is not limited to a so-called three-dimensional model created using photogrammetry, which will be described later. In the present disclosure, the "data of the three-dimensional virtual space" is used as a concept including substantially two-dimensional image data associated with the position of the camera 20. In this case, under the condition that the position of the virtual camera is matched with the position of the camera 20, the drawing processing unit 13 can draw a correct virtual space image. According to this method, it is possible to efficiently draw a virtual space image while suppressing the amount of image data.

[0025] In this method, in order to ensure the continuity of rendering when the position of the virtual camera is switched, it is necessary to maintain an appropriate interval between the shooting positions of the camera 20. If the interval between the shooting positions of the camera 20 is too wide, it may be difficult to recognize the continuity of the movement of the viewpoint when the position of the virtual camera is switched.

[0026] This technique does not require the use of a spherical camera (360-degree camera), and can be effectively used as long as the camera 20 is capable of capturing images at a relatively wide angle, for example.

[0027] Furthermore, the data creation unit 12 can create data of a three-dimensional virtual space including an imaged object or a landscape based on the two-dimensional image data acquired by the acquisition unit 11. Specifically, the data creation unit 12 creates a three-dimensional model of a subject from a plurality of two-dimensional image data having a common subject, for example, using photogrammetry. This allows a three-dimensional model showing, for example, the shape of the ground (topography) or the shape of a building to be obtained. Photogrammetry can be executed by implementing application software. For example, in the case where a landscape is created as a three-dimensional model, when an object or a moving object goes around to the back of a structure, the object or the moving object can be properly shielded by the structure.

[0028] Incidentally, in order to obtain a 3D model of an object from a 2D image of the object using photogrammetry, in principle, multiple images of the same object taken from different angles are required.

[0029] 2 and 2A are diagrams explaining the camera position when photogrammetry is applied.

[0030] In photogrammetry, for example, the same shape is recognized from successively taken photos, the position and attitude of the camera 20 at the time of taking the photos are calculated based on the amount of movement of the part with the same shape, and a three-dimensional model is created based on the two-dimensional image from the position of the camera 20. Therefore, two or more images obtained by taking the same shooting range from different positions of the camera 20, that is, two or more images with parallax, are required. As shown in FIG. 2, parallax is created by shifting the position of the camera 20 for each shot, so that it is possible to make a two-dimensional photo three-dimensional. In contrast, as shown in FIG. 2A, even if the shooting angle of the camera 20 is different for each shot, if the shots are taken from the same position, parallax is not created and a three-dimensional model cannot be created.

[0031] When applying photogrammetry to a wide area, multiple images taken in sequence from different camera positions are required under shooting conditions where the shooting ranges overlap. In this case, a 3D model can be constructed using photogrammetry by ensuring sufficient overlap in the shooting ranges between the previous and next images, or between specified images, without interruption. If the objects photographed in the previous and next images are significantly different, recognition of a common shape will fail, and photogrammetry processing will not be possible.

[0032] Generally, when creating a 3D model using photogrammetry, many photographs must be taken, but it is possible to build a 3D model of a wide area. Depending on the shooting conditions, it can also handle distant locations and high places, as well as fine details. Therefore, photogrammetry is suitable for places where you can see far into the distance, such as outdoors or in large venues (gymnasiums, concert halls, etc.), when you want to ensure a wide range of movement, or when you need a detailed 3D model. Photogrammetry is also suitable when you need a 3D model of the tall trees, such as when expressing sunlight filtering through the trees in a forest, or when you want a three-dimensional look of a distant view when changing viewpoints.

[0033] On the other hand, the disadvantages of photogrammetry are that it takes a long time to take pictures, the number of steps required to generate a 3D model tends to increase, and it is difficult to take pictures in dark places, which makes it easy to fail to generate a 3D model.Further disadvantages of photogrammetry include the difficulty of generating 3D models from objects with few feature points to recognize surfaces, such as plain white walls, and the need to adjust the size of the model because the generated 3D model does not reflect the actual scale.

[0034] Instead of photogrammetry, a three-dimensional model can be created using a mobile terminal with a camera that has a three-dimensional scanning application installed. In this case, instead of photogrammetry processing, the acquisition unit 11 generates a three-dimensional model of the captured object based on infrared scanning or the like. In this case, for example, a mobile terminal with a camera or a computer that acquires a captured image by a camera can be installed with a specific application software to perform processing for generating a three-dimensional model. By taking an image using infrared rays, it is possible to scan dark places and plain white walls that are difficult to handle with photogrammetry, and a three-dimensional model can be obtained for these objects as well. Other advantages of using a three-dimensional scanning application include that it does not take as much time to take a photograph as photogrammetry does, and that unlike photogrammetry, the size of the three-dimensional model matches the actual scale, so there is no need to adjust the size again. On the other hand, disadvantages of using a three-dimensional scanning application include that it is difficult to recognize the shape of an object in a dark area, that it is difficult to recognize the shape of a fine or thin object, and that it is not possible to photograph distant or high places.

[0035] The method of generating 3D models using a 3D scanning app is suitable for cases where you want to reduce the labor required for photography, as well as for generating terrain models for indoor locations, small parks, gardens, etc., where models of distant or high places are not required. The 3D model generated by the 3D scanning app can be used as a guide to match the 3D model generated by photogrammetry to the actual scale. In this case, even if the 3D scanning app is applied to only a portion of the subject being photographed, the 3D model can be used as an effective guide.

[0036] Furthermore, the data creation unit 12 can apply lighting to the three-dimensional model that stably reproduces the local lighting (illumination, light incidence) conditions, or can create shadows for the three-dimensional model.

[0037] Furthermore, the data creation unit 12 assigns an attribute to each element included in the two-dimensional image data, and creates data of the three-dimensional virtual space by reflecting the attribute assigned to each element. For example, the attribute may indicate whether or not the element is to be included in the data of the three-dimensional virtual space.

[0038] In this case, for example, when a person is included in the two-dimensional image data, the data creation unit 12 extracts the area of ​​the person by image recognition (data analysis) and deletes the area from the two-dimensional image data. Generally, in public places that anyone can use, such as airports, roads, and parks, it is difficult to obtain two-dimensional image data that does not include a person by shooting. However, according to this embodiment, since the data creation unit 12 can remove unnecessary elements such as a person, for example, data of a three-dimensional virtual space in which a moving object such as an avatar described later can move freely can be obtained. The elements to be excluded from the three-dimensional virtual space data are arbitrary and are not limited to people. For example, moving objects such as vehicles and information that should be excluded (for example, advertisements and personal information) can be mentioned.

[0039] It is impossible to reflect an area hidden behind an element deleted from two-dimensional image data in three-dimensional virtual space data based on that two-dimensional image data. However, in this case, that area can be supplemented (images can be embedded) with photos or videos (two-dimensional image data) taken from the same position at a different time. Alternatively, three-dimensional virtual space data for depicting that area can be created (images can be embedded) based on photos or videos (two-dimensional image data) taken from different angles at the same or different times. For example, if part of a building where the person overlaps is removed due to the deletion of a person, an image of that part of the building can be embedded.

[0040] Furthermore, the data creating unit 12 can include an attribute relating to a restriction on the movement range of a moving object in a three-dimensional virtual space as an attribute given to an element included in the two-dimensional image data.

[0041] For example, the data creation unit 12 assigns an attribute to an element included in the two-dimensional image data, which indicates that the element is an obstacle that cannot share the three-dimensional virtual space with the avatar, such as a wall through which the avatar cannot pass. In this case, the avatar is prohibited from passing through or overlapping with the obstacle in the three-dimensional virtual space.

[0042] In addition, the data creation unit 12 can assign attributes to elements contained in the two-dimensional image data, which attributes indicate whether a moving object is hidden (masked) behind an element in the three-dimensional virtual space, depending on the position of the moving object in the three-dimensional virtual space.

[0043] For example, examples of elements that are given an attribute indicating that the avatar is hidden behind include objects such as pillars and guardrails. In this case, when the object is in front of the avatar in the three-dimensional virtual space, the fact that the avatar is hidden behind the object and the position of the object (coordinates in the three-dimensional virtual space) are given to the object as attributes.

[0044] The drawing processing unit 13 draws a virtual space image viewed from the virtual camera based on the three-dimensional virtual space data created by the data creation unit 12.

[0045] The position of the virtual camera may include the position of the camera 20 capturing an image of an object or scenery corresponding to the two-dimensional image data acquired by the acquisition unit 11. When the position of the virtual camera coincides with the position of the camera 20, the rendering processing unit 13 can render a virtual space image corresponding to the image of the object or scenery captured by the camera 20. For example, the rendering processing unit 13 renders a virtual space image viewed from the virtual camera using a panoramic photograph based on shooting with a spherical camera (360-degree camera) or a relatively wide-angle photograph. In this case, by matching the position of the virtual camera with the position of the camera 20 such as a spherical camera (360-degree camera), the panoramic photograph can be used directly and effectively for the virtual space image.

[0046] Also, if a three-dimensional model is obtained, the position of the virtual camera can be set at a position that does not coincide with the camera 20 that captured the imaging object or scenery. In this case, the drawing processing unit 13 can calculate and draw a virtual space image based on the data of the three-dimensional virtual space created by the data creation unit 12. For example, the drawing processing unit 13 draws a virtual space image using a three-dimensional model obtained by photogrammetry or by other methods. In this case, the drawing processing unit 13 can create a virtual space image from a virtual camera at an arbitrary position based on the three-dimensional model. That is, the position of the virtual camera is not limited to the shooting position of the camera 20. The position and angle of the virtual camera can be controlled according to, for example, a user's operation, and the user can experience the virtual space viewed from the position of an arbitrary virtual camera.

[0047] Furthermore, the drawing processing unit 13 can place objects created as three-dimensional models and moving objects such as characters in a virtual space, or can move them in the virtual space.

[0048] The scenery of the virtual space image drawn by the drawing processing unit 13 is based on the three-dimensional model created by the data creation unit 12, and the positions of the objects and moving bodies are also managed by the same three-dimensional coordinates. Therefore, it is possible to grasp whether the object or moving body is in front of the scenery or behind the scenery based on the coordinates of the virtual camera, and this is reflected in the virtual space image. For example, in an image area where the object or moving body is located in front of the scenery, the scenery is masked by the image of the object or moving body. Conversely, in an image area where the scenery is in front of the object or moving body, the object or moving body is masked by the image of the scenery. When a moving body is located behind the scenery, whether it is masked by the scenery is determined by the above-mentioned attribute that is assigned to the object as an element included in the two-dimensional image data by the data creation unit 12.

[0049] Moreover, the drawing processing unit 13 can execute all or part of a series of processes of assigning attributes to each element included in the two-dimensional image data and reflecting the attributes in the virtual space image. For example, the series of processes may be executed by the data creation unit 12 as described above, or may be executed by the drawing processing unit 13. In the latter case, the drawing processing unit 13 extracts (assigns attributes to) objects to be removed (deleted), such as a person, by image recognition, and creates data of a three-dimensional virtual space from which the extracted objects have been removed. Furthermore, the drawing processing unit 13 executes a process of embedding predetermined drawing data in an area (removal area) after the object has been removed. Thereby, for example, when a part of a building where the person overlaps is removed due to the removal of a person, an image of the corresponding part of the building can be embedded. A part of the series of processes described above may be executed by the data creation unit 12, and the remaining processes may be executed by the drawing processing unit 13.

[0050] The virtual space image drawn by the drawing processing unit 13 can be used for various purposes.

[0051] For example, a user who accesses a virtual space constructed online can have his / her avatar (alter ego) corresponding to a moving object participate in the virtual space. Also, for example, by freely moving the avatar in the virtual space with a 360-degree field of view, it is possible to simulate the experience of a real park, a tourist spot, or the like. In this case, the drawing processing unit 13 can draw the avatar moving in the virtual space as a virtual space image by drawing the background (a captured object or scenery captured as two-dimensional image data) superimposed on the avatar. The drawing processing unit 13 can sequentially select the position and shooting direction of the virtual camera according to the position of the avatar so that the avatar is reflected in the virtual space image.

[0052] When multiple avatars participate in a common virtual space, users can interact with each other through the avatars participating in the virtual space. In this case, users can interact with each other in a way that is unique to users who are in a real place.

[0053] The virtual space can also be used for various economic activities. For example, it may be possible to purchase products in a virtual store. In this case, the user can not only purchase the desired product, but also have the experience of shopping in an actual store.

[0054] Virtual space can also be used for advertising. For example, products (including services) can be represented as lifelike avatars, allowing users to interact with the avatars that represent the products. This can give users the feeling that they have become friends with the avatars that represent the products through a shared experience, thereby increasing the effectiveness of advertising.

[0055] By constructing a town or facility at a travel destination as a virtual space, the user can have a simulated experience of traveling. By allowing the user to operate the position and direction of a virtual camera, the user can have a simulated experience of the scenery of the town or facility at the travel destination. Alternatively, a user disguised as an avatar can have a simulated experience of freely walking around the town or facility at the travel destination. Such a simulated experience can increase the user's interest in travel, for example, and therefore function effectively as an advertisement for a travel agency, etc.

[0056] Next, we will show an example of creating a park landscape using images captured by a spherical camera (360-degree camera). In this example, the user can operate a moving character, which is a 3D model, to experience strolling through a real park.

[0057] Fig. 3 is a diagram showing an example of a screen of a character drawn in a three-dimensional virtual space, and Fig. 4 is a diagram showing how the position of the virtual camera changes in accordance with the movement of the character. Fig. 4 shows five captured screens with different positions of the virtual camera. Fig. 5 is a diagram showing an example of a 360-degree panoramic photograph.

[0058] In this example, the park scenery is created by photographing the park with a spherical camera (360-degree camera), and a character 101 moving through the park is drawn superimposed on the scenery. The park scenery is projected onto a three-dimensional model of the ground, which will be described later, and the character 101 also moves on the three-dimensional model of the ground. The character 101 is operated by a game pad or the like, in the same way as a character in a game is operated.

[0059] First, the landscape is photographed at multiple camera positions using a spherical camera (360-degree camera) and converted into a panoramic photograph as shown in FIG. 5.

[0060] The position of the virtual camera in the three-dimensional virtual space during the game is fixed to the camera position of the omnidirectional camera (360-degree camera). That is, the position of the virtual camera is selected sequentially from among a plurality of camera positions of the omnidirectional camera (360-degree camera) at the time of shooting, and the scenery can be freely viewed in 360 degrees from each position of the selected virtual camera. When switching the viewpoint (virtual camera position), the scenery from the viewpoint is cross-faded, that is, the viewpoint can be moved smoothly by smoothly changing from the scenery before switching to the scenery after switching. Note that marks 102A and 102B in FIG. 3 and mark 102 in FIG. 4 indicate the position of the viewpoint projected on the ground.

[0061] Additionally, panoramic photographs of the landscape (Figure 5) can also be used as images for image-based lighting (IBL), which is used to recreate on-site lighting.

[0062] As shown in FIG. 3, lighting is applied to the character 101 in such a way as to stably reproduce the lighting conditions of the local area, so that the character 101 can blend in with the scenery. Image-based lighting (IBL) and directional light can be set as lighting, and lighting can be changed smoothly in accordance with the change of viewpoint. In addition, by applying a dedicated shadow map, it is possible to prevent unnecessary shadows from being added to the scenery. The shadow 101a of the character 101 shown in FIG. 3 corresponds to the shadow of sunlight, etc., and the shadow 101a can be projected onto the ground in accordance with the terrain based on a three-dimensional terrain model. Note that image-based lighting (IBL) is applicable not only to outdoor lighting but also to indoor lighting.

[0063] A 3D terrain model in a 3D virtual space can be created, for example, using a mobile device with a camera equipped with a 3D scanning app. This method has the advantage that it is easier to take photographs and reduces the workload required to create a 3D model compared to photogrammetry.

[0064] The generated three-dimensional terrain model is used as a target for projecting a landscape based on a panoramic photograph. The character 101 can walk or run along the undulating ground surface projected onto the terrain model.

[0065] Furthermore, a three-dimensional terrain model in a three-dimensional virtual space can be generated by photogrammetry, based on an image of the terrain captured by a camera other than a spherical camera (360-degree camera). When photogrammetry is applied, it takes time and effort to capture the image and process it into three dimensions, but it is suitable for cases where a precise three-dimensional model is desired.

[0066] In addition, by using a specific application, it is possible to generate a 3D terrain model using the panoramic photographs of the landscape. In this case, it is possible to restore the terrain based on the camera position. However, in order to obtain an accurate terrain model, it is necessary to increase the number of shooting points (camera positions), which increases the workload.

[0067] The generated three-dimensional terrain model can also be used, for example, as a collision model for producing the movement and footsteps of the character 101. A collision model may be prepared separately from the terrain model, and footsteps and the like may be added using a physical model. Also, by reflecting the area stepped on by the character 101 in the amount of deformation (depth of depressions) in the terrain model, it is possible to depict the footprints of the character 101, tracks on the snow, and vehicle tracks as a moving object.

[0068] In addition to the character 101, a three-dimensional model generated by photogrammetry can also be placed in the virtual space. For example, when placing the object 103 in FIG. 3 in the virtual space, a three-dimensional model of the object 103 is generated by photogrammetry or the like, and the installation position and installation direction in the virtual space are specified. If the object 103 is a moving body, the position and orientation of the object 103 may be controlled by the user. The lighting and shadow on the object 103 are reproduced by applying the same method as the lighting and shadow on the character 101. For example, in FIG. 3, in the lighting on the object 103, the shadow 101b of the character 101 is projected onto the object 103, and the shadow 103a of the object 103 is projected onto the ground.

[0069] 3, the character 101 is drawn so as to be partially masked by an object 104 located in front of the character 101. In this case, the above attribute (the property that the character 101 is hidden behind the object 104) is given to the object 104, which is an element included in the two-dimensional image data, and is reflected in the three-dimensional model.

[0070] 6 to 6B are diagrams showing an example of the interior of a clothing store rendered as a virtual space. In this example, a three-dimensional model is created based on a photograph of the interior of the store taken by camera 20 by the functions of acquisition unit 11 and data creation unit 12, and rendering processing unit 13 renders the interior of the store as a virtual space based on this three-dimensional model.

[0071] As shown in Figures 6 to 6B, avatar 201 (the user's alter ego), which is a moving object, can freely move around the store according to the user's operation. Also, as shown in Figure 6B, in addition to avatar 201, store clerks and other customers are also represented as moving objects 202a to 202c.

[0072] Furthermore, if a user finds a product that he or she wants to purchase, the user can purchase the product in the virtual space. For example, when the user selects product 203a or product 203b in Fig. 6A, display frames 204a and 204b corresponding to the respective products are rendered. The user can purchase the corresponding product by operating display frames 204a and 204b. The virtual space not only reproduces the interior of an actual store, but also reproduces the products placed in the store, allowing the user to experience a realistic store atmosphere.

[0073] In this way, the user can search for products while freely walking around the store as the avatar 201. By accessing the virtual space, the user can not only purchase products, but also easily experience shopping at places that the user cannot actually go to, such as stores far from the user's home.

[0074] 7 to 7A are diagrams showing an example of drawing a city as a virtual space. In this example, a three-dimensional model is created based on a photograph of the city taken by a camera by the functions of the acquisition unit 11 and the data creation unit 12, and the drawing processing unit 13 draws the city as a virtual space based on this three-dimensional model.

[0075] 7 to 7A, an avatar 301 (a user's alter ego) which is a moving object can move freely around the city according to the user's operation. In addition to the avatar 301, other passersby are also represented as moving objects 302 and the like.

[0076] In the example of FIG. 7 to FIG. 7A, a tag 305 is attached to a display element, such as a building or a facility, that appears on the screen. The tag 305 indicates information that describes the display element, and includes the name of the commercial facility that appears on the screen. For example, the tag 305 is automatically attached to a display element that appears in a photograph taken by the camera 20, i.e., a photograph acquired by the acquisition unit 11, by image recognition using artificial intelligence. The tag 305 is acquired by the acquisition unit 11 in association with the display element, and is stored in the image data storage unit 14 in association with the display element. When the display element is drawn, the drawing processing unit 13 acquires the tag 305 associated with the display element from the image data storage unit 14, and displays the tag 305 in accordance with the display position of the display element. In the example of FIG. 7 to FIG. 7A, the tag 305 moves to match the position of a predetermined location of the commercial facility on the screen, and when the predetermined location moves out of the screen, the display of the tag 305 is also controlled to be erased from the screen.

[0077] 7 to 7A show an example in which a city is depicted as a virtual space, but it is also possible to depict a wide range of real-world spots, such as tourist sites, various facilities, theme parks, etc. Users can easily experience walking around real-world spots freely by simply accessing the virtual space.

[0078] 8 to 8C are diagrams showing an example of generating a 3D model in which a human figure (person) is deleted from a captured video.

[0079] Figure 8 shows the original image that was captured. In this example, a tourist spot was captured, and many human figures are visible in the image. The human figures, which are elements of the two-dimensional image data corresponding to this image, are given an attribute indicating that they are elements to be excluded from the three-dimensional virtual space data.

[0080] In this case, as shown in Fig. 8A, the data creation unit 12 extracts a region 401 of a human figure from the image shown in Fig. 8 by image recognition (data analysis). Then, the data creation unit 12 generates a 3D model corresponding to the image in Fig. 8B by deleting the region 401 from the image (Fig. 8) based on the extraction result of the region 401.

[0081] As shown in Fig. 8C, within this 3D model, a character 402 is movable. For example, the character 402 moves freely as a user's avatar in accordance with an operation of a user viewing the image of Fig. 8C drawn by the drawing processing unit 13. In this way, according to this embodiment, a 3D model can be generated in which unnecessary figures and the like are removed from the image, so that it is possible to provide the user with a virtual space in which the user's avatar, such as the character 402, can be freely moved.

[0082] Fig. 9 is a diagram showing an example of the representation of the footprints of an avatar. In the example shown in Fig. 9, footprints 502 of an avatar 501 are rendered on a sandy beach rendered by the rendering processing unit 13. As described above, the footprints of the avatar 501 are rendered by reflecting the area stepped on by the avatar 501 in the amount of deformation (depth of depression) in the terrain model.

[0083] In the above embodiment, an example of creating a virtual space based on real landscapes, etc., is shown, but for example, by replacing the setting of a so-called CG movie, CG animation, or CG game created using 3D CG with a virtual space, it is also possible to experience the world of that movie, animation, or game.

[0084] In addition, the technology of the present invention that creates virtual spaces can be simplified so that the general public can use it, and a scheme can be established that allows photographic data to be uploaded as virtual space data. This allows many people to easily use the technology of the present invention, which can contribute to activities that lead to regional revitalization, such as students living in rural areas creating metaverse spaces for the scenery of their hometowns.

[0085] As described above, according to the information processing device 10 of this embodiment, a virtual space image can be rendered based on a two-dimensional image. Therefore, a virtual space can be easily experienced. For example, based on a photo of an actual place, a user can be given the experience of visiting the place. In addition, the virtual space image rendered by the rendering processing unit 13 can be used for various purposes, and is suitable for use in economic activities and advertisements as well as participation in a virtual space via an avatar (alter ego) or character.

[0086] Furthermore, according to the information processing device 10 of this embodiment, actual photographs can be used to create a virtual space. In this case, compared to creating a virtual space as so-called high-quality three-dimensional CG data equivalent to a photograph, it is possible to significantly reduce the work time and work costs.

[0087] Although each embodiment has been described above in detail, the present invention is not limited to the specific embodiment, and various modifications and changes are possible within the scope of the claims. In addition, it is also possible to combine all or a plurality of the components of the above-described embodiments. The following supplementary notes are further provided with respect to the above embodiment. [Appendix 1] An acquisition unit that acquires two-dimensional image data capturing an object or a scene to be imaged; a data creation unit that creates data of a three-dimensional virtual space including the captured object or scenery based on the two-dimensional image data; a rendering processing unit that renders a virtual space image viewed from a virtual camera based on the data of the three-dimensional virtual space, the data creation unit assigns attributes to elements included in the two-dimensional image data and reflects the attributes in the three-dimensional virtual space data; The attribute indicates whether or not the element is to be included in the data of the three-dimensional virtual space. [Appendix 2] 2. The information processing device according to claim 1, wherein the three-dimensional virtual space is a space in which an avatar drawn by the drawing processing unit can move. [Appendix 3] 3. The information processing device according to claim 2, wherein the avatar is movable within the three-dimensional virtual space in accordance with operations by a user viewing the virtual space image drawn by the drawing processing unit. [Appendix 4] The information processing device according to claim 2 or 3, wherein the attribute relates to a restriction on a movement range of the avatar in the three-dimensional virtual space. [Appendix 5] The information processing device of claim 2 or 3, wherein the attribute indicates whether the avatar is hidden behind the element in the three-dimensional virtual space, depending on a position of the avatar in the three-dimensional virtual space. [Appendix 6] the data creation unit generates the data of the three-dimensional virtual space as image data associated with a position corresponding to a camera that captured the imaged object or scenery; The information processing device according to claim 1, wherein the rendering processing unit renders a virtual space image viewed from a virtual camera placed at the position based on the data of the three-dimensional virtual space. [Appendix 7] The information processing device according to claim 1, wherein the drawing processing unit controls a position or an angle of the virtual camera based on a user's operation. [Appendix 8] The data creation unit creates a three-dimensional model of an object or a moving object, 2. The information processing device according to claim 1, wherein the drawing processing unit draws the three-dimensional model created by the data creation unit by superimposing it on the virtual space image. [Appendix 9] The information processing device of claim 5, wherein the drawing processing unit switches the position of the virtual camera depending on the position of the avatar in the virtual space. [Appendix 10] The information processing device according to claim 1, wherein the drawing processing unit displays a tag associated with a display element included in the two-dimensional image data in association with the display element in the virtual space image. [Appendix 11] 2. The information processing device according to claim 1, wherein the image capturing object or scenery is a stationary image capturing object or scenery. [Appendix 12] An acquisition step of acquiring two-dimensional image data capturing an object or a scene to be imaged; a data creation step of creating data of a three-dimensional virtual space including the captured object or scenery based on the two-dimensional image data; a rendering processing step of rendering a virtual space image viewed from a virtual camera based on the data of the three-dimensional virtual space, In the data creation step, attributes are assigned to elements included in the two-dimensional image data, and the attributes are reflected in the three-dimensional virtual space data; An information processing method, wherein the attribute indicates whether or not the element is to be included in the data of the three-dimensional virtual space. [Appendix 13] An acquisition step of acquiring two-dimensional image data capturing an object or a scene to be imaged; a data creation step of creating data of a three-dimensional virtual space including the captured object or scenery based on the two-dimensional image data; a rendering processing step of rendering a virtual space image viewed from a virtual camera based on the data of the three-dimensional virtual space; In the data creation step, attributes are assigned to elements included in the two-dimensional image data, and the attributes are reflected in the three-dimensional virtual space data; The attribute indicates whether or not the element is to be included in the data of the three-dimensional virtual space. [Appendix 14] An acquisition unit that acquires two-dimensional image data capturing an object or a scene to be imaged; a data creation unit that creates data of a three-dimensional virtual space including the captured object or scenery based on the two-dimensional image data; a rendering processing unit that renders a virtual space image viewed from a virtual camera based on the data of the three-dimensional virtual space, The rendering processing unit is an information processing device that creates data of the three-dimensional virtual space by removing predetermined objects that may be included in the two-dimensional image data. [Appendix 15] The information processing device according to claim 14, wherein the drawing processing unit extracts a predetermined object that may be included in the two-dimensional image data by image recognition. [Appendix 16] The information processing device according to claim 14, wherein the drawing processing unit removes a predetermined object that may be included in the two-dimensional image data, and then embeds drawing data in the removal area. [Explanation of symbols]

[0088] 10. Information processing device 11 Acquisition Department 12 Data Creation Department 13 Drawing processing section 14 Image data storage section 20 Camera

Claims

1. an acquisition unit that acquires two-dimensional image data capturing an object or a scene to be imaged; a data creation unit that creates data of a three-dimensional virtual space including the captured object or scenery based on the two-dimensional image data; a rendering processing unit that renders a virtual space image viewed from a virtual camera based on the data of the three-dimensional virtual space, The information processing device, wherein the three-dimensional virtual space is a space in which an avatar drawn by the drawing processing unit can move.

2. An information processing device as described in claim 1, wherein the avatar is movable within the three-dimensional virtual space in accordance with operations by a user viewing the virtual space image drawn by the drawing processing unit.

3. The data creation unit assigns attributes to elements included in the two-dimensional image data, and reflects the attributes in the data of the three-dimensional virtual space, The information processing apparatus according to claim 1 , wherein the attribute indicates whether or not the element is to be included in the data of the three-dimensional virtual space.

4. An information processing device as described in Claim 3, wherein the attribute relates to restricting the range of movement of the avatar in the three-dimensional virtual space.

5. An information processing device as described in Claim 3, wherein the attribute indicates whether the avatar is hidden behind the element in the three-dimensional virtual space depending on the position of the avatar in the three-dimensional virtual space.

6. The data creation unit generates the data of the three-dimensional virtual space as image data associated with a position corresponding to a camera that captured the imaged object or scenery, The information processing device according to claim 1 , wherein the rendering processing unit renders a virtual space image viewed from a virtual camera placed at the position based on the data of the three-dimensional virtual space.

7. An information processing device as described in claim 1 or 2, wherein the drawing processing unit controls the position or angle of the virtual camera based on user operation.

8. The data creation unit creates a three-dimensional model of an object or a moving object, The information processing apparatus according to claim 1 , wherein the rendering processing unit renders the three-dimensional model created by the data creating unit so as to be superimposed on the virtual space image.

9. An information processing device as described in Claim 5, wherein the drawing processing unit switches the position of the virtual camera depending on the position of the avatar in the virtual space.

10. An information processing device as described in claim 1 or 2, wherein the drawing processing unit displays tags associated with display elements contained in the two-dimensional image data in association with the display elements in the virtual space image.

11. An information processing device as described in claim 1 or 2, wherein the imaged object or scenery is a stationary imaged object or scenery.

12. An information processing device as described in claim 1 or 2, wherein the drawing processing unit creates data of the three-dimensional virtual space from which specified objects that may be included in the two-dimensional image data have been removed.

13. The information processing device described in Claim 12, wherein the drawing processing unit extracts a specified object that may be included in the two-dimensional image data by image recognition.

14. The information processing device described in Claim 12, wherein the drawing processing unit removes a specified object that may be included in the two-dimensional image data, and then embeds drawing data in the removed area.

15. An acquisition step of acquiring two-dimensional image data capturing an object or a scene to be imaged; a data creation step of creating data of a three-dimensional virtual space including the captured object or scenery based on the two-dimensional image data; a rendering processing step of rendering a virtual space image viewed from a virtual camera based on the data of the three-dimensional virtual space, An information processing method, wherein the three-dimensional virtual space is a space in which an avatar drawn in the drawing processing step can move.

16. An acquisition step of acquiring two-dimensional image data capturing an object or a scene to be imaged; a data creation step of creating data of a three-dimensional virtual space including the captured object or scenery based on the two-dimensional image data; a rendering processing step of rendering a virtual space image viewed from a virtual camera based on the data of the three-dimensional virtual space, The three-dimensional virtual space is a space in which an avatar drawn in the drawing processing step can move.

Images