Information processing device, information processing program, and information processing method

Abstract

To provide an information processing device, information processing program, and information processing method that enable easy utilization of a virtual space high in reproductivity of an actual space.SOLUTION: Objects generated based on detection of three-dimensional objects Tg1 and Tg2 from a picture Pt obtained by photographing an actual space are shield objects Oj1 and Oj2. An information processing device is configured to implement: reading data on a virtual space Vs where the shield objects Oj1 and Oj2 are arranged; setting positions of an avatar and a view point in the virtual space Vs; and plotting out the virtual space Vs viewing from the view points.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing program, and an information processing method that execute processes related to virtual space. [Background technology]

[0002] In recent years, various uses of virtual spaces have been proposed. One example of such uses is an attempt to realize user activities in virtual spaces similar to those in real spaces. For example, Patent Document 1 describes a system that recreates a housing exhibition center in a virtual space, allowing users to view and purchase homes in the virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-8638 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to effectively realize user activities in a virtual space similar to those in the real world, it is desirable for the virtual space to have a landscape that is as close as possible to the real world. In other words, it is desirable for the placement, shape, appearance, etc. of three-dimensional objects in the virtual space to be highly reproducible to the real world. Generally, three-dimensional computer graphics (3DCG) is widely used to generate three-dimensional objects in virtual spaces, but building a highly reproducible virtual space through the processes of 3DCG modeling and material and texture setting requires a significant amount of work. This inevitably leads to increased production costs and time. Therefore, there is a need for technology that can easily build a virtual space that is highly reproducible to the real world. [Means for solving the problem]

[0005] Various aspects of an information processing device, an information processing program, and an information processing method for solving the above problems will be described. [Mode 1] An information processing device that reads data of a virtual space in which an occluding object is placed, where the object generated based on the detection of a three-dimensional object from a photograph taken of real space is an occluding object, sets the position of an avatar and a viewpoint in the virtual space, and draws the virtual space as seen from the viewpoint.

[0006] According to the above configuration, a virtual space is constructed by generating occluding objects from a photograph, and an avatar is placed in this virtual space. Therefore, a virtual space that highly reproduces the real world can be easily constructed without increasing production costs or time, and activities in the virtual space that are similar to those in the real world can be accurately realized.

[0007] [Aspect 2] The information processing device is connected to an information relay device via a network, and the avatars include a self-avatar, which is the avatar of the user operating the information processing device, and other-user avatars, which are avatars of other users.The information processing device transmits location information of the self-avatar, which changes in response to operation of the information processing device, to the information relay device, and receives location information of the other-user avatars from the information relay device, and renders the virtual space including each avatar reflecting the location information of the self-avatar and the location information of the other-user avatars.This is the information processing device described in [Aspect 1].

[0008] According to the above configuration, by exchanging location information through the information relay device, it is possible to reduce the amount of data to be transmitted and to reflect the movements of other users' avatars in the depiction of the virtual space.

[0009] [Aspect 3] The information processing device according to [Aspect 1] or [Aspect 2], wherein the information processing device combines a background generated from the photograph with an image generated by drawing the virtual space. According to the above configuration, it is possible to generate an image of a virtual space scene that is in line with the scene in real space, even in terms of depiction in the depth direction.

[0010] [Aspect 4] The information processing device according to any one of [Aspect 1] to [Aspect 3], wherein the information processing device limits the movable range of the viewpoint position, which is moved in response to user operation, to a predetermined range.

[0011] According to the above configuration, even if the area in which real space is reproduced in the virtual space or its background is limited, the structure and background within the virtual space are prevented from being disrupted in the drawn image of the virtual space.

[0012] [Aspect 5] An information processing device according to any one of [Aspect 1] to [Aspect 4], wherein the data of the virtual space includes data of a first occluding object generated from a first photograph, data of a second occluding object generated from a second photograph, and data of an intermediate object which is an object generated by averaging configuration information of the first occluding object and configuration information of the second occluding object, and a condition is set for switching the occluding object to be drawn from the first occluding object to the second occluding object for the position of the viewpoint, and when the condition is met, the information processing device switches the occluding object to be drawn from the first occluding object to the second occluding object via the intermediate object, and draws the virtual space.

[0013] With the above configuration, by switching the object to be rendered, it is possible to expand the range of movement of the viewpoint and avatar, and the reproduction range in the virtual space can be expanded as a whole using the data of multiple occluding objects. Furthermore, by switching the occluding object to be rendered via an intermediate object, natural transitions with a three-dimensional feel can be achieved in the scenery of the virtual space.

[0014] [Aspect 6] An information processing program that causes one or more computers to execute the following steps: reading data on a virtual space in which an occluding object is placed, where the object generated based on the detection of a three-dimensional object from a photograph taken of real space is an occluding object; setting the position of an avatar and a viewpoint in the virtual space; and drawing the virtual space as seen from the viewpoint.

[0015] According to the above configuration, a virtual space is constructed by generating occluding objects from a photograph, and an avatar is placed in this virtual space. Therefore, a virtual space that highly reproduces the real world can be easily constructed without increasing production costs or time, and activities in the virtual space that are similar to those in the real world can be accurately realized.

[0016] [Aspect 7] An information processing method in which one or more computers read data of a virtual space in which an occluding object is placed, where the object generated based on the detection of a three-dimensional object from a photograph taken of real space is an occluding object; set the position of an avatar and a viewpoint in the virtual space; and draw the virtual space as seen from the viewpoint.

[0017] According to the above method, a virtual space is constructed by generating occluding objects from a photograph, and an avatar is placed in this virtual space. Therefore, a virtual space that highly reproduces the real world can be easily constructed without increasing production costs or time, and activities in the virtual space that are similar to those in the real world can be accurately realized. [Effects of the Invention]

[0018] According to the present disclosure, a virtual space that has a high degree of reproducibility of real space can be easily utilized. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an information processing system according to an embodiment. [Figure 2]FIG. 2 is a diagram showing the configuration of a virtual space generating device according to an embodiment. [Figure 3] FIG. 3 is a diagram illustrating a configuration of an information relay device according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating a configuration of a user terminal according to an embodiment. [Figure 5] FIG. 5 is a diagram showing a processing procedure for generating a virtual space according to an embodiment. [Figure 6] FIG. 6 is a diagram illustrating the generation of a virtual space from a photograph in one embodiment. [Figure 7] FIG. 7 is a diagram showing a virtual space in which an avatar and a viewpoint are placed in one embodiment. [Figure 8] FIG. 8 is a diagram showing a processing procedure for information exchange between user terminals in one embodiment. [Figure 9] FIG. 9 is a diagram showing a processing procedure for rendering a virtual space in one embodiment. [Figure 10] FIG. 10 is a diagram illustrating information held by a user terminal in one embodiment. [Figure 11] FIG. 11 is a diagram schematically showing data for switching the drawing target in one embodiment. [Figure 12] FIG. 12 is a diagram showing a procedure for switching the drawing target in one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] An embodiment of an information processing device, an information processing program, and an information processing method will be described with reference to the drawings. [Overall configuration of information processing system] The overall configuration of an information processing system including an information processing device will be described with reference to Fig. 1. As shown in Fig. 1, the information processing system 100 includes a management system 10 and a plurality of user terminals 40. The management system 10 includes a virtual space generating device 20 and an information relay device 30. The user terminal 40 is an example of an information processing device. The management system 10 and the user terminals 40 are each connected to a network such as the Internet.

[0021] The virtual space generation device 20 generates a virtual space from a photograph of a real space. The virtual space generation device 20 is a computer device such as a personal computer or a server device. The user terminal 40 is a terminal operated by a user who acts as an avatar in a virtual space. The user terminal 40 sets the position of the avatar and viewpoint within the virtual space in response to the user's operation, and renders the virtual space. The user terminal 40 is a computer device such as a smartphone, tablet terminal, or personal computer.

[0022] The information relay device 30 transmits and receives data to and from each user terminal 40 via the network. The information relay device 30 relays information such as the position of an avatar required for rendering a virtual space on the user terminal 40 between multiple user terminals 40. The information relay device 30 is a computer device such as a personal computer or a server device.

[0023] The use of the virtual space provided by the information processing system 100 of this embodiment is not particularly limited as long as it can be used for activities that can be performed in real space. For example, the virtual space can be used for sightseeing, various tours of factories, cultural activities such as appreciating artworks, and the display and sale of real estate and various products.

[0024] [Configuration of each device] The configurations of the virtual space generating device 20, the information relay device 30, and the user terminal 40 will be described in detail with reference to FIGS.

[0025] As shown in FIG. 2, the virtual space generating device 20 includes a communication unit 21, a control unit 22, and a storage unit . The communication unit 21 performs communication processing between the virtual space generating device 20 and other devices, such as connecting to a network and sending and receiving data.

[0026] The control unit 22 executes a program stored in the storage unit 23 to function as a target space generation unit 22a. The target space generation unit 22a analyzes a photograph of the real space, detects one or more three-dimensional objects contained in the photograph, and generates an occluding object, which is a 3D model of the three-dimensional object. The detected three-dimensional object is, for example, a structure such as a pillar, fixture, or ornament, and is an occluding object that has the function of blocking part of the space. The generated occluding object has a texture based on the color information of the three-dimensional object extracted from the photograph. Known photogrammetry techniques may be used to generate a 3D model from the photograph.

[0027] The target space generation unit 22a places occluding objects in a virtual three-dimensional space, thereby generating a target space, which is a virtual space that is the target of drawing processing in the user terminal 40. When multiple occluding objects are generated from a photograph, the multiple occluding objects are placed in the target space according to the positional relationships of multiple three-dimensional objects in the real space shown by the photograph. The storage unit 23 stores various programs and data required for the control unit 22 to execute processes.

[0028] As shown in FIG. 3, the information relay device 30 includes a communication unit 31, a control unit 32, and a storage unit 33. The communication unit 31 performs communication processing between the information relay device 30 and other devices, such as connection to a network and sending and receiving data.

[0029] The control unit 32 executes a program stored in the storage unit 33 to function as a relay processing unit 32a.

[0030] The relay processing unit 32a controls the exchange of information between the user terminals 40. Specifically, when the relay processing unit 32a receives relay information from one user terminal 40, which is information to be reflected in other user terminals 40, the relay processing unit 32a identifies the destination user terminal 40 and transmits the relay information to the identified user terminal 40 via the communication unit 31. The relay information includes the location information of the avatar.

[0031] The storage unit 33 stores various programs and data required for the execution of processes by the control unit 32. Such data includes data for managing users involved in relaying information.

[0032] As shown in FIG. 4, the user terminal 40 includes a communication unit 41, a control unit 42, a storage unit 43, and an input / output unit 44. The communication unit 41 performs communication processing between the user terminal 40 and other devices, such as connecting to a network and sending and receiving data.

[0033] The control unit 42 executes a program stored in the storage unit 43 to function as an information management unit 42a and a drawing processing unit 42b. The information management unit 42a manages the positions of occluding objects, avatars, and viewpoints in the target space, and instructs the rendering of the target space.

[0034] The drawing processing unit 42b performs drawing processing in response to instructions from the information management unit 42a. That is, the drawing processing unit 42b determines a drawing range in the target space based on the position information of the occluding object, the avatar, and the viewpoint, and performs rendering to generate display data, which is data of a two-dimensional image.

[0035] The storage unit 43 stores various programs and data necessary for the control unit 42 to execute processing. The storage unit 43 stores, as an example of such data, drawing data 43a. The drawing data 43a includes data required for drawing the target space. Specifically, the drawing data 43a includes target space data, which is data on the target space including occluding objects generated by the virtual space generating device 20, avatar data, which is data on the model and appearance of an avatar, and background data, which is data on an image used as a background when drawing the target space.

[0036] The input / output unit 44 includes an operation unit that receives user operations and outputs signals corresponding to the operations to the control unit, and a display unit that displays images based on data from the control unit. The operation unit is a touch panel, a mouse, or a keyboard, and the display unit is a panel or display such as a liquid crystal display.

[0037] The physical configuration, i.e., hardware configuration, of each device, i.e., the virtual space generation device 20, information relay device 30, and user terminal 40, which have the above functions, will be described below. Each device includes electronic circuits, such as a CPU, MPU, or GPU, memory, such as ROM, RAM, registered memory, or unbuffered memory, and storage, such as an SSD or HDD. The computing device loads the operating system and various programs from the storage into memory and executes instructions retrieved from the memory. Each device may also include an integrated circuit, such as an ASIC or FPGA.

[0038] Each device also has a communication interface for transmitting and receiving data to and from a connected device via a network. The communication interface is implemented as hardware, software, or a combination of these.

[0039] The processes performed by each device may be executed by software included in the device, or may be realized by a combination of software and an integrated circuit included in the device.

[0040] [Generate target space] The processing steps for generating the target space will be described with reference to FIGS. As shown in Fig. 5, first, virtual space generation device 20 reads photograph data, which is data of a photograph taken of the real space to be reproduced (step S10). Next, virtual space generation device 20 analyzes the photograph data and extracts three-dimensional objects, thereby generating data of occluding objects (step S11). Then, virtual space generation device 20 generates target space data by placing the occluding objects in the virtual three-dimensional space, that is, by setting the positions of the occluding objects in the three-dimensional space (step S12). The processing of steps S11 and S12 is processing performed by target space generation unit 22a.

[0041] A single target space may be generated from multiple photographs of the real space taken from different positions, or from a single photograph of the real space taken from a single position. A single target space may also be generated from a photograph covering a wide range, such as a panoramic photograph or a 360-degree photograph. When the photographic position or range of the base photograph is limited, the occluding object does not need to reproduce the shape and color of a three-dimensional object in real space around the entire periphery of the object. It is sufficient for the occluding object to reproduce the shape and color of a three-dimensional object in real space when viewed from at least one viewpoint.

[0042] Fig. 6 shows a schematic diagram of a target space Vs generated from a photograph Pt. In the example shown in Fig. 6, two occluding objects Oj1 and Oj2 are generated based on two three-dimensional objects Tg1 and Tg2, which are a display stand and a figurine, included in the photograph Pt, and a target space Vs is constructed in which these occluding objects Oj1 and Oj2 are arranged.

[0043] 5, virtual space generating device 20 outputs the generated target space data via a network so that it can be used by user terminal 40 (step S13). For example, virtual space generating device 20 may store drawing data 43a including the target space data in cloud storage, or may provide it as application software.

[0044] The background data included in the drawing data 43a may also be generated by the virtual space generation device 20. The virtual space generation device 20 generates the background data from the photograph data used to generate the target space data. For example, the background data is generated from the entire photograph or from the portion other than the three-dimensional object that served as the basis for the occluding object. The generation of such background data is also a process performed by the target space generation unit 22a.

[0045] [Draw target space] The procedure for processing related to rendering of the target space will be described with reference to FIGS. First, an overview of the process of drawing a target space by the user terminal 40 will be described. The user terminal 40 acquires drawing data 43a via a network. The user terminal 40 then sets the position of the user's own avatar, which is the avatar of the user operating the user terminal 40, and the position of the viewpoint in the target space. The positions of the user's own avatar and the viewpoint are changed, for example, from a predetermined initial position in response to the user's operation on the user terminal 40. The position of the viewpoint may move in accordance with the movement of the avatar's position. Furthermore, the user terminal 40 sets the position of the other user's avatar, which is the avatar of another user, in the target space based on information from the information relay device 30.

[0046] The user terminal 40 determines a rendering range according to the position of the viewpoint, i.e., a predetermined range as seen from the viewpoint, and generates an image of the rendering range in the target space based on the data and position information of the occluding object and avatar.The user terminal 40 then uses background data to synthesize the image with a background according to the position of the viewpoint, thereby generating display data.The display unit displays the image using the display data, and the image of the target space is displayed on the user terminal 40.

[0047] 7 is a schematic diagram of a target space Vs in which the positions of avatars and viewpoints are set. In the example shown in FIG. 7, a user's own avatar Aa and one other user's avatar Ab are placed in the target space Vs. Depending on the positions of the occluding objects Oj1, Oj2 and the avatars Aa, Ab relative to the position of the viewpoint Cm, the occluding objects Oj1, Oj2 and the avatars Aa, Ab are drawn so as to be partially hidden by others. This allows a user viewing an image of the target space Vs displayed on the user terminal 40 to understand the relative positions of the structures and avatars within the target space Vs. Note that setting and changing the positions of the viewpoint and avatar also includes setting and changing the orientation of the viewpoint and avatar.

[0048] As described above, the user terminal 40 moves the position of the viewpoint in response to a user operation on the user terminal 40. At this time, the movable range of the viewpoint is limited to a range that does not disrupt the structure or background in the target space. Specifically, the movable range of the viewpoint is limited to a range in which the shape, color, and arrangement of the occluding object as seen from the viewpoint are reproduced according to the real space, or a range in which a background according to the viewpoint can be generated from background data. For example, the movable range of the viewpoint is a predetermined range centered on a position in the target space that corresponds to the shooting position of the photograph used to generate the target space. Such processing related to the movement of the viewpoint is processed by the information management unit 42a.

[0049] This makes it less likely that a user will feel uncomfortable viewing an image of the target space, even if the area in which the real space is reproduced is limited within the target space or its background area, such as when the target space is generated from a photograph with a limited viewpoint or range.Furthermore, it is possible to reduce the load required to construct the target space compared to when the real space is reproduced in all areas of the target space.

[0050] Next, each process performed by the user terminal 40 will be described in detail. First, with reference to FIG. 8, the exchange of information between the user terminals 40 via the information relay device 30 will be described. The user terminal 40 and the information relay device 30 are connected via a network, for example, when drawing of a target space begins. FIG. 8 shows, as an example, the procedure for exchanging information between two user terminals 40, a first user terminal 40A and a second user terminal 40B. Both the first user terminal 40A and the second user terminal 40B are performing drawing processing on a common target space.

[0051] As shown in FIG. 8, when the position of the avatar of the first user operating the first user terminal 40A is changed based on the operation of the user terminal 40A, the user terminal 40A transmits avatar position information IaA indicating the position of the avatar of the first user to the information relay device 30 (step S20).

[0052] Similarly, when the position of the avatar of the second user operating the second user terminal 40B is changed based on the operation of the user terminal 40B, the user terminal 40B transmits avatar position information IaB indicating the position of the avatar of the second user to the information relay device 30 (step S21).

[0053] When the information relay device 30 receives information from the user terminal 40, it identifies another user terminal 40 as a destination and transmits the information as a relay process (step S22). For example, the information is transmitted to a user terminal 40 that is drawing on the same target space.

[0054] As a result, the avatar position information IaA of the first user is transmitted to the second user terminal 40B, and the avatar position information IaB of the second user is transmitted to the first user terminal 40A. Then, in the first user terminal 40A, the avatar position information IaB is reflected in the positions of the other user's avatars, and the rendering process of the target space is performed (step S23). In the first user terminal 40A, the avatar position information IaA is reflected in the position of the user's own avatar.

[0055] Similarly, in second user terminal 40B, the avatar position information IaA is reflected in the positions of the other person's avatars, and the rendering process of the target space is performed (step S24). In second user terminal 40B, avatar position information IaB is reflected in the position of the user's own avatar.

[0056] As a result, the movement of the avatars of other users is reflected in the image of the target space displayed on the user terminal 40. Since the avatar position information is transmitted as text data, it is possible to reflect the movement of the avatar while suppressing the volume of transmitted data. Note that such exchange of avatar position information may be performed between three or more user terminals 40, and information other than avatar position information may also be exchanged. Furthermore, the avatar position information may include information on the orientation of the avatar. Such information exchange is managed by the relay processing unit 32a of the information relay device 30.

[0057] Next, with reference to FIGS. 9 and 10, a process relating to drawing of the target space by the user terminal 40 will be described. In the user terminal 40, information indicating the positions of the occluding object, the viewpoint, the user's own avatar, and the other person's avatar is stored in a predetermined area of the memory based on the reading of the drawing data 43a, the operation of the user terminal 40, and the reception of information from the information relay device 30. Of these, the position information of the occluding object is fixed. The position information of the viewpoint and the user's own avatar is updated based on the operation of the user terminal 40, and the position information of the other person's avatar is updated based on the information from the information relay device 30.

[0058] 9, when any of the position information of the viewpoint, the position information of the user's own avatar, and the position information of another user's avatar is changed due to an information update (step S30), a flag is set and each piece of information is copied to an area in memory for rendering processing (step S31). These processes are performed by the information management unit 42a.

[0059] 10, the above-mentioned information, i.e., the shielding object position information Io, the viewpoint position information Ic, the user's own avatar position information Ia, and the other person's avatar position information Ib, is stored in the first storage area 45 of the user terminal 40. When there are multiple shielding objects, multiple pieces of shielding object position information Io are stored, and when there are multiple other person's avatars, multiple pieces of other person's avatar position information Ib are stored.

[0060] As described above, the information held in the first storage area 45 is updated based on the operation of the user terminal 40 and information from the information relay device 30. When at least one of the viewpoint position information Ic, the self avatar position information Ia, and the other person's avatar position information Ib is changed, the information held in the first storage area 45 is copied to the second storage area 46. For changed information, the changed information is copied.

[0061] At this time, viewpoint flag information Fc, self avatar flag information Fa, and other-person avatar flag information Fb are also stored in the second storage area 46. The viewpoint flag information Fc has a function of indicating whether or not the viewpoint position information Ic has been changed. That is, when the viewpoint position information Ic has been changed, the viewpoint flag information Fc is set to true, and when the viewpoint position information Ic has not been changed, the viewpoint flag information Fc is set to false. The self avatar flag information Fa has a function of indicating whether or not the self avatar position information Ia has been changed. That is, when the self avatar position information Ia has been changed, the self avatar flag information Fa is set to true, and when the self avatar position information Ia has not been changed, the self avatar flag information Fa is set to false. The other-person avatar flag information Fb has a function of indicating whether or not the other-person avatar position information Ib has been changed. That is, when the other-person avatar position information Ib has been changed, the other-person avatar flag information Fb is set to true, and when the other-person avatar position information Ib has not been changed, the other-person avatar flag information Fb is set to false.

[0062] Returning to FIG. 9, after duplicating the information, the information management unit 42a issues a drawing instruction (step S32). In response to this, the drawing processing unit 42b draws the target space using the information stored in the second storage area 46 and the drawing data 43a (step S33). That is, the drawing processing unit 42b sets a drawing range according to the viewpoint position information Ic, and draws the occluding objects and avatars included in the drawing range based on the occluding object position information Io, the subject avatar position information Ia, and the other person's avatar position information Ib. When drawing, the respective flag information Fa, Fb, and Fc may be used. For example, if the viewpoint flag information Fc is false, a new drawing range is not set, and the same drawing range as in the previous drawing process may be used.

[0063] Furthermore, the information management unit 42a checks whether the self avatar flag information Fa is true (step S34), and if the self avatar flag information Fa is true (positive determination in step S34), transmits the self avatar position information Ia to the information relay device 30 via the communication unit 41 (step S35). If the self avatar flag information Fa is false (negative determination in step S34), the self avatar position information Ia is not transmitted.

[0064] The rendering processing unit 42b generates display data by combining the background with the rendered image of the target space generated in step S33, and outputs the display data to the display unit (step S36). As a result, the display image on the display unit of the user terminal 40 is updated, and a new image of the target space is displayed. Note that the background may be combined with the rendering of the target space in step S33, or the avatar may be rendered first and then combined with the occluding object and background.

[0065] As described above, in this embodiment, a target space is constructed by generating occluding objects from photographs. Photographs of real spaces are readily available, so compared to constructing a virtual space through 3DCG modeling and material and texture setting processes, a virtual space that highly reproduces the real space can be easily constructed without increasing production costs or time. Furthermore, even if the interior of a store or other space in the real space is changed, the changes can easily be reflected in the virtual space.

[0066] Furthermore, because the texture of the object is generated from a photograph, the user is likely to get the same impression as a photograph regarding the appearance, such as the color and texture, of the structure in the virtual space, i.e., the impression that it closely resembles the real space.

[0067] By placing an avatar in such a virtual space, it becomes possible to carry out the same activities in the virtual space as in the real world. The ease with which virtual spaces can be constructed also allows for the expansion of the uses of virtual spaces.

[0068] [Switch drawing target] In the user terminal 40, the occluding object to be drawn may be switched in response to movement of the viewpoint. Such switching processing will be described with reference to Figs.

[0069] When the switching process is performed, the target space data of the rendering data 43a includes a plurality of shielding object data and intermediate object data. These data are generated by the target space generation unit 22a of the virtual space generation device 20.

[0070] A single occluding object data set includes 3D model and position data of one or more occluding objects simultaneously positioned in the target space. In other words, when no switching process is performed, the target space data is composed of a single occluding object data set. A single intermediate object data set also includes 3D model and position data of one or more occluding objects simultaneously positioned in the target space.

[0071] Specifically, the virtual space generation device 20 generates occluding object data from a plurality of pieces of photographic data obtained by photographing a single real space from different positions. For example, as shown in Fig. 11, a first photograph Pt1 and a second photograph Pt2 obtained by photographing the real space from a position closer to the three-dimensional object (subject) than the first photograph Pt1 are used. The first photograph Pt1 and the second photograph Pt2 include at least one common three-dimensional object.

[0072] Then, by extracting a three-dimensional object from the first photograph Pt1, occluding object data Ds1, which is data on a first occluding object, is generated. If the first photograph Pt1 includes multiple three-dimensional objects, each occluding object corresponding to each three-dimensional object is a first occluding object, and the occluding object data Ds1 is data on a group of occluding objects. Similarly, by extracting a three-dimensional object from the second photograph Pt2, occluding object data Ds2, which is data on a second occluding object, is generated.

[0073] Furthermore, from the first photograph Pt1, image data is generated to be used as a background when the first occluding object is the drawing target when the target space is drawn on the user terminal 40. Similarly, from the second photograph Pt2, image data is generated to be used as a background when the second occluding object is the drawing target. The occluding object data and background data generated based on the same photograph are associated with each other and included in the drawing data 43a.

[0074] Furthermore, the virtual space generating device 20 generates intermediate object data Dm by using the first occluding object data Ds1 and the second occluding object data Ds2 and averaging the information contained in these data. That is, the intermediate object data Dm is data on an occluding object that has the average values of the first occluding object and the second occluding object as information on the coordinates and colors of each part that makes up the object. In other words, the occluding object indicated by the intermediate object data Dm is an intermediate object, and the intermediate object has a configuration intermediate between the first occluding object and the second occluding object in terms of size and position.

[0075] As described above, one intermediate object is generated for two occluding objects to be switched. For example, if the first occluding object and the second occluding object are to be switched, and further, the second occluding object and the third occluding object are to be switched, a first intermediate object is generated from the first occluding object and the second occluding object, and a second intermediate object is generated from the second occluding object and the third occluding object.

[0076] A processing procedure for switching the occluding object to be drawn by the user terminal 40 will be described with reference to Fig. 12. Fig. 12 shows, as an example, the procedure for switching the drawing target from a first occluding object based on a first photograph to a second occluding object based on a second photograph.

[0077] As shown in FIG. 12, in a state where the first blocking object is being drawn (step S50), the user terminal 40 determines whether a drawing target switching condition is met (step S51).

[0078] In detail, when the first occluding object is the drawing target, the first occluding object is drawn as an object located in the target space when drawing the target space described above with reference to FIGS. 7 to 10. A condition for switching the drawing target from the first occluding object to the second occluding object is set in advance, and this switching condition is included in the drawing data 43a and stored in the user terminal 40. The switching condition is a condition regarding the position of the viewpoint, and for example, the switching condition is that the viewpoint has moved within a predetermined range, such as when the viewpoint is closer to a specific occluding object than a predetermined position. In one example, the switching condition is that the viewpoint is closer to a position corresponding to the shooting position of the second photograph than a position corresponding to the shooting position of the first photograph.

[0079] If the switching condition is not met (negative determination in step S51), the drawing target remains the first occluding object. If the switching condition is met (positive determination in step S51), the user terminal 40 draws the target space using the drawing target as an intermediate object (step S52), and then draws the target space using the drawing target as the second occluding object (step S53). The intermediate object is an object generated by averaging the first occluding object and the second occluding object. In the above processing, the processing in step S51 is performed by the information management unit 42a, and the processing in steps S50, S52, and S53 is performed by the drawing processing unit 42b.

[0080] In this way, the rendering target is switched from the first occluding object based on the first photograph, through the intermediate object, to the second occluding object based on the second photograph. The rendering period of the intermediate object may be set in advance to, for example, a period corresponding to several frames. Because the position of each object is determined relative to a reference position in the target space, the positional relationship between the object representing the same three-dimensional object and the avatar is maintained even when the rendering target object is switched.

[0081] Switching the rendering target is, in other words, switching the occluding object placed in the target space, and if the virtual space for each occluding object is considered to be the target space, it can also be said to be switching the target space of the rendering target.

[0082] When the object to be drawn is a first occluded object, the background of the composite object is a background based on the first photograph, and when the object to be drawn is a second occluded object, the background of the composite object is a background based on the second photograph. When the object to be drawn is an intermediate object, the background may be generated from the first photograph or the second photograph, or may be generated by averaging the first photograph and the second photograph, etc.

[0083] By the above processing, the positions and sizes of the objects change gradually when the image of the target space displayed on the user terminal 40 switches from the scene corresponding to the first photograph to the scene corresponding to the second photograph. For example, if the first occluding object is an object based on the first photograph Pt1 shown in Fig. 11 and the second occluding object is an object based on the second photograph Pt2 shown in Fig. 11, the rendering target is switched across the intermediate object, and the display content changes in the image of the target space displayed on the user terminal 40 so that the three-dimensional object, which is the object, shifts to the left and expands toward the front.

[0084] Therefore, the scenery of the displayed target space changes smoothly with a three-dimensional effect as the viewpoint moves, and the user does not feel uncomfortable when the rendering target is switched. By switching the rendering target in this way, even if the reproduction range of the target space using one occluding object data and background data is limited, it is possible to expand the range in which the viewpoint and avatar can move, and the reproduction range can be expanded as a whole using multiple occluding object data.

[0085] As described above, according to this embodiment, the following effects can be obtained. (1) A virtual space is constructed by generating occluding objects from a photograph, and an avatar is placed in this virtual space. Therefore, a virtual space that is highly reproducible to the real world can be easily constructed without increasing production costs or time, and activities in the virtual space that are similar to those in the real world can be accurately realized.

[0086] (2) The location information of avatars is exchanged between user terminals 40 via the information relay device 30, so that the movements of other users' avatars are reflected in each user terminal 40. This makes it possible to reduce the amount of data transmitted while reflecting the movements of other users' avatars in the depiction of the virtual space in the user terminal 40.

[0087] (3) The image generated by drawing the virtual space is combined with a background generated from a photograph. This makes it possible to generate a virtual space scene that is consistent with the real world scene, even in terms of the depth direction.

[0088] (4) By limiting the range of movement of the viewpoint position, even if the area in which real space is reproduced in the virtual space or its background is limited, the structure and background of the virtual space are prevented from being distorted in the image of the rendered virtual space.

[0089] (5) By switching between occluding objects to be rendered, it is possible to expand the range of movement of the viewpoint and avatar, and the reproduction range in virtual space can be expanded using multiple occluding object data as a whole. Furthermore, by switching between occluding objects to be rendered via intermediate objects, natural transitions with a sense of three-dimensionality can be achieved in the transition of scenery in virtual space.

[0090] [Variations] The above embodiment can be modified as follows. The rendering process of the target space may be performed by the virtual space generation device 20 or the information relay device 30, and the generated image may be transmitted to the user terminal 40. In this case, the rendering data 43a is read by the device that performs the rendering process, and operation information of the avatar and viewpoint in the user terminal 40 is transmitted to the device that performs the rendering process and reflected in the drawing. The device that performs this rendering process is an information processing device, and a program that is stored in the device and causes the rendering process to be performed is an information processing program. Furthermore, the information processing device may perform both the generation process and the rendering process of the target space.

[0091] The virtual space generation device 20 and the information relay device 30 may be a single computer device. The information processing device may be a single computer device, or may be configured from multiple computer devices. [Explanation of symbols]

[0092] Aa, Ab...Avatar Oj1, Oj2...occluding objects Pt, Pt1, Pt2...Photos Tg1,Tg2…3D object Vs…target space 10...Management system 20...Virtual space generation device 30...Information relay device 40...User terminal 100...Information Processing Systems

Claims

1. an object generated based on detection of a three-dimensional object from a photograph taken of a real space is an occluding object, and data of a virtual space in which the occluding object is placed is read; setting the positions of an avatar and a viewpoint in the virtual space; Rendering the virtual space as seen from the viewpoint. Information processing device.

2. the information processing device is connected to an information relay device via a network, the avatars include a self-avatar, which is an avatar of a user who operates the information processing device, and other-user avatars, which are avatars of other users; The information processing device includes: transmitting location information of the user's own avatar, which is changed in response to an operation of the information processing device, to the information relay device, and receiving location information of the other user's avatar from the information relay device; The virtual space including each avatar is drawn, reflecting the position information of the self avatar and the position information of the other avatar. The information processing device according to claim 1 .

3. The information processing device includes: The background generated from the photograph is synthesized with the image generated by drawing the virtual space. The information processing device according to claim 1 .

4. The information processing device includes: The range in which the viewpoint position can be moved in response to a user operation is limited to a predetermined range. The information processing device according to claim 1 .

5. the virtual space data includes data of a first occluding object generated from a first photograph, data of a second occluding object generated from a second photograph, and data of an intermediate object that is an object generated by averaging configuration information of the first occluding object and configuration information of the second occluding object; a condition for switching the occluding object to be drawn from the first occluding object to the second occluding object is set for the position of the viewpoint; When the condition is met, the information processing device switches the occluding object to be drawn from the first occluding object to the second occluding object via the intermediate object, and draws the virtual space. The information processing device according to claim 1 .

6. On one or more computers, an object generated based on detection of a three-dimensional object from a photograph taken of a real space is an occluding object, and data of a virtual space in which the occluding object is placed is read; setting the positions of an avatar and a viewpoint in the virtual space; Rendering the virtual space as seen from the viewpoint; An information processing program that executes the above.

7. One or more computers an object generated based on detection of a three-dimensional object from a photograph taken of a real space is an occluding object, and data of a virtual space in which the occluding object is placed is read; setting the positions of an avatar and a viewpoint in the virtual space; Rendering the virtual space as seen from the viewpoint; An information processing method that performs the above.

Images