Information processing device, method, program, and information processing system

Abstract

To provide a new hearing experience to a user.SOLUTION: The information processor of the present invention includes: an acquisition unit for acquiring space information showing the position of an object in a first space 1 around a first user; a space forming unit for forming a shared space shared by the first user and a second user in a second space 2, which is not the first space 1, on the basis of the space information, the arrangement of the object in the first space 1 being reflected on the shared space; and a determination unit for determining the position of the second user in the shared space.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, a method, a program, and an information processing system. [Background technology]

[0002] In recent years, technologies have been developed that allow multiple users in different locations to experience being in the same place by sharing body movements and space in real time. For example, a technology called telexistence provides an environment in which a user can perform operations in real time while feeling as if an object or person in a remote location is nearby through a head-mounted display or the like. Summary of the Invention [Problem to be solved by the invention]

[0003] In the above-mentioned technology, there is a demand for providing users with a novel viewing experience.

[0004] In view of the above problems, one object of the present invention is to provide a user with a novel viewing experience. [Means for solving the problem]

[0005] In order to solve the above problem, an information processing device of one embodiment of the present invention includes an acquisition unit that acquires spatial information indicating the position of an object in a first space around a first user, a space construction unit that constructs a shared space shared by the first user and a second user existing in a second space different from the first space based on the spatial information, the shared space reflecting the position of the object in the first space, and a determination unit that determines the position of the second user in the shared space.

[0006] In addition, any combination of the above, or mutual substitution of the components or expressions of the present invention among methods, devices, programs, temporary or non-temporary storage media recording programs, systems, etc., are also valid aspects of the present invention. Effect of the Invention

[0007] According to the present invention, a novel viewing experience can be provided to the user. [Brief description of the drawings]

[0008] [Figure 1] 1 is an overall schematic diagram of an information processing system. [Diagram 2] 13 illustrates an example of a user using the embodiment. [Diagram 3] FIG. 2 is a functional block diagram of the information processing system. [Figure 4A] An example of an AR space image displayed by user A's HMD is shown. [Figure 4B] An example of an AR space image displayed by user A's HMD is shown. [Diagram 5] FIG. 2 is a sequence diagram showing a process flow in the information processing system. [Figure 6A] An example of an AR space image displayed by user A's HMD is shown. [Figure 6B] An example of an AR space image displayed by user A's HMD is shown. [Figure 7] 13 illustrates an example of an AR space image displayed by user B's HMD. [Figure 8] FIG. 2 is a sequence diagram showing a process flow in the information processing system. [Figure 9] 1 is an overall schematic diagram of an information processing system. [Figure 10] FIG. 2 is a functional block diagram of the information processing system. [Figure 11] FIG. 2 is a sequence diagram showing a process flow in the information processing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [First embodiment] Fig. 1 is an overall schematic diagram of an information processing system 10 according to this embodiment. The information processing system 10 in Fig. 1 includes a plurality of information processing terminals 100. The plurality of information processing terminals 100 are connected to each other so as to be able to perform data communications via a communication network 5 such as the Internet. The information processing system 10 of this embodiment includes two information processing terminals, an information processing terminal 100A used by a user A and an information processing terminal 100B used by a user B. However, the present invention is not limited to this, and the information processing system 10 may include three or more information processing terminals 100.

[0010] The information processing terminals 100A and 100B include a control unit 11, a storage unit 12, a communication unit 13, and an interface unit 14. The information processing terminals 100A and 100B are connected to a head mounted display (HMD) 15, a stereo camera 16, a tracker 17, and an input device 18. The control unit 11 includes a processor and executes a program stored in the storage unit 12 to execute various information processing. The storage unit 12 includes a memory device such as a RAM and stores the program executed by the control unit 11 and the data processed by the program. The communication unit 13 is a communication interface for performing data communication via the communication network 5. In this embodiment, the information processing terminals 100A and 100B communicate with each other through their respective communication units 13 by P2P (peer-to-peer) connection. This P2P connection establishes a low-delay, high-image-quality, bidirectional communication path compared to a connection between a server 200 and the information processing terminal 100 described later.

[0011] The interface unit 14 is an interface for data communication between the HMD 15, the stereo camera 16, the tracker 17, and the input device 18. The information processing terminal 100 is connected to the HMD 15, the stereo camera 16, the tracker 17, and the input device 18 via the interface unit 14 in a wired or wireless manner. Specifically, the interface unit 14 includes a multimedia interface such as HDMI (registered trademark) (High-Definition Multimedia Interface), which is a standard for a communication interface that transmits video and audio as digital signals. The interface unit 14 also includes a data communication interface such as USB to receive a video signal transmitted from the stereo camera 16.

[0012] The HMD 15 is a viewing device worn on the user's head to view still images and videos displayed on a display and listen to audio and music output from headphones. The HMD 15 is provided with a gyro sensor and an acceleration sensor. Using these sensors, the HMD 15 measures the position information of the head of the user wearing the HMD 15 and the rotation angle and inclination of the head. The HMD 15 is equipped with a camera unit. Using the camera unit, the HMD 15 can capture the outside world while the user is wearing the HMD.

[0013] The stereo camera 16 includes a plurality of imaging elements arranged side by side. By analyzing the images captured by each of these imaging elements, the information processing terminal 100 can measure the distance from the imaging position to an object present within the field of view of the stereo camera 16. This allows the information processing terminal 100 to estimate the position and shape of an object present in the space around the stereo camera 16. In this embodiment, the stereo camera 16 includes the user and objects present around the user, such as floors and walls, within its field of view, and is used to identify their positions.

[0014] The tracker 17 includes sensors such as an inertial sensor, a geomagnetic sensor, an acceleration sensor, and a motion sensor, and can detect the position and posture of the user's body parts. The input device 18 includes a plurality of input devices such as a plurality of operation buttons and an analog stick capable of inputting analog amounts. The input device 18 supplies operation information input by the user via the input device to the control unit. The input device 18 includes a plurality of input devices such as a plurality of push-type operation buttons, an analog stick capable of inputting analog amounts, and a rotary button.

[0015] Posture information of users A and B is generated by estimating the posture and position of the entire body of each of users A and B based on detection data from the HMD 15, the stereo camera 16, and the tracker 17. The posture information of users A and B is reflected in a shared space (to be described later) shared by users A and B.

[0016] Fig. 2 shows an example of how this embodiment is used by a user. As shown in Fig. 2, users A and B exist in different spaces in the real world. Specifically, user A exists in a first space 1, and user B exists in a second space 2 that is different from the first space 1. HMDs 15A and 15B are worn on the heads of users A and B, respectively. The first space 1 and second space 2 in this embodiment are the rooms of users A and B, respectively.

[0017] In this embodiment, the HMD 15A of the user A mainly functions as a video see-through type HMD that can capture an image of the outside world using a mounted camera unit and display it on a display panel. This video see-through type HMD can generate and display an image of augmented reality (AR (Augmented Reality)) (AR space image) by superimposing a virtual object generated by computer graphics (CG) on an image of the outside world. Note that, as described later, the HMD 15A of this embodiment can function as a non-see-through type HMD that can display an image of virtual reality (VR (Virtual Reality)) (VR space image) separated from the real world when, for example, the avatar of the user A falls from the floor surface of the shared space.

[0018] In this embodiment, the HMD 15B of the user B is a non-transparent HMD that can display a VR space image on a display panel. When the user B turns his / her head, this non-transparent HMD displays a 360-degree panoramic VR space image.

[0019] The stereo cameras 16A and 16B are disposed in front of the users A and B, respectively. The stereo cameras 16A and 16B each have a field of view that includes the users A and B in front, the floor on which the users A and B are standing, the walls of the room where the users A and B are present, and the like.

[0020] Trackers 17A and 17B are attached to both hands of user A and both hands and feet of user B, respectively. Here, an example in which trackers are attached to both hands and both feet of a user is described, but tracker 17 may be attached to only one of a hand or a foot, or tracker 17 may be attached to another body part such as the torso. Input devices 18A and 18B are held by both hands of users A and B, respectively.

[0021] In this embodiment, two spaces, a first space 1 and a second space 2 located away from the first space 1, are combined by a shared space described below, providing an experience in which both user A and user B can interact as if they were in the same place (first space 1 on user A's side). For example, by providing an experience in which the avatar of user B, who is not in the first space 1, is exactly in the first space 1, it becomes possible to enjoy interaction and communication.

[0022] User A is provided with a video see-through experience through the AR space image. For example, User A can destroy a virtual object that resembles a real wall in the AR space image by interacting with the body. In addition, User B's avatar appears to be in the AR space image and can communicate with the user using their body, such as gestures.

[0023] User B is provided with an experience based on the VR space image. For example, in the VR space image, User B can enter User A's room (first space 1) with the body of his avatar. Similarly, User B can destroy a virtual object that imitates a real wall in the VR space image by interacting with his body. Also, in the VR space image, the body of User B's avatar can be changed to be larger or smaller.

[0024] Fig. 3 is a functional block diagram of the information processing system 10 according to this embodiment. The functional blocks in each diagram including Fig. 3 can be realized in various forms by hardware only, software only, or a combination thereof. The information processing terminal 100A includes a first acquisition unit 51, a space construction unit 52, a space update unit 53, and a first drawing unit 54. The information processing terminal 100B includes a second acquisition unit 55, a provision unit 56, and a second drawing unit 57.

[0025] The first acquisition unit 51 acquires space information indicating the position, color, and texture of objects in the first space 1 around the user A. Here, the "objects" include the floor, walls, ceiling, and objects (such as a table) in the space in which the user is present, as well as the user's body itself. The first acquisition unit 51 is an example of an acquisition unit. The second acquisition unit 55 acquires shared space information and space update information, which will be described later.

[0026] The first acquisition unit 51 and the second acquisition unit 55 acquire posture information indicating the postures of the users A and B, respectively. As described above, the posture information is generated based on detection data of the HMD 15, the stereo camera 16, and the tracker 17. The first acquisition unit 51 and the second acquisition unit 55 acquire operation information indicating a desired operation of the users in response to operation inputs of the users A and B, respectively. The operation information is generated based on the operation inputs via the input device 18.

[0027] The space construction unit 52 constructs a shared space that reflects the positions of objects in the first space 1 based on the spatial information. The space construction unit 52 generates shared space information that indicates a position coordinate system (hereinafter referred to as a shared coordinate system) of the constructed shared space.

[0028] The space update unit 53 updates the shared space based on the space information, the posture information and the operation information of users A and B. As a result, the position of user B in the shared space is determined. The space update unit 53 generates space update information indicating the state of the updated shared space. The space update unit 53 is an example of a determination unit.

[0029] The providing unit 56 provides the posture information and operation information of the user B acquired by the second acquiring unit 55 to the first acquiring unit 51. The first drawing unit 54 and the second drawing unit 57 draw the state in the shared space by the image generated by using the space update information, and display it on the HMD 15. At this time, the first drawing unit 54 and the second drawing unit 57 identify the positions of the heads of the users A and B who are viewing the displayed shared space image in the shared coordinate system, and place a virtual camera at the identified position. Then, the first drawing unit 54 and the second drawing unit 57 draw the state as seen in the shared space from the placed virtual camera. This allows the users A and B to view the state in the shared space together with other users who are in remote locations.

[0030] 4A and 4B show examples of AR space images displayed by the HMD 15A of user A. In the example of FIG. 4A, a virtual object of a sword is superimposed on the hand of user A in the AR space image. Also, in the AR space image, a virtual object of a humanoid avatar of user B is superimposed in the center of the room. Therefore, the HMD 15A of user A displays, via the AR space image, a scene in which user A is fighting with a sword in the foreground of the screen against the humanoid avatar of user B in the center of the screen in the room where user A is present (first space 1). In the example of FIG. 4B, user A is being attacked by the avatar of user B.

[0031] A process flow in the information processing terminals 100A and 100B according to this embodiment will be described with reference to the sequence diagram of Fig. 5. In this embodiment, the information processing terminal 100A of user A functions as a server.

[0032] First, in S11, the information processing terminal 100A establishes a connection with the information processing terminal 100B. In this embodiment, the information processing terminal 100A establishes a P2P (peer-to-peer) connection with the information processing terminal 100B via the communication network 5. In this embodiment, the information processing terminal 100A notifies the acceptance of game participation and approves a game play request from the information processing terminal 100B, and then establishes a P2P connection with the information processing terminal 100B.

[0033] In S12, the first acquisition unit 51 acquires spatial information indicating the position, color, and texture of an object in the first space 1 around the user A. In this embodiment, the first acquisition unit 51 identifies the position, color, and texture of an object present within the imaging range of the stereo camera 16A by analyzing an image captured by the stereo camera 16A. The first acquisition unit 51 acquires spatial information based on the position, color, and texture of the identified object.

[0034] Specifically, the first acquisition unit 51 calculates the distance to a part of the subject captured in each of a plurality of unit areas obtained by dividing the imaging range of the stereo camera 16 into a grid. This allows the first acquisition unit 51 to acquire a distance image (depth map) including information on the distance to the object captured in each unit area for each unit area. Hereinafter, the part of the object existing in the real space that is captured in each unit area in the distance image is referred to as a unit part.

[0035] The first acquisition unit 51 calculates the position coordinates of each unit area in the real space using the position of the unit area in the distance image (i.e., the direction of the unit area as seen from the installation position of the stereo camera 16) and the numerical value of the distance to the unit part captured in the unit area. These position coordinates are three-dimensional position coordinates in a coordinate system with the installation position of the stereo camera 16 as the reference position. By using the position coordinates of each of the multiple unit parts thus obtained, the position and shape of an object having a complex structure such as a user's body or a table are identified. Furthermore, for the position coordinates of each of the multiple unit parts thus obtained, the color and texture are identified based on the captured image.

[0036] The spatial information includes floor surface information that specifies the floor surface in the first space 1. In this embodiment, the first acquisition unit 51 generates the floor surface information by assuming that the plane formed by the unit parts located at the lowest position is the floor surface.

[0037] In S13, the space construction unit 52 constructs a shared space based on the spatial information. In S13, the space construction unit 52 constructs a shared coordinate system based on the position coordinates specified in the spatial information so as to reflect the position of an object in the first space 1. In addition, in the shared coordinate system, the floor surface of the shared space is set so as to correspond to the floor surface of the first space 1 based on the floor surface information included in the spatial information. Note that a predetermined virtual object (for example, a virtual object showing outer space) is placed in the space outside the space corresponding to the first space 1 in the shared space. In this embodiment, the initial position of user B is set to a predetermined position in the space corresponding to the first space 1 in the shared space.

[0038] In S14, the space constructing unit 52 generates shared space information indicating a shared coordinate system of the constructed shared space, and provides the information processing terminal 100B.

[0039] At S15, the second rendering unit 57 generates a VR space image based on the shared space information so as to reflect the constructed shared space. The second rendering unit 57 aligns the floor surface of the second space 2 with the floor surface set in the shared space based on the shared coordinate system, and generates and renders a VR space image so as to form walls, ceilings, and objects based on this floor surface. By displaying the VR space image in this manner, the user B can confirm his / her own state in the shared space through the VR space image. The user B can execute an operation input based on the VR space image in which only the position of the object is reflected.

[0040] In S16, the first acquisition unit 51 acquires the space information of the first space 1, the posture information and operation information of the user A, and also acquires the posture information and operation information of the user B provided via the provision unit 56 of the information processing terminal 100B. Here, the operation information may include, for example, operation information for making the user's avatar execute a specific operation (for example, firing a bullet, making a sword appear, etc.). In addition, when an HMD for VR space images such as the HMD 15B of the user B is used, the operation information further includes movement information for moving the user's avatar to a desired position in the shared space, and scale information for setting the scale of the user's avatar. The scale information includes a scale value for adjusting the scale of the user's avatar.

[0041] In S17, the space update unit 53 updates the shared space based on the space information of the first space 1 acquired in S16, and the posture information and operation information of the users A and B. Specifically, the space update unit 53 determines the position, shape (including size), color, and texture of the virtual object in the shared space for each unit part. The virtual objects in this embodiment are, for example, floors, walls, ceilings, objects, avatars of the users A and B, swords, bullets, etc. Note that the virtual objects of the floors, walls, ceilings, and objects in the shared space are composed of a collection of fragments so that they can be destroyed.

[0042] Based on the spatial information and the posture information of user A, the space update unit 53 determines the position and posture of user A in the shared space so as to correspond to the current position and posture of user A in the first space 1. Based on the determined position and posture of user A, the space update unit 53 updates the position and posture of the virtual object of user A's avatar in the shared space.

[0043] The space update unit 53 determines the posture of user B in the shared space so as to correspond to the current posture of user B based on the posture information of user B, and determines the position of user B in the shared space to a position specified by the movement information. In addition, in this embodiment, the space update unit 53 determines the size of a virtual object of user B's avatar in the shared space based on the scale information of user B. For example, based on a scale value of the scale information, the space update unit 53 changes the size so as to increase or decrease the scale of user B's avatar in the shared space when the specified scale is different from the current scale. The space update unit 53 updates the position, posture, and scale of the virtual object of user B's avatar in the shared space based on the determined position, posture, and scale of user B.

[0044] Based on the spatial information, the space update unit 53 determines the positions, colors and textures of the floor, walls, ceiling and items in the shared space so as to reflect the positions, colors and textures of the floor, walls, ceiling and items in the current first space 1. Based on the determined positions, colors and textures of the floor, walls, ceiling and items, the space update unit 53 updates the positions, colors and textures of the virtual objects of the floor, walls, ceiling and items in the shared space.

[0045] In this embodiment, the space update unit 53 determines whether or not a virtual object related to at least one of the avatars of the users A and B has come into contact with a first-space virtual object forming a space corresponding to the first space 1 in the shared space, based on the space information, the posture information of the users A and B, and the operation information. Here, the virtual object related to the avatar in this embodiment is a virtual object operated by the avatar, such as a predetermined body part (arm, etc.) of the avatar, a sword held by the avatar, and a bullet fired by the avatar. In addition, the first-space virtual object in this embodiment is a virtual object forming a floor, a wall, a ceiling, and an object corresponding to the first space 1 in the shared space. However, without being limited thereto, the first-space virtual object may be a virtual object forming at least one of the floor, the wall, the ceiling, and an object corresponding to the first space 1 in the shared space. The first-space virtual object is composed of a plurality of virtual objects that become its fragments.

[0046] When it is determined that the virtual object related to the avatar has come into contact with the first-space virtual object, the space update unit 53 changes the state of the contacted portion of the first-space virtual object. This change in state includes, for example, destruction, burning, freezing, or coloring of the contacted portion of the first-space virtual object. When this change in state is, for example, destruction of the contacted portion of the first-space virtual object, the space update unit 53 changes the state of the virtual object at the contacted portion so as to cause the virtual object at the contacted portion to disappear in the shared space.

[0047] In this embodiment, when a first space virtual object corresponding to the floor surface of the shared space is destroyed and at least one of the avatars of user A and user B is located at the location of the destroyed floor surface, the space update unit 53 determines the user's position relative to that avatar in the shared space so that the avatar located at the location of the destroyed floor surface falls from the position of the destroyed floor surface.

[0048] Furthermore, for example, when it is determined that a virtual object of a sword held by an avatar of one user has come into contact with a virtual object of a bullet fired by the other user, the space update unit 53 generates a virtual object to execute a specific action such as deflecting the bullet. As a result of such interactions between users A and B and the virtual objects in the shared space, the number of virtual objects is increased or decreased, or their appearance is changed in S19A and S19B described below.

[0049] In S18, the space update unit 53 provides the generated space update information to the first drawing unit 54 and also to the information processing terminal 100B.

[0050] In S19A, the first drawing unit 54 draws an AR space image so as to reflect the updated shared space based on the space update information. In S19A, the first drawing unit 54 draws a virtual object when the updated shared space is viewed from the viewpoint position and viewpoint direction of the user A wearing the HMD 15A. In this embodiment, the first drawing unit 54 draws a first space virtual object so as to reflect the floor, wall, ceiling, and object of the first space 1 in the shared space by laying out virtual objects of these fragments in accordance with the positions of the floor, wall, ceiling, and object of the first space 1 in the field of view of the HMD 15A. The first drawing unit 54 superimposes the photographed image (video see-through image) of the floor, wall, ceiling, and object photographed by the HMD 15 in a post-process on the surface of the generated first space virtual object of the floor, wall, ceiling, and object. Here, the "surface of the first space virtual object" is the surface of the first space virtual object when viewed from within the space corresponding to the first space 1 in the shared space. The first drawing unit 54 draws an AR space image by superimposing virtual objects such as the avatar and sword of user B on the captured image superimposed on the virtual objects based on the space update information.

[0051] When a first-space virtual object (such as a wall) is destroyed, the virtual objects of the fragments spread out in the virtual object of the destroyed portion crumble into pieces and disappear. As a result, the first drawing unit 54 can draw a virtual object that exists on the rear side of the user A's viewpoint direction for the disappeared portion. A texture image showing a cross section of the broken virtual object of the fragments is pasted. In addition, as described above, a photographed image (video see-through image) of the HMD 15 is superimposed on the surface of the first-space virtual object in a post-process. As a result, when a first-space virtual object such as a wall is destroyed, an image shown by the virtual object of the fragments changes from the video see-through image to a predetermined texture image showing the destroyed state. As a result, it is possible to produce an effect in which the wall or the like at the contact point is actually destroyed. In this case, the video see-through image may use a texture image generated from a photographed image photographed by the HMD 15, or may use the photographed image as it is.

[0052] When the avatar of user A is located at a location on the destroyed floor surface in the shared space, the position of user A in the shared space is determined so that the avatar of user A falls from the location of the destroyed floor surface. In this case, the HMD 15A of user A displays a VR space image reflecting a space outside the space corresponding to the first space 1 in the shared space as the avatar of user A falls from the space corresponding to the first space 1 in the shared space.

[0053] In S19B, the second rendering unit 57 renders a VR space image so as to reflect the updated shared space based on the space update information. In S19B, the second rendering unit 57 renders a VR space image by generating a virtual object when the updated shared space is viewed from the viewpoint position and viewpoint direction of the user B wearing the HMD 15B based on the space update information. Specifically, the second rendering unit 57 renders a VR space image by generating a virtual object so as to reflect the floor, wall, ceiling, object, avatar and sword of the user A, etc. in the shared space. For example, the second rendering unit 57 renders a VR space image so that the surface of the first space virtual object shows an image reflecting the color and texture of the floor, wall, ceiling, object of the first space 1. When the virtual object of the fragment at the contact point disappears as described above, the second rendering unit 57 renders a virtual object present on the far side in the viewpoint direction of the user B for the disappeared part.

[0054] When the first space virtual object is destroyed, the image displayed by the virtual object of the fragment changes from an image reflecting the color and texture of the destroyed part to a texture image showing the destroyed state. As a result, it is possible to produce an effect as if the wall or the like at the contact point was actually destroyed. Note that S19A and S19B are executed simultaneously.

[0055] After this, the information processing system 10 repeatedly executes the processes from S16 to S19A and S19B. Specifically, the information processing system 10 repeatedly executes acquisition of spatial information of the first space 1, acquisition of posture information and operation information of each of the users A and B, and updating and drawing the shared space according to the acquired information. This makes it possible to present to the users A and B an appearance of the shared space in which the avatars of the users A and B exist, reflecting the body movements and the like of the users A and B.

[0056] This completes the processing in the information processing terminals 100A and 100B.

[0057] According to this embodiment, users A and B can interact with users in other spaces by using a shared space that reflects the objects in user A's room. This provides a novel experience for the users.

[0058] In this embodiment, the position of user B is determined in the shared space in a space corresponding to the first space 1. As a result, for user A, an AR space image is rendered so that an avatar of user B appears in his / her room, and for user B, a VR space image is rendered so that the user B appears as if he / she is in user A's room. This makes it possible to provide users A and B with a more immersive and realistic experience.

[0059] In this embodiment, the space update unit 53 determines the scale of the user's avatar in the shared space based on the scale information. With this configuration, it becomes possible to change the scale of the avatar of each user in the shared space. Therefore, users A and B can interact with avatars with different scales in the shared space, providing a more innovative experience.

[0060] In this embodiment, the space construction unit 52 sets the floor surface of the shared space so as to correspond to the floor surface of the first space 1. This makes the floor surface of user A and the floor surface of user B match, so that users A and B can get the feeling that they can interact with each other in the same room.

[0061] In this embodiment, the spatial information indicates the color and texture of the object in addition to the position of the object. With this configuration, an image closer to the actual room of the user A (first space 1) can be obtained, so that a more realistic experience can be provided.

[0062] In this embodiment, when a virtual object related to at least one of the avatars of user A and user B comes into contact with a first space virtual object, the state of the contact point on the first space virtual object is changed. According to this configuration, a specific effect is applied to the wall of the actual room of user A, making it possible to provide a more realistic experience.

[0063] In this embodiment, when the first space virtual object corresponding to the floor surface of the shared space is destroyed and at least one of the avatars of user A and user B is located at the location of the destroyed floor surface, the user's position relative to the avatar in the shared space is determined so that the avatar located at the location of the destroyed floor surface falls from the location of the destroyed floor surface. With this configuration, it is possible to provide the user with a novel experience of accidentally falling from the actual room of user A.

[0064] In this embodiment, the surface of the first space virtual object shows an image showing an object in the first space 1 (for example, a video see-through image, an image reflecting the color and texture of the object), and the destruction cross section of the first space virtual object shows a predetermined texture image showing the destroyed state. With this configuration, it is possible to present the user A's room as if it were actually destroyed, so that a more novel experience can be provided.

[0065] In the present embodiment, the HMD 15A mainly functions as a video see-through HMD, and the HMD 15B functions as a non-see-through HMD, but the present invention is not limited to this, and the HMDs 15A and 15B may function as either a video see-through HMD or a non-see-through HMD.

[0066] In this embodiment, the floor position of the first space 1 on the user A side and the floor position of the second space 2 on the user B side are aligned, but this is not limiting. For example, any of the floor, wall, or ceiling on one user side may be aligned with the floor, wall, or ceiling on the other user side.

[0067] In this embodiment, the information processing terminal 100A functions as a server, but is not limited thereto. For example, the information processing terminal 100B may function as a server. In this embodiment, the spatial information indicates the position, color, and texture of an object in the first space 1, but is not limited thereto, and may at least indicate the position of an object in the first space 1. In this embodiment, the spatial information indicates the position, color, and texture of an object, but is not limited thereto, and may at least indicate the position of an object. In this embodiment, the operation information is input via the input device 18, but is not limited thereto, and may be input in response to any action such as a user's gesture.

[0068] [Second embodiment] A second embodiment of the present invention will be described below. In the drawings and description of the second embodiment, the same or equivalent components and members as those of the first embodiment are denoted by the same reference numerals. Explanations that overlap with the first embodiment will be omitted as appropriate, and the description will focus on the configurations that differ from the first embodiment.

[0069] In the above first embodiment, the shared space is constructed so that an avatar of user B appears in a space corresponding to first space 1 in the shared space. On the other hand, in the second embodiment, the shared space is constructed so that an avatar of user B appears outside a space corresponding to first space 1 in the shared space. Specifically, the shared space is constructed so that first space 1 of user A appears in second space 2 of user B.

[0070] 6A and 6B illustrate an AR space image displayed by the HMD 15A of user A. In the example of FIG. 6A, user B is displayed as a giant robot avatar, not as a humanoid avatar. In this example, the robot avatar of user B is scaled to be about 10 meters tall in the AR space image. In addition, a part of the wall of user A's room is shown by a virtual object in a destroyed state. As a result, the HMD of user A displays a state in which user B's robot arm comes into contact with the wall of the room from outside user A's room, destroying the wall and entering user A's room. In the example of FIG. 6B, a state in which most of user A's room has been destroyed by user B's robot avatar is displayed.

[0071] In this embodiment, the HMD 15B of user B is a video see-through HMD that can generate and display an AR space image by superimposing a virtual object on an image of the outside world. FIG. 7 illustrates an AR space image displayed by the HMD 15B of user B. In the example of FIG. 7, both hands of user B are superimposed by a virtual object of a robot arm in the foreground of the drawing and are displayed in the AR space image. In addition, the mesh wall showing user A's room (first space) is partially destroyed, and user A's avatar can be seen from the destroyed part. Since the scale of user B's avatar is large, user A's room is reflected small in the AR space image.

[0072] According to this embodiment, user A can interact with an avatar of user B who has come from outside the room, and user B can interact with an avatar of user A who is in user A's room from within the user's room. This makes it possible to provide a more realistic experience. The process will be described below with reference to FIG. 8.

[0073] In S22, the first acquisition unit 51 acquires spatial information of the first space 1, and also acquires spatial information of the second space 2 indicating the position of an object in the second space 2. The spatial information of the second space 2 is provided via the provision unit 56 of the information processing terminal 100B. In this embodiment, the second acquisition unit 55 of the information processing terminal 100B acquires the spatial information of the second space 2 using the stereo camera 16B in a manner similar to that of the spatial information of the first space 1. The provision unit 56 provides the acquired spatial information of the second space 2 to the information processing terminal 100A. The spatial information of the second space 2 includes floor surface information indicating the floor surface of the second space 2.

[0074] In S23, the space construction unit 52 constructs a shared space based on each of the space information of the first space 1 and the second space 2. In S23, the space construction unit 52 constructs a shared space based on the position coordinates of the first space 1 and the second space 2 specified in each of the space information so that the space corresponding to the first space 1 is arranged at a predetermined position in the space corresponding to the second space 2 in the shared space. In this embodiment, the scale of the space corresponding to the second space 2 in the shared space is set so that it is large enough to accommodate the space corresponding to the first space 1. In this space corresponding to the first space 1, the position of the object in the first space 1 is reflected. The shared coordinate system in this embodiment is set so that the floor surface of the space corresponding to the first space 1 and the floor surface of the space corresponding to the second space are horizontal based on the floor surface information included in each of the space information. In this embodiment, the initial position of the user B is set to a predetermined position outside the space corresponding to the first space 1 in the shared space.

[0075] After S24, in S25, the second rendering unit 57 generates an AR space image based on the shared space information so as to reflect the shared space. In this AR space image, a box-shaped virtual object representing the first space is superimposed at a predetermined position in the second space 2. Note that the position of this box-shaped virtual object representing the first space may be changed (for example, onto a table, etc.) based on operation information of the second user. After S26 to S28, in S29B, the second rendering unit 57 generates an AR space image based on the space update information so as to reflect the updated shared space.

[0076] After that, the information processing system 10 repeatedly executes the processes from S26 to S29A and S29B. Specifically, the information processing system 10 repeatedly executes acquisition of space information of the first space 1, acquisition of posture information and operation information of each of the users A and B, and updating and drawing the shared space according to the acquired information.

[0077] This completes the processing in the information processing terminals 100A and 100B.

[0078] [Third embodiment] Hereinafter, a third embodiment of the present invention will be described. In the drawings and description of the third embodiment, the same or equivalent components and members as those of the first embodiment are denoted by the same reference numerals. Explanations that overlap with the first embodiment will be omitted as appropriate, and the configurations that differ from the first embodiment will be described in detail.

[0079] 9 includes a server 200 and a plurality of information processing terminals 100. The server 200 is connected to the plurality of information processing terminals 100 via a communication network 5 so as to be able to perform data communication with the plurality of information processing terminals 100. The server 200 includes a control unit 211, a storage unit 212, and a communication unit 213.

[0080] The control unit 211 includes a processor and executes various information processing by executing programs stored in the storage unit 212. The storage unit 212 includes a memory device such as a RAM and stores the programs executed by the control unit 211 and the data processed by the programs. The communication unit 213 is a communication interface for performing data communication via the communication network 5. In this embodiment, each information processing terminal 100 and the server 200 communicate with each other via the communication units 13 and 213 in a client-server type connection.

[0081] Fig. 10 is a functional block diagram of the information processing system 10 according to this embodiment. As shown in Fig. 10, the server 200 includes a first acquisition unit 51, a space construction unit 52, and a space update unit 53. The information processing terminal 100 includes a second acquisition unit 55, a provision unit 56, and a drawing unit 58. The drawing unit 58 has the same functions as the first drawing unit 54 and the second drawing unit 57. The first acquisition unit 51 and the second acquisition unit 55 of this embodiment are examples of a server-side acquisition unit and a terminal-side acquisition unit, respectively.

[0082] 11, a process flow in the information processing terminal 100 and the server 200 according to this embodiment will be described. In S31, the server 200 establishes a connection with each information processing terminal 100. In this embodiment, the server 200 establishes a client-server connection with each information processing terminal 100 via the communication network 5.

[0083] In S32, the first acquisition unit 51 acquires the spatial information from the information processing terminal 100. In this embodiment, the first acquisition unit 51 acquires the spatial information from at least one of the information processing terminals 100A and 100B via the provision unit 56.

[0084] In S33, the space construction unit 52 constructs a shared space based on the acquired spatial information. In this embodiment, the shared space is constructed using spatial information of a pre-specified information processing terminal 100 (information processing terminal A in this embodiment). After S34 to S36, in S37, the space update unit 53 updates the shared space based on the spatial information of the first space 1 in the information processing terminal 100A acquired in S36, and the posture information and operation information of the users A and B. In S38, the space update unit 53 provides the space update information to the information processing terminals 100A and 100B. In S39, the drawing units 58 of the information processing terminals 100A and 100B respectively draw AR space images or VR space images so as to reflect the updated shared space based on the space update information acquired via the second acquisition unit 55.

[0085] Thereafter, the information processing system 10 repeatedly executes the processes from S36 to S39. With the above, the processes in the information processing terminal 100 and the server 200 are completed.

[0086] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0087] 1 first space, 2 second space, 5 communication network, 10 information processing system, 11 control unit, 12 memory unit, 13 communication unit, 14 interface unit, 15 HMD, 16 stereo camera, 17 tracker, 18 input device, 51 first acquisition unit, 52 space construction unit, 53 space update unit, 54 first drawing unit, 55 second acquisition unit, 56 provision unit, 57 second drawing unit, 58 drawing unit, 100 information processing terminal, 200 server.

Claims

1. an acquisition unit that acquires spatial information indicating a position of an object in a first space based on an image captured of the first space around a first user, operation information generated based on an operation input by a second user who exists in a second space different from the first space, the operation information including movement information for moving an avatar of the second user to a desired position within a shared space, and posture information indicating the postures of the first user and the second user, respectively, generated based on a detection result of a detection unit that detects positions and postures of body parts of the first user and the second user; a space construction unit that constructs the shared space in which positions of virtual objects of the objects are set in the same positional relationship as a positional relationship between the objects in the first space indicated by the spatial information; a determination unit that determines positions of the first user's avatar and the second user's avatar in the shared space based on the spatial information, the position specified by the movement information, and the posture information; Including, the determination unit determines whether a virtual object operated by an avatar of at least one of the first user and the second user comes into contact with a first-space virtual object constituting an object in the shared space corresponding to at least one of a floor, a wall, a ceiling, and an object among the objects in the first space, based on the space information, the position specified in the movement information, and the posture information; and when it is determined that the operated virtual object comes into contact with the first-space virtual object, updates the shared space so as to change a state of the contacted portion of the first-space virtual object; The information processing device, wherein the change in state includes destruction, burning, freezing, or coloring of the contacted portion of the first space virtual object.

2. The information processing device according to claim 1 , wherein the determination unit determines a position of the second user within a space surrounded by the first space virtual object in the shared space.

3. The acquisition unit further acquires other space information indicating a position of an object in the second space based on an image captured in the second space, 3. The information processing device according to claim 1 or 2, wherein the space construction unit constructs the shared space so that the positions of the virtual objects of the objects are set in the same positional relationship as the positional relationship between the objects in the second space indicated by the other space information, and so that one of a space surrounded by the first space virtual object in the shared space and a space surrounded by a second space virtual object constituting an object corresponding to at least one of a floor, a wall, a ceiling, and an object among the objects in the second space is located within the other space.

4. the operation information includes scale information for setting a scale of an avatar of at least one of the first user and the second user in the shared space, The information processing device according to claim 1 , wherein the determination unit determines the scale of the avatar based on the scale information.

5. the spatial information includes floor surface information that specifies a floor surface in the first space, The information processing apparatus according to claim 1 , wherein the space construction unit sets a floor of the first space specified in the floor information as a floor of the shared space.

6. the spatial information further indicates a color and a texture of the object in the first space; The information processing device according to claim 1 , wherein the determination unit determines a color and a texture of a virtual object of the object in the shared space so that the color and texture are the same as the color and texture of the object in the first space.

7. the change in state is destruction of the contacted portion of the first space virtual object, 2. The information processing device according to claim 1, wherein when the first space virtual object constituting the floor of the shared space is destroyed and at least one of the avatars of the first user and the second user is located at the location of the destroyed floor, the determination unit determines the position of the positioned avatar in the shared space so as to drop the positioned avatar from the position of the destroyed floor.

8. the change in state is destruction of the contacted portion of the first space virtual object, the first spatial virtual object is composed of a plurality of virtual objects that become fragments of the first spatial virtual object, a surface of the first space virtual object showing an image showing the object in the first space; The information processing device according to claim 1 , wherein the destruction cross section of the first space virtual object shows a predetermined texture image showing a destroyed state.

9. acquiring spatial information indicating a position of an object in a first space based on an image captured of the first space around a first user, operation information generated based on an operation input of a second user present in a second space different from the first space, the operation information including movement information for moving an avatar of the second user to a desired position within a shared space, and posture information indicating the postures of the first user and the second user, respectively, generated based on a detection result of a detection unit that detects positions and postures of body parts of the first user and the second user; constructing the shared space in which positions of virtual objects of the objects are set in the same positional relationship as a positional relationship between the objects in the first space indicated by the spatial information; determining positions of the first user's avatar and the second user's avatar in the shared space based on the spatial information, the position specified by the movement information, and the posture information; determining whether or not a virtual object operated by an avatar of at least one of the first user and the second user comes into contact with a first space virtual object constituting an object in the shared space that corresponds to at least one of a floor, a wall, a ceiling, and an object among the objects in the first space, based on the spatial information, the position specified in the movement information, and the posture information; updating the shared space so as to change a state of the contacted portion of the first space virtual object when it is determined that the operated virtual object and the first space virtual object have come into contact with each other, the state change including destruction, burning, freezing, or coloring of the contacted portion of the first space virtual object; An information processing method comprising:

10. On the computer, acquiring spatial information indicating a position of an object in a first space based on an image captured of the first space around a first user, operation information generated based on an operation input of a second user present in a second space different from the first space, the operation information including movement information for moving an avatar of the second user to a desired position within a shared space, and posture information indicating the postures of the first user and the second user, respectively, generated based on a detection result of a detection unit that detects positions and postures of body parts of the first user and the second user; constructing the shared space in which positions of virtual objects of the objects are set in the same positional relationship as a positional relationship between the objects in the first space indicated by the spatial information; determining positions of the first user's avatar and the second user's avatar in the shared space based on the spatial information, the position specified by the movement information, and the posture information; determining whether or not a virtual object operated by an avatar of at least one of the first user and the second user comes into contact with a first space virtual object constituting an object in the shared space that corresponds to at least one of a floor, a wall, a ceiling, and an object among the objects in the first space, based on the spatial information, the position specified in the movement information, and the posture information; and a step of updating the shared space so as to change a state of the contacted portion of the first space virtual object when it is determined that the virtual object to be operated and the first space virtual object have come into contact, the change in state including destruction, burning, freezing, or coloring of the contacted portion of the first space virtual object.

11. An information processing system including a server and a plurality of information processing terminals, A first information processing terminal of the plurality of information processing terminals, a first terminal side acquisition unit that acquires spatial information indicating a position of an object in a first space based on an image captured of the first space around a first user who uses the first information processing terminal, and first posture information indicating a posture of the first user generated based on a detection result of a first detection unit that detects a position and posture of a body part of the first user; a first providing unit that provides the acquired spatial information and the acquired first attitude information to the server; Including, A second information processing terminal of the plurality of information processing terminals, a second terminal side acquisition unit that uses the second information processing terminal to acquire operation information generated based on an operation input of a second user who is present in a second space different from the first space, the operation information including movement information for moving an avatar of the second user to a desired position within a shared space, and second posture information indicating a posture of the second user generated based on a detection result of a second detection unit that detects a position and posture of a body part of the second user; a second providing unit that provides the acquired operation information and the acquired second attitude information to the server; Including, The server, a server-side acquisition unit that acquires the spatial information and the first attitude information from the first information processing terminal and acquires the operation information and the second attitude information from the second information processing terminal; a space construction unit that constructs the shared space in which positions of virtual objects of the objects are set in the same positional relationship as a positional relationship between the objects in the first space indicated by the spatial information; a determination unit that determines positions of the first user's avatar and the second user's avatar in the shared space based on the spatial information, the position specified by the movement information, the first attitude information, and the second attitude information; Including, the determination unit determines whether a virtual object operated by an avatar of at least one of the first user and the second user comes into contact with a first-space virtual object constituting an object in the shared space corresponding to at least one of a floor, a wall, a ceiling, and an object among the objects in the first space, based on the space information, the position specified in the movement information, the first posture information, and the second posture information; and when it is determined that the operated virtual object comes into contact with the first-space virtual object, updates the shared space so as to change a state of the contacted portion of the first-space virtual object; An information processing system, wherein the change in state includes destruction, burning, freezing, or coloring of the contacted portion of the first spatial virtual object.

Images