Interface device and coexistence experience providing system
The interface device uses force plates and photodetectors to accurately determine user position and posture, addressing measurement inaccuracies in virtual environments, ensuring precise coexistence experiences.
Patent Information
- Application Number
- JP2023087220
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-26
- Publication Date
- 2024-12-06
AI Technical Summary
Existing virtual environment construction devices struggle to accurately measure the position and posture of players who move outside the field of view of multiple cameras or are hidden in shadows, leading to inaccuracies in virtual space representation.
An interface device utilizing force plates to detect foot pressure and photodetectors to determine user position and posture, combined with a simulation device to generate a virtual space that accurately reflects these measurements.
Enables precise specification of user position and posture at arbitrary locations, allowing for accurate coexistence experiences in virtual spaces, even when players are outside the camera's view or obscured.
Smart Images

Figure 2024170194000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an interface device and a coexistence experience providing system. [Background technology]
[0002] Attention is being drawn to technologies (e.g., metaverse technology) that provide users with a coexistence experience in a virtual space. One example of such technology is the virtual environment construction device described in Patent Document 1. The virtual environment construction device described in Patent Document 1 provides users with a game experience from the perspective of a player in a virtual space. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] International Publication No. 2017 / 131071 Summary of the Invention [Problem to be solved by the invention]
[0004] In the virtual environment construction device described in Patent Document 1, multiple cameras are used to optically measure the positions and postures of players in real space, and a virtual environment is constructed based on the measurement results. This can cause a problem in that the positions and postures of players who move outside the angles of view of the multiple cameras or are hidden in the shadows of other players (ranges beyond the reach of optical measurements by the multiple cameras) cannot be measured correctly.
[0005] One aspect of the present invention has been made in consideration of the above-mentioned problems, and its object is to realize an interface device that can identify the position and posture of a user who is located at any position on the floor. [Means for solving the problem]
[0006] An interface device according to one embodiment of the present invention is an interface device for providing multiple users with an experience of coexistence in a virtual space, and includes a plurality of force plates laid out on the floor of a real space, and an output unit that provides, for each user, an output signal from the force plates detecting the user's foot pressure, and a signal representing the user's position on the floor and the user's posture, determined based on the position of the force plates in the real space, to a simulation device that generates the virtual space. Effect of the Invention
[0007] According to one aspect of the present invention, the position and posture of a user at any position on a floor can be identified. [Brief description of the drawings]
[0008] [Figure 1] 1 is a block diagram showing a configuration of an interface device according to an embodiment of the present invention; [Diagram 2] 2 is a block diagram showing the flow of operations of an output unit included in the interface device shown in FIG. 1. [Diagram 3] 2 is a block diagram showing a configuration of a coexistence experience providing system including the interface device shown in FIG. 1. [Figure 4] 4 is a block diagram showing a modified example of the coexistence experience providing system shown in FIG. 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] (Configuration of interface device) The configuration of an interface device 1 according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a perspective view showing the configuration of the interface device 1.
[0010] The interface device 1 is a man-machine interface for providing a coexistence experience in a virtual space to multiple users U. As shown in FIG. 1, the interface device 1 includes multiple force plates 11, multiple photodetectors 12, and an output unit 13. The multiple force plates 11 are spread over a floor F of a real space RS (a room in this embodiment). The multiple photodetectors 12 are distributed on the walls W of the real space RS. Note that FIG. 1 illustrates two of the four walls W of the real space RS. The multiple photodetectors 12 are also distributed on two walls not illustrated.
[0011] In the following description, a global coordinate system (XYZ coordinate system) associated with the real space RS, a global coordinate system (XY coordinate system) associated with the floor F, and a local coordinate system (xy coordinate system) associated with each force plate 11 are used. Here, the global coordinate system associated with the real space RS is a three-dimensional orthogonal coordinate system with a corner of the floor F as the origin O. The global coordinate system associated with the floor F is a two-dimensional orthogonal coordinate system with a common origin O with the global coordinate system associated with the real space RS. The local coordinate system associated with each force plate 11 is a two-dimensional orthogonal coordinate system with a corner of the upper surface of the force plate 11 as the origin o. The X-axis of the global coordinate system associated with the floor F is the same as the X-axis of the global coordinate system associated with the real space RS, and the Y-axis of the global coordinate system associated with the floor F is the same as the X-axis of the global coordinate system associated with the real space RS. In addition, the x-axis of the local coordinate system associated with each force plate 11 is parallel to the X-axis of the global coordinate system associated with the real space RS, and the y-axis of the local coordinate system associated with each force plate 11 is parallel to the Y-axis of the global coordinate system associated with the real space RS. In addition, the top surface of each force plate 11 is included in the XY plane (Z=0) in the global coordinate system associated with the real space RS.
[0012] The force plate 11 is a sensor for acquiring information required to identify the position of each user U on the floor F and the posture of each user U. In this embodiment, a force plate equipped with a rectangular plate and four six-axis force sensors supporting the four corners of the plate is used as the force plate 11. In this case, the output signal of the force plate 11 represents the floor reaction force (Fx, Fy, Fz), the center of pressure (x, y), and the free moment N calculated from the forces (fx, fy, fz) detected by these force sensors. The force plate 11 supplies an output signal representing the detection result to the output unit 13. The shape of the plate equipped in each force plate 11 may be square. In addition, the force sensor equipped in each force plate 11 may be a three-axis force sensor.
[0013] In this embodiment, to represent the position of each force plate 11 on the floor F, the coordinates (Xo, Yo) of the representative point of the force plate 11 in the global coordinate system associated with the floor F are used. The representative point of each force plate 11 is a corner of the upper surface of the force plate 11, i.e., the origin o of the local coordinate system associated with the force plate 11. In the following description, the position (Xo, Yo) of the representative point of each force plate 11 on the floor F will also be simply referred to as the position (Xo, Yo) of the force plate 11.
[0014] In addition, in this embodiment, to represent the position of each user U on the floor F, the coordinates (X, Y) of the representative point of the user U in the global coordinate system associated with the floor F are used. The representative point of each user U is the center of pressure (x, y) represented by the output signal of the force plate 11 that detected the foot pressure of the user U. In the following description, the position (X, Y) of the representative point of each user U on the floor F is simply referred to as the position (X, Y) of the user U. Note that, if the position (Xo, Yo) of the force plate 11 that detected the foot pressure of the user U is used, the position (X, Y) of the user U is expressed as (X, Y) = (Xo + x, Yo + y).
[0015] The photodetector 12 is a sensor for acquiring information necessary to identify the line of sight direction of each user U. In this embodiment, a photodetector equipped with a photodiode is used as the photodetector 12. In this case, the output signal of the photodetector 12 represents the intensity I of light detected by this photodiode. Each photodetector 12 supplies an output signal representing the detection result to the output unit 13.
[0016] In this embodiment, the user U wears a head mounted display 3, and the head mounted display 3 irradiates light L in the line of sight of the user U. The light detected by the photodetector 12 is the light L irradiated by the head mounted display 3. For this reason, it is preferable that the real space RS is a darkroom. This is because the real space RS being a darkroom allows the photodetector 12 to accurately detect the light L irradiated from the head mounted display 3.
[0017] In this embodiment, the position of each photodetector 12 in the real space RS is represented by the coordinates (Xp, Yp) of the photodetector 12 in a global coordinate system associated with the real space RS. In the following description, the position (Xp, Yp) of each photodetector 12 in the real space RS is also simply referred to as the position (Xp, Yp) of the photodetector 12.
[0018] The output unit 13 is configured to generate a signal Σ indicating the position (X, Y), posture P, and line of sight (Vx, Vy, Vz) of each user U, and provide the generated signal Σ to the simulation device 2. The output unit 13 includes a memory that stores the position (Xo, Yo) of each force plate 11 and the position (Xp, Yp, Zp) of each photodetector 12. The operation of the output unit 13 will be described later with reference to a different drawing.
[0019] (Output section operation) The operation of the output unit 13 included in the interface device 1 shown in Fig. 1 will be described with reference to Fig. 2. Fig. 2 is a flow chart showing the flow of the output unit 13.
[0020] 2, the output unit 13 performs a position identification process S11, a posture identification process S12, and a gaze direction identification process S13 for each user U, and further performs a signal providing process S14. The output unit 13 repeats these processes to provide the simulation device 2 with a signal Σ indicating the position (X, Y), posture P, and gaze direction (Vx, Vy, Vz) of each user U in each cycle.
[0021] The position identification process S11 is a process for identifying the position (X, Y) of each user U from the output signal of the force plate 11 (particularly, the center of foot pressure (x, y) indicated by the output signal) by referring to the position (Xo, Yo) of the force plate 11 that detected the foot pressure of the user U. In this embodiment, the position (X, Y) of each user U is identified as the coordinates (Xo+x, Yo+y) in the global coordinate system associated with the floor F of the center of foot pressure (x, y) indicated by the output signal of the force plate 11 that detected the foot pressure of the user U.
[0022] The position identification process S11 is composed of, for example, a force plate estimation process S111, a foot pressure center identification process S112, a force plate position reading process S113, and a user position calculation process S114. The force plate estimation process S111 is a process for estimating the force plate 11 that detected the foot pressure of the user U in the current cycle based on the position (X, Y) and posture P of the user U identified in the previous cycle. The foot pressure center identification process S112 is a process for identifying the foot pressure center (x, y) from the output signal of the force plate 11 estimated in the force plate estimation process S111. The force plate position reading process S113 is a process for reading the position (Xo, Yo) of the force plate 11 estimated in the force plate estimation process S111 from memory. The user position calculation process S114 is a process for calculating the position (X, Y) of the user U (X, Y) = (Xo + x, Yo + y) by adding the position (Xo, Yo) read out in the force plate position reading process S113 to the foot pressure center (x, y) identified in the foot pressure center identification process S112, component by component.
[0023] The posture identification process S12 is a process for identifying the posture P of each user U from the output signal (particularly the floor reaction force (Fx, Fy, Fz) represented by the output signal) of the force plate 11 that detects the foot pressure of the user U. In this embodiment, it is identified whether the posture P of the user U is a sitting posture P1 or a standing posture P2. If the posture P of the user U is a standing posture P2, it is further identified whether the posture P of the user U is a standing posture P21 or a walking posture P22. If the posture P of the user U is a walking posture P22, it is further identified whether the posture P of the user U is a left-foot-loaded posture P22L (a posture in which the left foot is on the floor F and the right foot is forward) or a right-foot-loaded posture P22R (a posture in which the right foot is on the floor F and the left foot is forward). That is, in this embodiment, it is determined whether the posture P of the user U is one of the sitting posture P1, the standing posture P21, the left-foot-loaded posture P22L, and the right-foot-loaded posture P22R.
[0024] When the posture P of the user U changes from a standing posture P2 to a sitting posture P1, the magnitude of the floor reaction force (Fx, Fy, Fz) detected by the force plate 11 temporarily decreases. Also, when the posture P of the user U changes from a sitting posture P1 to a standing posture P2, the magnitude of the floor reaction force (Fx, Fy, Fz) detected by the force plate 11 temporarily increases. The output unit 13 utilizes these properties to determine whether the posture P of the user U is a sitting posture P1 or a standing posture P2 from the floor reaction force (Fx, Fy, Fz) represented by the output signal of the force plate 11.
[0025] Furthermore, if the posture P of the user U is an upright posture P21, the floor reaction forces (Fx, Fy, Fz) are perpendicular or approximately perpendicular to the floor F. Furthermore, if the posture P of the user U is a walking posture P22, the floor reaction forces (Fx, Fy, Fz) are inclined in the moving direction of the user U. The output unit 13 utilizes these properties to determine whether the posture P of the user U is the standing posture P21 or the walking posture P22 from the floor reaction forces (Fx, Fy, Fz) represented by the output signal of the force plate 11.
[0026] Also, if the posture P of the user U is a left-foot-loaded posture P22L, the center of gravity of the user U is located to the right of the center of pressure (x, y) (left foot landing point), so the floor reaction force (Fx, Fy, Fz) is inclined to the right from the direction of travel. Also, if the posture P of the user U is a right-foot-loaded posture P22R, the center of gravity of the user U is located to the left of the center of pressure (x, y) (right foot landing point), so the floor reaction force (Fx, Fy, Fz) is inclined to the left from the direction of travel. The output unit 13 uses these properties to determine whether the posture P of the user U is a left-foot-loaded posture P22L or a right-foot-loaded posture P22R from the floor reaction force (Fx, Fy, Fz) represented by the output signal of the force plate 11.
[0027] The gaze direction identification process S13 is a process for identifying the gaze direction (Vx, Vy, Vz) of each user U from the output signal of each photodetector 12 with reference to the position (Xp, Yp, Zp) of each photodetector 12. In this embodiment, as the gaze direction (Vx, Vy, Vz) of each user U, a vector (Xp-Xh, Yp-Yh, Zp-Zh) is identified, starting from the position (Xh, Yh, Zh) in the real space RS of the head mounted display 3 worn by the user U and ending from the position (Xp, Yp, Zp) in the real space RS of the photodetector 12 that detected the light L irradiated from the head mounted display 3. Note that while the gaze direction identification process S13 is being performed for a certain user U, the light L is irradiated only from the head mounted display 3 worn by the user U (light is not irradiated from the head mounted display 3 worn by other users U).
[0028] The gaze direction identification process S13 is composed of, for example, an intensity comparison process S131, a photodetector position read process S132, a head mounted display position calculation process S133, and a gaze direction calculation process S134. The intensity comparison process S131 is a process for comparing the intensities of light detected by each photodetector 12 when light L is irradiated from the head mounted display 3 worn by the user U, and identifying the photodetector 12 that detected the light with the highest intensity. The photodetector position read process S132 is a process for reading from a memory the position (Xp, Yp, Zp) in the real space RS of the photodetector 12 identified by the intensity comparison process. The head mounted display position calculation process S133 is a process for calculating the position (Xh, Yh, Zh)=(X, Y, H) of the head mounted display 3 worn by the user U in the real space RS based on the position (X, Y) specified in the position specification process S11 and the posture P specified in the posture specification process S12 described above. Here, X and Y are the X coordinate and the Y coordinate of the position (X, Y) specified in the position specification process S11, respectively, and H is the height of the head mounted display 3 determined according to the posture P specified in the posture specification process S12. For example, when the posture P of the user U is a sitting posture P1, H=80 cm, and when the posture P of the user U is a standing posture P2 (either a standing posture P21, a left foot weighted posture P22L, or a right foot weighted posture P22R), H=160 cm. The gaze direction calculation process S134 is a process for calculating the gaze direction (Vx, Vy, Vz) = (Xp-Xh, Yp-Yh, Zp-Zh) of the user U by subtracting the position (Xh, Yh, Zh) calculated in the head mounted display position calculation process S133 from the position (Xp, Yp, Zp) read out in the photodetector position reading process S132, component by component.
[0029] The signal providing process S14 is a process of generating a signal Σ representing the position (X, Y) of each user U identified in the position identification process S11, the posture P of each user U identified in the posture identification process S12, and the gaze direction (Vx, Vy, Vz) of each user P identified in the gaze direction identification process S13, and providing the generated signal Σ to the simulation device 2.
[0030] (Configuration of the coexistence experience provision system) The configuration of the coexistence experience providing system S according to one embodiment of the present invention will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the coexistence experience providing system S.
[0031] As shown in Fig. 3, the coexistence experience providing system S includes a simulation device 2 and a head mounted display 3 in addition to the above-mentioned interface device 1. The simulation device 2 includes a processor 21 and a memory 22. In Fig. 3, "head mounted display" is abbreviated as "HMD". The interface device 1 and the simulation device 2 are configured to be able to communicate with each other via wireless communication or wired communication. In addition, the simulation device 2 and each head mounted display 3 are configured to be able to communicate with each other via wireless communication.
[0032] As described above, the interface device 1 generates a signal Σ indicating the position (X, Y), posture P, and gaze direction (Vx, Vy, Vz) of each user U, and provides the generated signal Σ to the simulation device 2. The simulation device 2 acquires this signal Σ from the interface device 1. Then, the simulation device 2 generates an image M representing the virtual space VS seen by each user U based on this signal Σ, and provides the generated image M to the head mounted display 3 worn by the user U. The process of generating the image M representing the virtual space VS seen by each user U based on the signal Σ is executed by the processor 21 according to instructions of the program stored in the memory 22. The head mounted display 3 acquires this image M from the simulation device 2. Then, the head mounted display 3 displays this image M.
[0033] The signal Σ provided from the interface device 1 to the simulation device 2 includes information representing the position (X, Y) and posture P of each user U. Therefore, the simulation device 2 can generate a virtual space VR including an avatar corresponding to each user U, which accurately reflects the position (X, Y) and posture P of the user U in the real space RS. In addition, the signal Σ provided from the interface device 1 to the simulation device 2 includes information representing the position (X, Y) and line of sight (Vx, Vy, Vz) of each user U. Therefore, it is possible to generate an image M of the generated virtual space VR as seen from each user U, which accurately reflects the position (X, Y) and line of sight (Vx, Vy, Vz) of the user U in the real space RS. This allows multiple users U in the real space RS to feel as if they are coexisting in the virtual space VR.
[0034] An example of an application of the coexistence experience providing system S is a shooting game. In this case, the coexistence experience providing system S further includes an imitation gun. When a user U pulls the trigger of the imitation gun, the simulation device 2 calculates the trajectory of a bullet virtually shot from the imitation gun. Then, the simulation device 2 processes the image M to be provided to the head mounted display U worn by each user U based on the calculated trajectory. As an example, the image M to be provided to the user U who has been shot (a user U located on the calculated trajectory) is darkened, and a message indicating that the user has been shot is displayed. Also, in the image M to be provided to the other users U, the avatar corresponding to the user U who has been shot is erased.
[0035] (Modification of the coexistence experience provision system) A modified example of the coexistence experience providing system S shown in Fig. 3 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the coexistence experience providing system S' according to this modified example.
[0036] The coexistence experience providing system S' according to this modified example is obtained by adding another interface device 1' different from the interface device 1 to the coexistence experience providing system S shown in FIG. 3. The other interface device 1' is configured similarly to the interface device 1 and has similar functions to the interface device 1. The interface devices 1, 1' and the simulation device 2 are configured to be able to communicate with each other via wireless communication or wired communication. Furthermore, the simulation device 2 and each head mounted display 3 are configured to be able to communicate with each other via wireless communication.
[0037] The interface device 1 generates a signal Σ indicating the position (X, Y), posture P, and gaze direction (Vx, Vy, Vz) of each user U in the real space RS, and provides the generated signal Σ to the simulation device 2. Another interface device 1' generates a signal Σ' indicating the position (X, Y), posture P, and gaze direction (Vx, Vy, Vz) of each user U' in another real space RS' different from the real space RS, and provides the generated signal Σ' to the simulation device 2. The simulation device 2 acquires these signals Σ, Σ' from the interface devices 1, 1'. Then, the simulation device 2 generates images M, M' representing the virtual space VS seen from each user U, U' based on these signals Σ, Σ', and provides the generated images M, M' to the head mounted display 3 worn by the users U, U'. The head mounted display 3 acquires this image M from the simulation device 2. Then, the head mounted display 3 displays this image M.
[0038] The coexistence experience providing system S' according to this modified example can provide users U, U' who are in different real spaces RS, RS' with the experience of coexisting in the same virtual space VR.
[0039] (summary) The interface device of aspect 1 of the present invention is an interface device for providing multiple users with an experience of coexistence in a virtual space, and includes a plurality of force plates laid out on the floor of a real space, and an output unit that provides, for each user, an output signal from the force plates detecting the user's foot pressure, and a signal representing the user's position on the floor and the user's posture determined based on the position of the force plates in the real space, to a simulation device that generates the virtual space.
[0040] An interface device according to aspect 2 of the present invention is the interface device according to aspect 1, further comprising a plurality of photodetectors distributed on the walls of the real space for detecting light irradiated from a head-mounted display worn by each user, and the signal further represents the gaze direction of each user determined based on the output signals of the plurality of photodetectors.
[0041] A coexistence experience providing system according to aspect 3 of the present invention includes an interface device according to aspect 1 or 2 and the simulation device, wherein the simulation device generates an image of the virtual space as seen by each user, including avatars of other users, based on the signal, and supplies the generated image to a head-mounted display worn by the user.
[0042] A coexistence experience providing system according to aspect 4 of the present invention is a coexistence experience providing system according to aspect 3, in which the simulation device acquires other signals representing the position and posture of each user in another real space different from the real space from another interface device different from the interface device, and the simulation device generates an image of the virtual space including avatars of the other users in the real space and avatars of the other users in the other real space based on the signals and the other signals, as seen from each user in the real space or the other real space, and supplies the generated image to a head-mounted display worn by the user.
[0043] A coexistence experience providing system according to aspect 5 of the present invention is a coexistence experience providing system according to aspect 3, further including an imitation gun carried by each user, wherein the simulation device calculates the trajectory of a bullet virtually fired from the imitation gun, and processes an image to be supplied to a head-mounted display worn by each user based on the calculated trajectory. [Explanation of symbols]
[0044] S: Coexistence experience providing system, 1: Interface device, 11: Force plate, 12: Photodetector, 13: Output unit, 2: Simulation device, 21: Processor, 22: Memory, 3: Head mounted display, U: User
Claims
1. An interface device for providing a coexistence experience in a virtual space to a plurality of users, comprising: Multiple force plates are laid out on the floor of the real space, and an output unit that provides, for each user, an output signal of the force plate that detects the foot pressure of the user, and a signal that represents the position of the user on the floor and the posture of the user, which are specified based on the position of the force plate in the real space, to a simulation device that generates the virtual space.
1. An interface device comprising:
2. The present invention further includes a plurality of photodetectors that are distributed on walls of the real space and detect light emitted from a head mounted display worn by each user, the signal further represents, for each user, a gaze direction of the user determined based on the output signals of the plurality of photodetectors.
2. The interface device according to claim 1 .
3. An interface device according to claim 1 or 2; The simulation device, the simulation device generates an image of the virtual space including the avatars of the other users as seen by each user based on the signal, and supplies the generated image to a head mounted display worn by the user; A coexistence experience providing system.
4. the simulation device acquires, from an interface device different from the interface device, another signal representing a position and a posture of each user in another real space different from the real space; the simulation device generates an image of the virtual space, including avatars of other users in the real space and avatars of other users in the other real spaces, as viewed from each user in the real space or the other real space, based on the signal and the other signal, and supplies the generated image to a head mounted display worn by the user; 4. The coexistence experience providing system according to claim 3.
5. Further including a replica gun possessed by each user; the simulation device calculates a trajectory of a bullet virtually fired from the imitation gun, and processes an image to be supplied to a head mounted display worn by each user based on the calculated trajectory; 4. The coexistence experience providing system according to claim 3.
Citation Information
Patent Citations
Virtual environment construction device, video presentation device, model learning device, optimum depth determination device, method therefor, and program
WO2017131071A1
