Environment generating method, environment generating system, environment generating device, program, and recording medium

Abstract

The present invention provides an environment generation method capable of calibrating predetermined parameters of virtual objects in a virtual space or a mixed reality space under appropriate conditions. The environment generation method generates a virtual space VS, avatars A1 and A2, and virtual objects VO1 to VO3. Furthermore, when a calibration process has not been performed, a virtual space environment is generated using the virtual space VS, avatars A1 and A2, and virtual objects VO1 to VO3. When a calibration process has been performed, a virtual space environment that is recognized by a user is generated using the virtual space VS, avatars A1 and A2, and calibrated VO1 to VO3.

Description

[Technical Field]

[0001] The present invention relates to an environment generation method for generating an environment in a virtual space and a mixed reality space. [Background technology]

[0002] A conventional method for generating an environment in a virtual space is described in Patent Document 1. This environment generation method generates a virtual environment in which an avatar exists and allows a user to recognize the virtual environment via a head-mounted device. In this environment generation method, when the facial expression of the avatar in the virtual space becomes distorted due to misalignment of the head-mounted device or the like, calibration of the avatar's facial expression is performed so that the avatar's facial expression returns to a normal state. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-114036 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, when generating a virtual space environment corresponding to a real space, a method has been used in which virtual objects are generated in the virtual space environment corresponding to real objects in the real space. With this method, a situation may arise in which the coordinates, orientation, and shape of a real object in the real space do not match the coordinates, orientation, and shape of a virtual object in the virtual space. If such a situation occurs, for example, when playing a game using a virtual space, the progress of the game may be hindered. For these reasons, it is desirable to perform calibration of parameters such as the coordinates, orientation, and shape of a virtual object (hereinafter referred to as "predetermined parameters") under appropriate conditions.

[0005] In contrast, the conventional environment generation method described above can calibrate the facial expressions of an avatar in a virtual space, but has the problem that it cannot calibrate predetermined parameters of a virtual object under appropriate conditions. This problem occurs not only when generating a virtual space, but also when generating a mixed reality space in which virtual objects are placed in a real space.

[0006] The present invention has been made to solve the above-mentioned problems, and aims to provide an environment generation method that can perform calibration of specified parameters of a virtual object in a virtual space or a mixed reality space under appropriate conditions. [Means for solving the problem]

[0007] In order to achieve the above object, the invention of claim 1 is an environment generation method executed by a computer system and generating a virtual space environment to be recognized by a user via an environment output device, the method comprising: a virtual space generation step in which a virtual space generation unit generates a virtual space to correspond to a real space in which the user exists; an avatar generation step in which an avatar generation unit generates an avatar in the virtual space corresponding to the user; a virtual object generation step in which a virtual object generation unit generates a virtual object in the virtual space to correspond to an actual object existing in the real space; a user state recognition step in which a user state recognition unit recognizes movement of coordinates and changes in posture of the user; an avatar state control step in which an avatar state control unit controls movement of coordinates and changes in posture of the avatar based on the movement of coordinates and changes in posture of the user; A virtual object has already been generated in the virtual space, and When an execution condition is met that the user's coordinates are located within a predetermined range determined based on the coordinates of a predetermined trigger object existing in real space, thea calibration processing step of executing a calibration process for making a first predetermined parameter, which is at least one of coordinates, posture, and shape of a virtual object, correspond to a second predetermined parameter, which is at least one of coordinates, posture, and shape of a real object; and an environment generation unit, when the calibration process has not been executed, generating an environment of a virtual space to be recognized by a user using a virtual space, an avatar, and a virtual object, and when the calibration process has been executed, generating an environment of the virtual space to be recognized by a user using the virtual space, an avatar, and a calibration Processing is performed and an environment generating step of generating a virtual space environment that the user can recognize using the virtual objects generated.

[0008] According to this environment generation method, a virtual space is generated to correspond to a real space in which a user exists, an avatar corresponding to the user is generated in the virtual space, and virtual objects are generated in the virtual space to correspond to real objects existing in the real space. Furthermore, based on the movement of the user's coordinates and changes in the user's posture, the movement of the avatar's coordinates and changes in the user's posture are controlled. When an execution condition is met in which the user's coordinates are located within a predetermined range determined based on the coordinates of a predetermined trigger object existing in the real space, a calibration process is executed to match a first predetermined parameter of the virtual object with a second predetermined parameter, which is at least one of the coordinates, posture, and shape of the real object. When the calibration process has not been executed, an environment in the virtual space that the user perceives is generated using the virtual space, the avatar, and the virtual objects. When the calibration process has been executed, an environment in the virtual space that the user perceives is generated using the virtual space, the avatar, and the calibrated virtual objects.

[0009] In this way, by performing a calibration process to match the first predetermined parameter of the virtual object with the second predetermined parameter of the real object under the condition that the user's coordinates are located within a predetermined range, the relationship between the user and the real object can be appropriately matched with the relationship between the avatar and the virtual object. For example, it is possible to prevent a discrepancy between the timing when the user moves toward and reaches the real object and the timing when the user's avatar reaches the virtual object.

[0010] Therefore, by appropriately setting a predetermined trigger object, it is possible to execute a calibration process for the first predetermined parameter of a virtual object under the appropriate condition that the user's coordinates are located within a predetermined range. Note that in this specification, the "calibration process for matching the first predetermined parameter of a virtual object with the second predetermined parameter of a real object" is not limited to a process for matching the first predetermined parameter of a virtual object with the second predetermined parameter of a real object, but also includes a process for adjusting the first predetermined parameter of a virtual object to a state that approaches the second predetermined parameter of a real object.

[0011] In the present invention, the trigger object is preferably a real object.

[0012] According to this environment generation method, since the trigger object is a real object, a calibration process can be performed to correspond a first predetermined parameter of the virtual object to a second predetermined parameter of the real object under the appropriate condition that the user's coordinates are located within a predetermined range determined based on the coordinates of the real object.

[0013] In the present invention, it is preferable that the users include a first user and a second user, the trigger object is one of the first user and the second user, and the execution condition is that the coordinates of the other of the first user and the second user are located within a predetermined range determined based on the coordinates of one of the first user and the second user.

[0014] According to this environment generation method, a calibration process can be performed to match a first predetermined parameter of a virtual object to a second predetermined parameter of a real object under appropriate conditions in which the coordinates of the other of the first user and the second user are located within a predetermined range determined based on the coordinates of one of the first user and the second user.

[0015] In the present invention, in the calibration process, it is preferable that a first predetermined parameter of the virtual object is calibrated so as to correspond to a second predetermined parameter of the real object, and at least one of the posture, shape, and relative position of the virtual space with respect to the real space is calibrated based on the calibration of the first predetermined parameter of the virtual object.

[0016] According to this environment generation method, a first predetermined parameter of the virtual object is calibrated so that the first predetermined parameter of the virtual object corresponds to a second predetermined parameter of the real object, and at least one of the posture, shape, and relative position of the virtual space with respect to the real space is calibrated accordingly. This makes it possible to prevent a deviation from occurring between the positional relationship between the avatar, the virtual object, and the virtual space and the positional relationship between the user, the real object, and the real space when the first predetermined parameter of the virtual object is calibrated.

[0017] In the present invention, it is preferable that the real space recognition unit further includes a step of recognizing the posture and shape of the real space, and that the calibration process calibrates at least one of the posture, shape, and relative position of the virtual space with respect to the real space based on the calibration of a first predetermined parameter of the virtual object and the posture and shape of the real space.

[0018] According to this environment generation method, when a first predetermined parameter of a virtual object is calibrated, at least one of the posture, shape, and relative position of the virtual space with respect to the real space can be calibrated in accordance with the calibration and the posture and shape of the real space. This makes it possible to more accurately prevent a deviation from occurring between the positional relationship between the avatar, virtual object, and virtual space and the positional relationship between the user, real object, and real space when the first predetermined parameter of the virtual object is calibrated.

[0019] In this specification, "at least one of the posture, shape, and relative position of the virtual space with respect to the real space is calibrated" is not limited to calibrating at least one of the posture and shape of the virtual space to match at least one of the posture and shape of the real space, and / or calibrating the relative position of the virtual space with respect to the real space to match a position in the real space, but also includes adjusting at least one of the posture and shape of the virtual space to approach at least one of the posture and shape of the real space, and / or adjusting the relative position of the virtual space with respect to the real space to approach a position in the real space.

[0020] In the present invention, in the calibration processing step, when the user's coordinates are moving closer to the coordinates of a predetermined trigger object, it is preferable that the predetermined range is determined to be a larger range the faster the speed of movement of the user's coordinates is.

[0021] According to this environment generation method, when the user's coordinates are moving closer to the coordinates of a predetermined trigger object, the faster the speed of the user's coordinates is, the larger the predetermined range is determined to be. As a result, when the user moves faster toward the predetermined trigger object, which in turn causes the avatar to move faster, the relationship between the user and real objects can be quickly adapted to the relationship between the avatar and virtual objects.

[0022] In order to achieve the above-mentioned object, another environment generation method of the present invention is an environment generation method that is executed by a computer system and generates an environment in a mixed reality space that is recognized by a user via an environment output device, the environment generation method including: a real space recognition step in which a real space recognition unit recognizes the posture and shape of the real space; a virtual object generation step in which a virtual object generation unit generates a virtual object so that it corresponds to an actual object existing in the real space; a user state recognition step in which a user state recognition unit recognizes the coordinates and posture of the user; and a calibration processing unit A virtual object has already been generated in the mixed reality space, and When an execution condition is met that the user's coordinates are located within a predetermined range determined based on the coordinates of a predetermined trigger object existing in real space, the a calibration processing step of executing a calibration process for making a first predetermined parameter, which is at least one of coordinates, posture, and shape of a virtual object, correspond to a second predetermined parameter, which is at least one of coordinates, posture, and shape of a real object; and an environment generation step of generating an environment of a mixed reality space to be recognized by a user using the real space and the virtual object when the calibration process has not been executed, and generating an environment of a mixed reality space to be recognized by a user when the calibration process has been executed. Processing is performed and an environment generating step of generating an environment of a mixed reality space to be recognized by the user using the virtual object generated. 。

[0023] According to this environment generation method, a real space is recognized, and virtual objects are generated to correspond to real objects existing in the real space. Furthermore, when an execution condition is satisfied that the user's coordinates are located within a predetermined range determined based on the coordinates of a predetermined trigger object existing in the real space, a calibration process is executed to cause a first predetermined parameter, which is at least one of the coordinates, orientation, and shape of the virtual object in the mixed reality space, to correspond to a second predetermined parameter, which is at least one of the coordinates, orientation, and shape of the real object. If the calibration process has not been executed, a mixed reality space environment to be perceived by the user is generated using the real space and the virtual objects. If the calibration process has been executed, a mixed reality space environment to be perceived by the user is generated using the real space and the calibrated virtual objects.

[0024] In this way, by performing a calibration process so that the first predetermined parameter of the virtual object corresponds to the second predetermined parameter of the real object under the condition that the user's coordinates are located within a predetermined range, the relationship between the user and the real object can be made to correspond to the relationship between the user and the virtual object. For example, it is possible to prevent a discrepancy between the timing at which the user moves toward and reaches the real object and the timing at which the user reaches the virtual object. Therefore, by appropriately setting a predetermined trigger object, it is possible to perform calibration of the first predetermined parameter of the virtual object under the appropriate condition that the user's coordinates are located within a predetermined range.

[0025] In order to achieve the above-mentioned object, the environment generating system of the present invention is characterized by being configured to execute any one of the above-mentioned environment generating methods.

[0026] In order to achieve the above-mentioned object, an environment generating device of the present invention is characterized by being configured to execute any one of the above-mentioned environment generating methods.

[0027] In order to achieve the above-mentioned object, the present invention provides a program for causing a computer system to execute any one of the above-mentioned environment generation methods.

[0028] In order to achieve the above object, a recording medium according to the present invention is characterized in that the above program is recorded thereon and the program is readable by a computer system. [Brief explanation of the drawings]

[0029] [Figure 1] 1 is a perspective view showing the configuration of an environment generating system for executing an environment generating method according to a first embodiment of the present invention, a real space, etc.; [Figure 2] FIG. 2 is a block diagram showing the functional configuration of a server. [Figure 3] FIG. 2 is a perspective view showing a virtual space environment generated corresponding to the real space environment of FIG. 1. [Figure 4] 10 is a flowchart showing a control process executed by the server. [Figure 5] FIG. 2 is a plan view showing an example of a state of real space. [Figure 6] FIG. 6 is a plan view showing a state of a virtual space generated corresponding to the real space of FIG. 5. [Figure 7] FIG. 10 is a plan view showing the state of real space when the execution conditions for the calibration process are met. [Figure 8] 8 is a plan view showing the state of a virtual space generated corresponding to the real space of FIG. 7 when a calibration process is executed. FIG. [Figure 9] FIG. 10 is a block diagram showing the functional configuration of a server in an environment generating system according to a second embodiment of the present invention. [Figure 10] 10 is a flowchart showing a control process executed by a server according to a second embodiment. [Figure 11] FIG. 10 is a plan view showing the state of mixed reality space before the execution of a calibration process. [Figure 12]FIG. 10 is a plan view showing the state of the mixed reality space after the calibration process is executed. Modes for carrying out the invention

[0030] An environment generating method and an environment generating system for executing the same according to a first embodiment of the present invention will be described below with reference to the drawings.

[0031] As shown in Fig. 1, the environment generation system S (hereinafter referred to as "system S") of this embodiment is designed to allow a first user U1 and a second user U2, who are both present in a predetermined area (e.g., a single room) in a real space RS, to experience a virtual space VS. In the following description, the first user U1 and the second user U2 will be collectively referred to as "users U."

[0032] Here, "virtual space" refers to a virtual space that the user perceives instead of the real space in which the user exists, a so-called virtual reality space (VR space), in which virtual objects, avatars that move in response to the user's actions, etc. are placed.

[0033] In this embodiment, for ease of understanding, there are two users, a first user U1 and a second user U2, but the number of users using the environment generation system of the present invention is not limited to such a configuration and may be one, or three or more.

[0034] First, the schematic configuration of the system S will be described with reference to Fig. 1. As shown in Fig. 1, the system S includes a plurality of signs 1 attached to a user U present in a real space RS, a camera 2 that captures images of the user U (strictly speaking, the signs 1 attached to the user U), a server 3 that determines the environment of a virtual space VS that the user U will experience, and a head-mounted display 4 (hereinafter referred to as "HMD4") that allows the user to recognize the determined environment.

[0035] In this embodiment, the server 3 corresponds to a computer system, and the head-mounted display 4 corresponds to an environment output device.

[0036] In the system S, the camera 2, server 3, and HMD 4 can transmit and receive information to and from each other wirelessly via the Internet network, public lines, short-range wireless communication, etc. However, any of them may be configured to transmit and receive information to and from each other via a wired connection.

[0037] The multiple markers 1 are attached to the head, both hands, and both feet of the user U via the HMD 4, gloves, and shoes worn by the user U. The multiple markers 1 are used to recognize the movements of the user U in the real space RS, as will be described later. Therefore, the positions at which the markers 1 are attached, the number of markers 1, etc. may be changed as appropriate depending on the other devices that make up the system S.

[0038] The camera 2 is installed so as to be able to capture images from multiple directions of the user U's movable range (that is, the range in which the user U can move and act) in the real space RS in which the user U exists.

[0039] The server 3 recognizes (e.g., detects or calculates) the sign 1 from the image captured by the camera 2, and recognizes the user U's actions (e.g., changes in posture and coordinates) based on the position of the recognized sign 1 in the real space RS. Furthermore, the server 3 determines the environment of the virtual space VS that the user U will recognize based on the actions.

[0040] The HMD 4 outputs and allows the user U to recognize the environment of the virtual space VS (for example, images and sounds), and is worn on the head of the user U. As shown in FIG. 2, the HMD 4 has a monitor 40 and a speaker 41. The monitor 40 is configured to allow the user U to recognize the image of the virtual space VS determined by the server 3, and the speaker 41 is configured to allow the user U to recognize the sound of the virtual space VS determined by the server 3.

[0041] When a user U experiences a virtual space VS using the system S, the user U recognizes that he or she is present in the virtual space VS by perceiving only the images and sounds of the virtual space VS via the HMD 4. In other words, the system S is configured as a so-called immersive system.

[0042] The environment generation system of the present invention is not limited to a configuration using signs and cameras as described above (so-called motion capture devices), but may be configured to recognize real space and the user's movements.

[0043] Therefore, for example, the number and arrangement of signs and cameras may be different from those shown in Fig. 1. Specifically, in order to recognize at least one of the user's feature points and the real-space feature points, signs may be attached not only to the user but also to real objects existing in the real space. Feature points may also be recognized from the image itself without using signs.

[0044] Alternatively, instead of signs and cameras, the HMD may be equipped with a sensor such as a GPS, and the user's movements may be recognized based on the output of the sensor. Such a sensor may also be used in combination with the motion capture device described above.

[0045] Furthermore, the environment generating system of the present invention is not limited to being composed of one server, but rather any of the devices constituting the environment generating system may be configured to have the processing unit described below.

[0046] Therefore, for example, the entire environment generating system may be configured with multiple servers. Also, at least one of the processing units or at least part of the functions of that processing unit may be implemented in a camera, HMD, or other device, and the environment generating system may be configured with the cooperation of those devices and the server, or with only those devices.

[0047] Next, we will explain the configuration of the system S. The system S is made up of a camera 2, a server 3, and an HMD 4, and these camera 2, server 3, and HMD 4 each have one or more electronic circuit units including a CPU, RAM, ROM, interface circuitry, etc.

[0048] As shown in FIG. 2, the server 3 has functions as a real space recognition unit 30, an environment generation unit 31, a user state recognition unit 32, and an avatar state control unit 33, and these functions are realized by at least one of a hardware configuration and a program implemented in the server 3.

[0049] The real space recognition unit 30 recognizes image data of the real space RS captured by the camera 2, and recognizes the situation of the real space RS based on the image data. The situation of the real space RS is, for example, the posture and coordinates of the user U and real objects (in this embodiment, the box RO1, bookshelf RO2, and drawer RO3 shown in FIG. 1) that exist in the real space RS. In the following description, the box RO1, bookshelf RO2, and drawer RO3 are collectively referred to as "real objects RO."

[0050] The environment generation unit 31 generates the virtual environment VS that the user U perceives via the monitor 40 and speaker 41 of the HMD 4. In this case, the "environment to be perceived" by the user U refers to the environment of the virtual space VS that the user U experiences with his or her five senses. For example, this environment is an environment that is configured by images of virtual objects that exist in the virtual space VS that the user U perceives, sounds that are generated based on the virtual objects, and the like.

[0051] The environment generation unit 31 includes a virtual space generation unit 31a, an avatar generation unit 31b, a virtual object generation unit 31c, and a calibration processing unit 31d.

[0052] The virtual space generation unit 31a generates a virtual space VS corresponding to the real space RS (see FIG. 1) in which the user U exists, as shown in FIG. 3, based on the recognition result of the real space recognition unit 30. Specifically, the virtual space generation unit 31a generates images that serve as the background of the virtual space VS and avatars that exist in the virtual space VS, and sounds associated with these images.

[0053] Although the system S of this embodiment does not have such a feature, if the environment generation system has a feature that realizes a predetermined feel (for example, a cushion with variable hardness) or a feature that generates a predetermined smell, the virtual space generation unit may generate the virtual space VS using these feel and smell in addition to images and sounds.

[0054] 3, the shape of the virtual space VS generated by the virtual space generation unit 31a is configured to be generated according to the shape of the real space RS. Here, the "shape of the real space" is recognized based on the shape of the area in which the virtual space VS is generated, and the coordinates, posture, shape, etc. of the real object RO existing in that real space.

[0055] In an immersive system such as this embodiment, the "shape of the virtual space VS" refers to the area in which the avatar corresponding to the user can move. Specifically, it is the area defined by the background image of the virtual space VS and the virtual objects placed in the virtual space VS.

[0056] The avatar generation unit 31b generates an avatar to be present in the virtual space VS based on the posture and coordinates of the user U recognized by the user state recognition unit 32, which will be described later.

[0057] The avatars include a first avatar A1 corresponding to a first user U1 and a second avatar A2 corresponding to a second user U2 (see FIG. 3). The first avatar A1 and the second avatar A2 move in the virtual space VS in accordance with the movements of the corresponding user U in the real space RS. Hereinafter, the first avatar A1 and the second avatar A2 will be collectively referred to as "avatar A."

[0058] In this embodiment, when the system S is used to experience the virtual space VS, the user U perceives only the images and sounds of the virtual space VS and is made to perceive that the user U himself / herself exists in the virtual space VS. In other words, the system S is configured as a so-called immersive system.

[0059] The virtual object generation unit 31c generates virtual objects (in this embodiment, a treasure chest VO1, a pillar VO2, and a rock wall VO3 shown in FIG. 3) that correspond to the coordinates, posture, and shape of the real object RO and are recognized by the user U. In the following description, the treasure chest VO1, the pillar VO2, and the rock wall VO3 are collectively referred to as the "virtual object VO."

[0060] As described below, the calibration processing unit 31d executes the calibration process when the execution condition is met, and the execution condition for this calibration process is that at least one of the following conditions (c1) and (c2) is met:

[0061] (c1) The coordinates of at least one of the first user U1 and the second user U2 are located within a first predetermined range X1 (the range indicated by the dashed lines in Figures 5 and 7) set based on the coordinates of the box RO1. (c2) The coordinates of the first user U1 are located within a second predetermined range X2 (the range indicated by the two-dot chain line in FIGS. 5 and 7) that is set based on the coordinates of the second user U2.

[0062] In this case, the coordinates of the first user U1, the second user U2, and the box RO1 are set as two-dimensional coordinates with the floor of the real space as the coordinate plane. Furthermore, when the first user U1 or the second user U2 is moving so that the coordinates of the first user U1 or the second user U2 approach the coordinates of the box RO1, the first predetermined range X1 is determined to be a larger range as the speed of movement of the coordinates of the first user U1 or the second user U2 is faster.

[0063] Furthermore, in the case of the above condition (c2), the second predetermined range X2 is set based on the coordinates of the second user U2, but instead, the second predetermined range X2 may be set based on the coordinates of the first user U1, in which case the condition that "the coordinates of the second user U2 are located within the second predetermined range X2" can be used in place of condition (c2).

[0064] In the calibration process, when the above execution conditions are met, that is, when at least one of the conditions (c1) and (c2) is met, a calibration value is first calculated, for example, by the method described below. That is, a plurality of points at which the positions of the real space RS and the virtual space VS are calibrated are set in the real space RS as a plurality of calibration points, and the position sensed by a sensing device (for example, VR goggles) and the calibration value at each calibration point are stored in advance in the server 3.

[0065] Then, based on each point distance, which is the distance between the user U and each calibration point, a calibration value is calculated by performing a weighting calculation for each point distance on the calibration value of each calibration point and the default calibration value.

[0066] Next, using these calibration values, the first predetermined parameters of the virtual object VO are calibrated so that the first predetermined parameters, which are the coordinates, attitude, and shape of the virtual object VO, correspond to the second predetermined parameters, which are the coordinates, attitude, and shape of the real object RO. Furthermore, based on the calibration of the first predetermined parameters and the attitude and shape of the real space RS, the attitude, shape, and relative position of the entire virtual space VS with respect to the real space RS are calibrated. For example, the virtual space VS is calibrated from the state shown in FIG. 6 (described later) to the state shown in FIG. 8 (described later).

[0067] In the calibration process, the first predetermined parameter may be set to at least one of the coordinates, orientation, and shape of the virtual object VO, and the second predetermined parameter may be set to at least one of the coordinates, orientation, and shape of the real object RO. In this case, it is sufficient that the first predetermined parameter and the second predetermined parameter are set to correspond to each other. For example, if the first predetermined parameter is set to the coordinates of the virtual object VO, the second predetermined parameter may be set to the coordinates of the real object RO.

[0068] Furthermore, in the calibration process, at least one of the posture, shape, and relative position of the entire virtual space VS with respect to the real space RS may be calibrated based on the calibration of the first predetermined parameter and the posture and shape of the real space RS, and at least one of the posture, shape, and relative position of the entire virtual space VS with respect to the real space RS may be calibrated based on the calibration of the first predetermined parameter.

[0069] Furthermore, the first predetermined range X1 and the second predetermined range X2 are not limited to circular ranges as shown in FIGS. 5 and 7, but may be elliptical ranges, polygonal ranges, or ranges surrounded by curved lines.

[0070] In the environment generation unit 31, when the above-mentioned calibration process has not been performed, the virtual environment VS is generated using the virtual object VO, avatar A, and virtual space VS generated as described above. On the other hand, when the above-mentioned calibration process has been performed, the virtual environment VS is generated using the calibrated virtual object VO, the calibrated virtual space VS, and avatar A.

[0071] In addition, in the above-mentioned calibration process, it is also possible to configure the configuration so that only the calibration of the first predetermined parameter of the virtual object VO is performed. In that case, the environment generation unit 31 is configured to generate the virtual environment VS using the calibrated virtual object VO, virtual space VS, and avatar A.

[0072] The user state recognition unit 32 recognizes image data of the user U captured by the camera 2 and recognizes the state of the user U in the real space RS based on the image data, and includes a user posture recognition unit 32a and a user coordinate recognition unit 32b. Here, the state of the user U in the real space RS refers to the posture and coordinates of the user U, and further refers to the movement of the user U indicated by the amount of change therein.

[0073] The user posture recognition unit 32a extracts feature points of the user U's body and the like from the input image data of the user U, and recognizes the posture of the user U in the real space RS based on the extraction results.

[0074] The user coordinate recognition unit 32b recognizes the coordinates of the user U in the real space RS based on the results of extraction of feature points such as the body of the user U extracted from the input image data of the user U, and the situation of the real space RS recognized by the real space recognition unit 30 (for example, the coordinates of real objects).

[0075] The avatar state control unit 33 controls the state of the avatar A corresponding to the user U in the virtual space VS based on the state (i.e., posture and coordinates) of the user U in the real space RS recognized by the user state recognition unit 32.

[0076] Next, with reference to FIG. 4, a control process executed by the server 3 of the system S when allowing the user U to experience the virtual space VS will be described.

[0077] In this process, first, a real space recognition process is executed (FIG. 4 / STEP 1). In this real space recognition process, the posture, shape, and other conditions of the real space RS in which the user U exists (see FIG. 1) are recognized. In this embodiment, the real space recognition process corresponds to the real space recognition step.

[0078] Next, a virtual space generation process is executed (FIG. 4 / STEP 2). In this virtual space generation process, a virtual space VS (see FIG. 3) is generated based on the situation of the real space RS, such as the posture and shape. In this embodiment, the virtual space generation process corresponds to the virtual space generation step.

[0079] Next, an avatar generation process is executed (FIG. 4 / STEP 3). In this avatar generation process, an avatar A corresponding to the user U is generated in the virtual space VS. In this embodiment, the avatar generation process corresponds to an avatar generation step.

[0080] After the avatar generation process is executed, a virtual object generation process is executed (FIG. 4 / STEP 4). In this virtual object generation process, a virtual object VO is generated in the virtual space VS so as to correspond to the real object R0. In this embodiment, the virtual object generation process corresponds to the virtual object generation step.

[0081] Next, a user state recognition process is executed (FIG. 4 / STEP 5). In this user state recognition process, a movement of the coordinates and a change in the posture of the user U are recognized. Specifically, a sign 1 is recognized from an image captured by the camera 2, and a movement of the coordinates and a change in the posture of the user U are recognized based on the position of the recognized sign 1 in the real space RS. In this embodiment, the user state recognition process corresponds to the user state recognition step.

[0082] After the user state recognition process is executed, an avatar state control process is executed (FIG. 4 / STEP 6). In this avatar state recognition process, the movement of coordinates and changes in posture of avatar A corresponding to user U are controlled based on the movement of coordinates and changes in posture of user U recognized by the above-described user state recognition process. In this embodiment, the avatar state control process corresponds to the avatar state control step.

[0083] Next, it is determined whether the execution conditions for the calibration process are met (FIG. 4 / STEP 7). In this determination, if at least one of the above-mentioned conditions (c1) and (c2) is met, it is determined that the execution conditions for the calibration process are met, and otherwise it is determined that the execution conditions for the calibration process are not met.

[0084] If this determination is negative (FIG. 4 / STEP 7...NO), and the conditions for executing the calibration process are not met, the process proceeds to STEP 9, which will be described later. On the other hand, if this determination is positive (FIG. 4 / STEP 7...YES), and the conditions for executing the calibration process are met, the calibration process is executed (FIG. 4 / STEP 8).

[0085] In this calibration process, a calibration value is calculated using the above-mentioned method, and this calibration value is used to calibrate the first predetermined parameter of the virtual object VO so that it corresponds to the second predetermined parameter of the real object.

[0086] At the same time, the posture, shape, and relative position of the entire virtual space VS with respect to the real space RS are calibrated based on the calibration of the first predetermined parameter and the posture and shape of the real space RS. Note that in this embodiment, the calibration process corresponds to the calibration step.

[0087] After the calibration process is executed, or if the calibration process is not executed because the execution conditions for the calibration process are not met, a virtual environment generation process is executed (FIG. 4 / STEP 9). In this virtual environment generation process, if the calibration process is not executed, the virtual space VS, avatar A, and virtual object VO generated as described above are used to generate the environment of the virtual space VS for the user U to recognize. On the other hand, if the calibration process is executed, the environment of the virtual space VS is generated using the calibrated virtual space VS, avatar A, and calibrated virtual object VO. In this embodiment, the virtual environment generation process corresponds to the virtual environment generation step.

[0088] Next, a virtual environment output process is executed (FIG. 4 / STEP 10). In this virtual environment output process, a signal for causing the user U to recognize the environment of the virtual space VS generated as described above is output to the HMD 4. As a result, the environment of the virtual space VS generated as described above is recognized by the user U.

[0089] Next, the effect of the calibration process when the above control process is executed will be described with reference to Figures 5 to 8. For example, when the environment of the real space RS is in the state shown in Figure 5, the environment of the virtual space VS experienced by the user U through the HMD 4 is in the state shown in Figure 6. In the environment of the virtual space VS shown in Figure 6, the orientation of the virtual object VO relative to the avatar A is misaligned with the orientation of the real object RO relative to the user U.

[0090] When this posture shift occurs and the first user U1 moves from the position shown in Figure 5 to the position shown in Figure 7, and the coordinates of the first user U1 are located within the first predetermined range X1, the above-mentioned calibration process is executed, and as shown in Figure 8, in the environment of the virtual space VS, the posture of the virtual object VO relative to the avatar A is calibrated to match the posture of the real object RO relative to the user U.

[0091] That is, by performing the calibration process, for example, the orientation of the real object RO relative to the user U can be made to match the orientation of the virtual object VO relative to the avatar A. At the same time, the orientation, shape, and relative position of the entire virtual space VS relative to the real space RS are calibrated. As a result, in the environment of the virtual space VS, the positional relationship of the avatar A relative to the virtual object VO and the virtual space VS can be made to match the positional relationship of the real object RO and the real space RS relative to the user U.

[0092] As described above, according to the environment generation method of the first embodiment, a virtual space VS is generated to correspond to a real space RS in which a user U exists, an avatar A corresponding to the user U is generated in the virtual space VS, and a virtual object VO is generated to correspond to an actual object RO existing in the real space RS. Furthermore, the movement of the coordinates and changes in the posture of the avatar A are controlled based on the movement of the coordinates and changes in the posture of the user U.

[0093] Then, when the execution conditions for the calibration process are met, i.e., when at least one of the above-mentioned conditions (c1) and (c2) is met, the calibration process is executed. In this calibration process, the first predetermined parameters of the virtual object VO are calibrated so that they correspond to the second predetermined parameters of the real object RO, and the posture, shape, and relative position of the entire virtual space VS with respect to the real space RS are calibrated based on the calibration of the first predetermined parameters and the posture and shape of the real space RS. This makes it possible to prevent the user U from feeling uncomfortable with the positional relationship of the virtual object VO and the virtual space VS with respect to the avatar A when the first predetermined parameters of the virtual object VO are calibrated.

[0094] Furthermore, the first predetermined range X1 in condition (c1) is determined to be larger as the speed of movement of the coordinates of the first user U1 or the second user U2 increases when the first user U1 or the second user U2 is moving so that the coordinates of the first user U1 or the second user U2 approach the coordinates of the box RO1. As a result, when the speed at which the user U moves toward the real object RO1 increases and the speed at which the avatar A moves toward the virtual object VO1 increases, the relationship between the user U and the real object RO1 can be quickly adjusted to the relationship between the avatar A and the virtual object VO1.

[0095] In the first embodiment, the environment generation method is described as being executed by one computer system, but the present invention is not limited to a computer system, and may be any system that executes the environment generation method of the present invention.

[0096] Therefore, for example, the environment generating device may be configured by one computer (server 3) described in the first embodiment. Also, the environment generating device may be a calibration program for causing any one or more computers to execute the environment generating method described above, and a recording medium on which the program is recorded and which is readable by a computer used by a user or the like.

[0097] Next, an environment generating method and an environment generating system for executing the same according to a second embodiment of the present invention will be described. Compared with the environment generating system S of the first embodiment, the environment generating system of this embodiment has the same mechanical and electrical configurations, and only the functional configuration of the server 3 and the content of the control processing are different, so the following description will focus on the differences. Furthermore, the same components as those of the first embodiment will be assigned the same reference numerals, and their description will be omitted.

[0098] In this embodiment, the environment generation system Sx (hereinafter referred to as "system Sx") shown in Fig. 9 allows the user U to experience a mixed reality space MRS (see Figs. 11 and 12). This mixed reality space MRS corresponds to a space in which real objects of the real space RS are superimposed on virtual objects, and in this embodiment, the box RO1 in Fig. 5 is the real object, and the treasure chest VO1 in Figs. 11 and 12 is the virtual object. In the following description, the box RO1 will be referred to as the "real object RO1," and the treasure chest VO1 will be referred to as the "virtual object VO1."

[0099] As shown in FIG. 9, the server 3 of this embodiment has the functions of a real space recognition unit 30, a user state recognition unit 32, and an environment generation unit 35, and these functions are realized by at least one of a hardware configuration and a program implemented in the server 3.

[0100] As described above, the physical space recognition unit 30 recognizes the situation of the physical space RS, and the user state recognition unit 32 recognizes the posture and coordinates of the user U in the physical space RS.

[0101] The environment generation unit 35 generates the environment of the mixed reality space MRS that the user U perceives via the monitor 40 and speaker 41 of the HMD 4, and includes a virtual object generation unit 35a and a calibration processing unit 35b.

[0102] The virtual object generation unit 35a generates a virtual object VO1 (see FIGS. 11 and 12) that is to be recognized by the user U, corresponding to the coordinates, posture, and shape of the real object RO1 (see FIG. 5).

[0103] Furthermore, the calibration processing unit 35b executes the calibration process when the aforementioned execution conditions are met, that is, when at least one of the aforementioned conditions (c1) and (c2) is met. In this calibration process, a calibration value is calculated by the aforementioned method, and the first predetermined parameter of the virtual object VO1 is calibrated using this calibration value.

[0104] In the environment generation unit 35, if the above calibration process has not been performed, the environment of the mixed reality space MRS is generated using the real space RS and the virtual object VO1 generated as described above. On the other hand, if the above calibration process has been performed, the environment of the mixed reality space MRS is generated using the calibrated virtual object VO1.

[0105] Next, with reference to FIG. 10, a control process executed by the server 3 of the system S when allowing the user U to experience the mixed reality space MRS will be described.

[0106] In this process, first, a real space recognition process is executed (FIG. 10 / STEP 21). In this real space recognition process, the situation (see FIG. 1) of the real space RS in which the user U exists, such as the posture and shape, is recognized. In this embodiment, the real space recognition process corresponds to the real space recognition step.

[0107] Next, a virtual object generation process is executed (FIG. 10 / STEP 22). In this virtual object generation process, a virtual object VO1 is generated so as to correspond to the real object R01. In this embodiment, the virtual object generation process corresponds to the virtual object generation step.

[0108] Next, a user state recognition process is executed (FIG. 10 / STEP 23). In this user state recognition process, the movement of the coordinates and the change in the posture of the user U are recognized by the above-mentioned method. In this embodiment, the user state recognition process corresponds to the user state recognition step.

[0109] Thereafter, it is determined whether the execution conditions for the calibration process are met (FIG. 10 / STEP 24). In this determination, if at least one of the above-mentioned conditions (c1) and (c2) is met, it is determined that the execution conditions for the calibration process are met, and otherwise it is determined that the execution conditions for the calibration process are not met.

[0110] If this determination is negative (FIG. 10 / STEP 24...NO), and the conditions for executing the calibration process are not met, the process proceeds to STEP 26, which will be described later. On the other hand, if this determination is positive (FIG. 10 / STEP 24...YES), and the conditions for executing the calibration process are met, the calibration process is executed (FIG. 10 / STEP 25).

[0111] In this calibration process, a calibration value is calculated by the above-mentioned method, and the above-mentioned first predetermined parameter of the virtual object VO1 is calibrated using this calibration value so that it corresponds to the above-mentioned second predetermined parameter of the real object RO1. Note that in this embodiment, the calibration process corresponds to the calibration step.

[0112] After the calibration process is executed, or when the execution conditions for the calibration process are not met, an environment generation process is executed (FIG. 10 / STEP 26). In this environment generation process, if the calibration process described above has not been executed, the environment of the mixed reality space MRS for recognition by the user U is generated using the recognition results of the virtual object VO1 and real space RS generated as described above. On the other hand, when the calibration process is executed, the environment of the mixed reality space MRS is generated using the recognition results of the calibrated virtual object VO1 and real space RS. In this embodiment, the environment generation process corresponds to the environment generation step.

[0113] Next, an environment output process is executed (FIG. 10 / STEP 27). In this environment output process, a signal is output to the HMD 4 to make the user U recognize the environment of the mixed reality space MRS generated as described above.

[0114] As described above, the monitor 40 mounted on the HMD 4 is configured so that the real space RS can be observed through the monitor 40. Therefore, when the above signal is output to the HMD 4, the HMD 4 allows the user U to perceive the real space RS through the monitor 40 and displays the virtual object VO1 so that it is superimposed on the real space RS. At the same time, a mixed reality space MRS environment is generated by generating sound from the speaker 41 mounted on the HMD 4. This allows the user U to perceive the mixed reality space MRS environment generated as described above.

[0115] Next, the effect of the calibration process when the above control process is executed will be described with reference to the above-mentioned Figures 5, 7, and 11 and 12. For example, when the environment of the real space RS is in the state shown in Figure 5, it is assumed that the environment of the mixed reality space MRS that the user U experiences through the HMD 4 is in the state shown in Figure 11. In the environment of the mixed reality space MRS shown in Figure 11, the orientation of the virtual object VO with respect to the user U is shifted from the orientation of the real object RO with respect to the user U.

[0116] When this posture shift occurs and the first user U1 moves from the position shown in Figure 5 to the position shown in Figure 7 and the coordinates of the first user U1 are located within the first predetermined range X1, the above-mentioned calibration process is executed, and as shown in Figure 12, in the mixed reality space MRS environment, the posture of the virtual object VO relative to the user U is calibrated to match the posture of the real object RO relative to the user U.

[0117] That is, by performing the calibration process, for example, the orientation of the real object RO1 relative to the user U can be made to coincide with the orientation of the virtual object VO1 relative to the user U.

[0118] As described above, according to the environment generation method of the second embodiment, the real space RS is recognized, and the virtual object VO1 is generated to correspond to the real object RO1 existing in the real space RS. Furthermore, if the execution conditions described above are met, i.e., if at least one of the conditions (c1) and (c2) is met, a calibration process is executed to cause the first predetermined parameter of the virtual object VO1 to correspond to the second predetermined parameter of the real object RO1. If the execution conditions are not met, the real space RS and the virtual object VO1 are used to generate the environment of the mixed reality space MRS to be recognized by the user U. If the execution conditions are met, the real space RS and the calibrated virtual object VO1 are used to generate the environment of the mixed reality space MRS to be recognized by the user U.

[0119] In this way, by performing the calibration process so that the first predetermined parameter of the virtual object VO1 corresponds to the second predetermined parameter of the real object RO1 under conditions where the conditions for performing the calibration process are met, the relationship between the user U and the real object RO1 can be made to correspond to the relationship between the user U and the virtual object VO1. For example, it is possible to prevent a discrepancy between the timing at which the user U moves toward the real object RO1 and reaches the real object RO1 and the timing at which the user U reaches the virtual object VO1. Therefore, it is possible to perform calibration of the first predetermined parameter of the virtual object VO1 under appropriate conditions where at least one of the conditions (c1) and (c2) is met. [Explanation of symbols]

[0120] S Environment Generation System 3. Server (computer system) 30 Real Space Recognition Department 31 Environment generation part 31a Virtual space generation unit 31b Avatar generation section 31c Virtual Object Generation Unit 31d Calibration processing section 32 User state recognition unit 33 Avatar state control unit 4 Head-mounted display (environment output device) VS Virtual Space RS Real Space A1 First Avatar (Avatar) A2 Second Avatar (Avatar) U1 First user (user) U2 Second user (user) VO1 Treasure Chest (Virtual Object) VO2 Rock Wall (Virtual Object) VO3 Pillar (virtual object) RO1 Box (real object) RO2 Bookshelf (real object) RO3 Drawer (Real Object) X1 First specified range (specified range) X2 Second specified range (specified range) Sx Environment Generation System 35 Environment generation part 35a Virtual object generation unit 35b Calibration processing section

Claims

1. 1. An environment generation method executed by a computer system, for generating a virtual space environment that is recognized by a user via an environment output device, comprising: a virtual space generating step in which a virtual space generating unit generates the virtual space so as to correspond to a real space in which the user exists; an avatar generation step in which an avatar generation unit generates an avatar corresponding to the user in the virtual space; a virtual object generating step in which a virtual object generating unit generates a virtual object in the virtual space so as to correspond to an actual object existing in the real space; a user state recognition step in which a user state recognition unit recognizes movement of coordinates and changes in posture of the user; an avatar state control step of controlling a movement of coordinates and a change in posture of the avatar based on the movement of coordinates and a change in posture of the user by an avatar state control unit; a calibration processing step in which, when an execution condition is satisfied that the virtual object has already been generated in the virtual space and the user's coordinates are located within a predetermined range determined based on the coordinates of a predetermined trigger object existing in the real space, a calibration processing unit executes a calibration process to make a first predetermined parameter, which is at least one of coordinates, orientation, and shape of the virtual object, correspond to a second predetermined parameter, which is at least one of coordinates, orientation, and shape of the real object; an environment generation step in which, when the calibration process has not been executed, an environment generation unit generates the environment of the virtual space to be recognized by the user using the virtual space, the avatar, and the virtual object, and, when the calibration process has been executed, generates the environment of the virtual space to be recognized by the user using the virtual space, the avatar, and the virtual object on which the calibration process has been executed; An environment generating method comprising:

2. The environment generating method according to claim 1, The environment generating method is characterized in that the trigger object is the real object.

3. The environment generating method according to claim 1, The users include a first user and a second user; the trigger object is one of the first user and the second user; An environment generating method characterized in that the execution condition is that the coordinates of the other of the first user and the second user are located within the specified range determined based on the coordinates of one of the first user and the second user.

4. The environment generating method according to claim 1, in the calibration process, the first predetermined parameter of the virtual object is calibrated so that the first predetermined parameter of the virtual object corresponds to the second predetermined parameter of the real object, and at least one of an attitude, a shape, and a relative position of the virtual object with respect to the real space is calibrated based on the calibration of the first predetermined parameter of the virtual object; In the environment generation step, if the calibration process has not been performed, the environment of the virtual space to be recognized by the user is generated using the virtual space, the avatar, and the virtual object, and if the calibration process has been performed, the environment of the virtual space to be recognized by the user is generated using the calibrated virtual space, the avatar, and the calibrated virtual object.

5. The environment generating method according to claim 4, a real space recognition step in which a real space recognition unit recognizes an orientation and a shape of the real space; an environment generating method, characterized in that the calibration process calibrates the first predetermined parameter of the virtual object, and at least one of the posture, shape, and relative position of the virtual space with respect to the real space based on the posture and shape of the real space.

6. The environment generating method according to claim 1, An environment generation method characterized in that, in the calibration processing step, when the user's coordinates are moving closer to the coordinates of the specified trigger object, the specified range is determined to be larger the faster the speed of movement of the user's coordinates.

7. An environment generation method executed by a computer system, for generating an environment in a mixed reality space to be recognized by a user via an environment output device, comprising: a real space recognition step in which a real space recognition unit recognizes the orientation and shape of the real space; a virtual object generation step in which a virtual object generation unit generates a virtual object so as to correspond to an actual object existing in the real space; a user state recognition step in which a user state recognition unit recognizes the coordinates and posture of the user; a calibration processing step in which, when an execution condition is satisfied that the virtual object has already been generated in the mixed reality space and the user's coordinates are located within a predetermined range determined based on the coordinates of a predetermined trigger object existing in the real space, a calibration processing unit executes a calibration process to make a first predetermined parameter, which is at least one of coordinates, orientation, and shape of the virtual object, correspond to a second predetermined parameter, which is at least one of coordinates, orientation, and shape of the real object; an environment generation step in which, when the calibration process has not been executed, an environment generation unit generates the environment of the mixed reality space to be recognized by the user using the real space and the virtual object, and, when the calibration process has been executed, generates the environment of the mixed reality space to be recognized by the user using the real space and the virtual object on which the calibration process has been executed; An environment generating method comprising:

8. An environment generating system configured to execute the environment generating method according to any one of claims 1 to 7.

9. An environment generating device configured to execute the environment generating method according to any one of claims 1 to 7.

10. A program causing a computer system to execute the environment generating method according to any one of claims 1 to 7.

11. 11. A recording medium on which the program according to claim 10 is recorded, the program being readable by the computer system.

Images