Information processing system, program, and information processing method

Abstract

To naturally reproduce an object in a virtual space.SOLUTION: An information processing system comprising: an acquisition unit configured to acquire a distance between a reference plane set in a real space and a target, the distance being obtained from a motion of the target; and a generation unit configured to generate a map used to reproduce the target in a virtual space, the map representing an undulation with respect to the reference plane on the basis of the distance acquired by the acquisition unit.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system, a program, and an information processing method. [Background technology]

[0002] In recent years, technologies have been developed to provide more realistic virtual spaces. For example, Patent Document 1 discloses a technology for making background images such as the ground in a virtual space appear clearer.

[0003] Furthermore, there has been an increase in the reproduction of real-world objects in virtual space. For example, objects are reproduced in virtual space based on motion data that captures the movements of objects in real space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-92635 Summary of the Invention [Problem to be solved by the invention]

[0005] When generating a virtual space, it is sometimes necessary to reduce the processing load of modeling, etc. To reduce the processing load, undulating surfaces such as the ground in the real world may be reproduced as flat surfaces in the virtual space.

[0006] However, for example, if an object moving on an uneven surface is reproduced exactly on a virtual surface that does not match the real space, such as a flat surface, the object may end up being reproduced unnaturally in the virtual space.

[0007] Therefore, the present disclosure proposes a new and improved technology that can naturally reproduce objects in a virtual space. [Means for solving the problem]

[0008] According to the present disclosure, an information processing system is provided that includes an acquisition unit that acquires the distance between the object and a reference plane set in real space, obtained from the movement of the object, and a generation unit that generates a map used to reproduce the object in a virtual space, representing the undulations relative to the reference plane, based on the distance acquired by the acquisition unit.

[0009] Furthermore, according to the present disclosure, a program is provided that causes a computer to function as an acquisition unit that acquires the distance between a reference plane set in real space and the object, obtained from the movement of the object, and a generation unit that generates a map used to reproduce the object in a virtual space, representing the undulations relative to the reference plane, based on the distance between the object and the reference plane acquired by the acquisition unit.

[0010] Furthermore, according to the present disclosure, there is provided an information processing method executed by a computer, which includes: acquiring a distance between a reference plane set in real space and the object, obtained from the movement of the object; and generating a map to be used to reproduce the object in a virtual space, which represents the undulations relative to the reference plane, based on the acquired distance between the reference plane and the object. [Brief explanation of the drawings]

[0011] [Figure 1] 10A and 10B are diagrams for explaining correction of the positions of the feet of a player in a virtual space. [Figure 2] FIG. 10 is a diagram for explaining the undulations pseudo-represented by the measurement results of characteristic parts of the court and the actual undulations. [Figure 3] 2 is a block diagram showing an example of a functional configuration of a server 10 according to an embodiment of the present disclosure. FIG. [Figure 4] 10 is a diagram for explaining an example of an operation process of the server 10 according to an embodiment of the present disclosure. FIG. [Figure 5] FIG. 10 is a diagram illustrating an example of a height map generated by setting the lowest player height among the player heights obtained on the block. [Figure 6]FIG. 6 is a diagram illustrating an example of a height map generated by resetting the player heights set in a block B that includes outliers in the height map M1 shown in FIG. 5. [Figure 7] FIG. 10 is a diagram illustrating weights set to adjacent blocks B. [Figure 8] 7 is a diagram illustrating an example of a height map obtained by smoothing the height map M2 shown in FIG. 6. FIG. [Figure 9] 10 is a diagram for explaining a virtual space generated by a spatial processing unit 122. FIG. [Figure 10] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device 900 that realizes the server 10 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0013] The explanation will be given in the following order. 1. Overview 2. Functional configuration example 3. Operation processing 4. Hardware configuration example 5. Supplementary Information

[0014] <1. Overview> The present disclosure relates to an information processing system for reproducing objects in real space in a virtual space. Examples of objects in real space reproduced by the information processing system according to the present disclosure include people (users) such as players playing sports like soccer or skateboarding, performers or spectators at dance, theater, or live music events, and moving objects such as robots or balls. Furthermore, the objects in real space reproduced by the information processing system according to the present disclosure may be parts of the people or objects. In this disclosure, the description will be centered on the assumption that soccer players are reproduced in a virtual space.

[0015] The number of objects in the real world that are reproduced in the virtual space is not particularly limited. In the present disclosure, an example in which multiple players participating in a match are reproduced in the virtual space will be mainly described.

[0016] The reproduction of the players in the virtual space is based on motion data captured from the players' movements in real space. The motion data may be obtained, for example, by analyzing images of the players captured by multiple cameras installed in the real space.

[0017] The motion data may include the position of a player in real space. More specifically, the motion data may include each position of a player's skeleton. The motion data may also include information other than the position of a player in real space and each position of a player's skeleton. Information such as the height and shoulder width of each player may also be attached to the motion data as metadata.

[0018] The position of the athlete may be a position on three-dimensional coordinate axes set in real space. The three-dimensional coordinate axes are calibrated and set when capturing an image of the subject or analyzing motion data. More specifically, two of the set three-dimensional coordinate axes may be set to align with a plane in real space (e.g., a surface that serves as a reference in real space, such as the ground, ceiling, or wall), and the remaining axis may be set in the height direction relative to the plane in real space. In the present disclosure, an example will be mainly described in which two of the three-dimensional coordinate axes are set to align with a plane in real space where the athlete plays a sport (e.g., the ground of a soccer field, the ground surface of a baseball stadium, a wall for bouldering, etc.), and the remaining axis is set in the height direction relative to the plane in real space. Note that the plane in real space that serves as the reference for setting the three-dimensional coordinate axes is not limited and may be any of the ground, ceiling, wall, etc.

[0019] The motion data also includes the player's height. The player's height is represented by the distance between the player (more specifically, the player's feet) and a reference plane. The reference plane may be a plane specified by two of the three-dimensional coordinate axes set in real space. More specifically, the reference plane according to the present disclosure is a plane set along real space that is specified by two axes set along the ground. Hereinafter, the player's height included in the motion data will also be referred to as "player height."

[0020] (Identifying issues) The real space in which players play may have undulations. Here, a soccer court is taken as an example of a real space. Generally, outdoor soccer courts have a ground shape that rises around the center circle. Furthermore, the court may have undulations due to, for example, the thickness of the turf and the unevenness of the ground. Here, since the motion data is represented based on three-dimensional coordinate axes set in the real space, the player heights included in the motion data are represented taking into account the undulations of the court.

[0021] On the other hand, in order to reduce the processing load in generating a virtual space, it is conceivable to reproduce an undulating surface such as an outdoor court as a horizontal surface in the virtual space.

[0022] In order to reproduce players in real space moving on an undulating court in a natural way in virtual space, it may be possible to correct the positions of the players in virtual space, more specifically, the positions of the players' feet.

[0023] Fig. 1 is a diagram for explaining correction of the positions of a player's feet in a virtual space. Fig. 1 shows a player's feet F (F1 to F3) and the player's feet Fc (Fc1 to Fc3) after position correction in a virtual space V.

[0024] The real-space corresponding surface RF shown in Fig. 1 is a surface in the virtual space V that corresponds to an undulating surface in the real space. The reproduction surface VF shown in Fig. 1 is a surface for reproducing the real-space corresponding surface as a flat surface in the virtual space V. The reproduction surface VF can be a surface in the virtual space V that corresponds to a reference surface in the real space.

[0025] As shown in the top of Figure 1, if the position of a player's foot F is reproduced without correction, even if the player's foot F is in contact with an undulating surface in real space, the player's foot F will float above or sink into the reproduced surface VF due to the difference between the real-space corresponding surface RF and the reproduced surface VF. For example, the player's foot F1, which was in contact with an undulating surface in real space, will be reproduced floating above the reproduced surface VF by the difference h.

[0026] On the other hand, as shown in the lower part of Figure 1, the position of the player's foot Fc after position correction is reproduced by correcting the position of the player's foot F by the difference between the real-space corresponding surface RF and the reproduction surface VF. For example, the position of the player's foot Fc1 after position correction is reproduced by correcting the position of the player's foot F1 by the difference h between the real-space corresponding surface RF and the reproduction surface VF. As a result, the foot that corresponds to the player's foot F1 and that is in contact with the undulating surface in the real space is also reproduced in the virtual space V so that it is in contact with the reproduction surface VF, making it possible to reproduce the player naturally in the virtual space V.

[0027] To perform such a correction, it becomes necessary to acquire the undulations of the undulating surface in real space.

[0028] Conventionally, maps showing the undulations of roadways and other surfaces are generated using depth information acquired by measuring machines equipped with infrared sensors, etc. However, because sports courts have many uneven surfaces, it is difficult to operate the measuring machines. In addition, the condition of the surface can change in a short period of time due to factors such as turf maintenance and wind and rain. For example, the turf on soccer courts is generally replaced each season, so the condition of the surface changes with each season. However, operating the measuring machines every so often is difficult.

[0029] It is also possible to obtain predicted values ​​for the undulations of each point on the court by linearly interpolating other parts of the court based on the results of measuring the height of characteristic points on the court (for example, the corners of the court, the center of the center circle, etc.). However, there will be differences between the undulations represented by the results of measuring the characteristic points on the court and the actual undulations due to the influence of local undulations, etc. Figure 2 is a diagram for explaining the undulations represented by the results of measuring the characteristic points on the court and the actual undulations.

[0030] Figure 2 shows players' feet F (F1 to F3), a real-space corresponding surface RF, and a pseudo-undulating surface MF in a virtual space V. The real-space corresponding surface RF is a surface in the virtual space V that corresponds to an undulating surface in real space. The pseudo-undulating surface MF is a surface in the virtual space V that corresponds to an undulating surface that is represented based on the measurement results of characteristic parts of the court.

[0031] As shown in Figure 2, there is a difference between the real-space corresponding surface RF and the pseudo-undulating surface MF, so even if the position of the player's foot F in the virtual space V is corrected according to the height of the pseudo-undulating surface MF, the object may still be reproduced unnaturally.

[0032] Therefore, this disclosure proposes a technology for generating a map that represents the undulations relative to a reference surface, i.e., the height of the court relative to the reference surface, which enables players to be reproduced naturally in a virtual space.

[0033] <2. Example of functional configuration> First, a functional configuration example of the server 10 for realizing an embodiment of the present disclosure will be described. Fig. 3 is a block diagram showing a functional configuration example of the server 10 according to an embodiment of the present disclosure. The server 10 is an information processing device that constitutes an information processing system according to the present disclosure. In this embodiment, an example in which the information processing system is constituted by a single server 10 will be described, but the information processing system may include multiple devices, and in this case, the functions of the server 10 described below may be realized by multiple devices.

[0034] As shown in FIG. 3, the server 10 according to this embodiment includes a communication unit 110, a control unit 120, and a storage unit .

[0035] (Communication unit 110) The communication unit 110 has a transmitting unit that transmits data to an external device and a receiving unit that receives data from the external device. The communication unit 110 according to this embodiment may be communicatively connected to an external device or the Internet using, for example, a wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution), 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system)), or the like.

[0036] The communication unit 110 functions as an acquisition unit that acquires motion data obtained from the movements of a player (an example of a user) from an external device. The motion data can be acquired by analyzing captured images of the player taken by multiple cameras installed in real space. However, the method of acquiring the motion data is not particularly limited as long as it is possible to acquire the position and height of the player. For example, a method such as OpenPose, which estimates the position of the player's joints from captured images, may be adopted, or the data may be acquired by a capture device worn by the player and configured with an inertial sensor or the like.

[0037] As described above, the motion data includes the position of each bone structure of a player in real space and the player's height. The position of each bone structure of a player and the player's height are expressed based on preset three-dimensional coordinate axes. More specifically, the player's height is expressed by the distance between a reference plane, which is a plane set along the court on which the player plays, and the player's feet. The distance between the reference plane and the player's feet is the first distance according to this embodiment.

[0038] For example, the communication unit 110 may acquire motion data of all players participating in a match at each time during the match. Furthermore, when the technology according to the present disclosure is applied to the reproduction of dance, theater, live music, or the like in a virtual space, the communication unit 110 may acquire motion data of all performers participating in the performance at each time during the performance. The more motion data acquired in an area reproduced in the virtual space, the more accurate the heights represented by the map generated by the map generation unit 121 (described below) will be.

[0039] (control unit 120) The control unit 120 functions as an arithmetic processing unit and a control device, and controls the overall operation of the server 10 in accordance with various programs. The control unit 120 is realized by an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor. The control unit 120 may also include a ROM (Read Only Memory) that stores the programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0040] The control unit 120 also functions as a map generation unit 121 and a spatial processing unit 122 .

[0041] (Map generation unit 121) The map generation unit 121 is a generation unit that generates a map that represents the undulations relative to a reference surface based on the player heights included in the motion data acquired by the communication unit 110. The map generation unit 121 generates a map for an area that is reproduced in a virtual space.

[0042] In this embodiment, an example will be described in which such a map represents the height of each position on the court relative to a reference plane. Here, height represents the distance between the reference plane and each position on the court, and is expressed as a positive or negative value, with a vertical upward direction relative to the reference plane being positive. Hereinafter, such a map will also be referred to as a "height map." Details of the height map generation process by the map generation unit 121 will be described later.

[0043] (Spatial Processing Unit 122) The spatial processing unit 122 generates an image that reproduces the player in the virtual space based on the height map generated by the map generation unit 121 and the player's motion data acquired by the communication unit 110.

[0044] The spatial processing unit 122 functions as a processing unit that corrects the positions of players in the virtual space according to the undulations that correspond to the positions of players placed in the virtual space, as represented by the height map. The processing for correcting the positions of players will be described in detail later.

[0045] (Storage unit 130) The storage unit 130 is realized by a ROM that stores programs and calculation parameters used in the processing of the control unit 120, and a RAM that temporarily stores parameters that change as appropriate. For example, the storage unit 130 stores a height map generated by the map generation unit 121.

[0046] <3. Operation processing> Next, an example of the operation process of the server 10 according to an embodiment of the present disclosure will be described.

[0047] 4 is a diagram illustrating an example of an operation process of the server 10 according to an embodiment of the present disclosure. As shown in FIG. 4, the operation of the server 10 according to an embodiment of the present disclosure is divided into the steps of (S10) acquiring motion data, (S20) generating a height map, and (S30) reproducing a virtual space. Each of these steps will be described below with reference to FIGS. 5 to 9.

[0048] (S10) Acquisition of motion data The communication unit 110 acquires motion data obtained from the movements of the players from an external device, and outputs the acquired motion data to the map generation unit 121 and the spatial processing unit 122.

[0049] (S20) Height map generation The map generation unit 121 generates a height map based on the player heights included in the motion data acquired by the communication unit 110. The height map is generated through steps S21 to S23.

[0050] The map generation unit 121 first divides the area to be reproduced in the virtual space, i.e., the court, into areas of a predetermined size. More specifically, a reference plane corresponding to the court is divided into areas of a predetermined size. The shape of the areas is not particularly limited and may be any shape. For example, it may be a rectangle, i.e., a block. More specifically, the block may be a square with one meter on each side. Here, an example is described in which the block is a square with one meter on each side, but the length of each side may be any length. The shape of the block and the length of each side may be determined automatically depending on the size of the area reproduced in the virtual section, or the user may be able to set them as appropriate.

[0051] In the subsequent processing, the map generation unit 121 generates a map in which the height of the court from the reference plane is represented for each block. Therefore, the smaller the block size, the more precisely a map can be created that shows the height of each position. On the other hand, the more player heights are obtained on a block, the more accurate the height represented for each block will be. Therefore, the size of the block can be changed as appropriate depending on the number of player height data points, etc.

[0052] Hereinafter, the height of the court from the reference plane, which is expressed for each block, will also be referred to as the “block height.” The block height is a distance that represents the undulations of the block according to this embodiment, and is the second distance.

[0053] The map generation unit 121 selects the smallest player height among the player heights acquired on the blocks included in the motion data acquired by the communication unit 110 (S21). The player height acquired on the block is the player height obtained from the foot of the player that is located in the perpendicular direction to the face of the block. For example, if three player heights of 10 cm, 0 cm, and -10 cm are acquired on the block, the map generation unit 121 selects -10 cm.

[0054] Here, the foot of the player corresponding to the lowest player height is likely to be in contact with the court. Therefore, the lowest player height is likely to represent the height of the court relative to the reference plane. Therefore, the map generation unit 121 generates a height map by setting the player height selected for each block as the height of each block.

[0055] Fig. 5 is a diagram showing an example of a height map generated by setting the lowest player height among the player heights obtained on the blocks. The height map M1 shown in Fig. 5 represents the height of each block B (e.g., blocks B1 to B5) into which the court is divided. Note that the diagram is simplified here for ease of explanation, but in reality, the blocks B may be divided into more blocks and set on the court.

[0056] Legend G is a legend for the heights of the blocks represented in height map M1. For example, in height map M1, the height of the court in block B1 is represented as approximately 20 cm. Also, in height map M2, the height of the court in block B2 is represented as approximately -20 cm.

[0057] It is conceivable that among the blocks B in the height map M1, there may be a block B for which the player height is not acquired. For example, if there is a block B through which no player passes during a match for which motion data is acquired, the player height is not acquired for that block B.

[0058] In such a case, the map generation unit 121 does not need to set the block height of the block B. In Fig. 5, block B3 in the height map M1 is blank. This indicates that the block height of block B3 has not been set.

[0059] Here, due to errors when generating the motion data, the motion data may contain a player height that is lower or higher than the actual height. Furthermore, motion data may be generated based on the capture results of a player who has lifted their feet by jumping, etc. This may cause the block height set for each block B to differ from the actual height. For example, block B4 in the height map M1 of Figure 5 is significantly lower than the surrounding blocks, so it is likely that a height different from the actual height is set as the block height.

[0060] Therefore, the map generating unit 121 generates a height map that represents more accurate block heights in the subsequent steps S22 and S23.

[0061] First, the map generating unit 121 removes outliers (S22). More specifically, the map generating unit 121 determines whether or not an outlier is set in the block B by determining whether or not each of the block heights set in the block B satisfies a predetermined condition.

[0062] Then, if the block height set for block B satisfies a predetermined condition, map generating unit 121 changes the block height set for block B, thereby removing the outlier.

[0063] Changing the height of a block may involve resetting the set height of the block, or changing the height of the block according to the height of the block set for the surrounding block B. Here, an example of changing the height of a block by resetting the set height of the block will be described.

[0064] The predetermined condition may include, for example, that the block height set for block B is equal to or greater than a first value or equal to or less than a second value.

[0065] The first and second values ​​may be set appropriately depending on the expected degree of undulation of the court (i.e., the expected block height) and the set position of the reference plane. This allows for the block height to be removed as an outlier if an unexpected block height is set. The first and second values ​​may be the upper and lower limits of the expected block height, respectively, or vice versa. For example, the first and second values ​​may be 20 cm and -20 cm, respectively.

[0066] The specified condition may also include that the difference between the block height set in block B and the average value of the block heights set in each block B adjacent to block B is equal to or greater than a third value.

[0067] The third value may be set appropriately depending on the expected degree of slope of the court's undulations. This allows the block height of a block B that is determined to have been set incorrectly based on its relationship with the block heights set for adjacent blocks B to be removed as an outlier. The third value may be, for example, 5 cm. Here, the first value, second value, and third value may each be a predetermined value determined in advance, or each value may be set arbitrarily by the user.

[0068] Fig. 6 is a diagram showing an example of a height map generated by resetting the player heights set in blocks B containing outliers in the height map M1 shown in Fig. 5. The height map M2 shown in Fig. 6 is a height map generated by resetting the player heights set in blocks B containing outliers in the height map M1. In the height map M2, as in the height map M1, the heights represented in the height map M2 are represented by legend G.

[0069] 5 and 6, blocks B such as block B4 that were not blank in height map M1, i.e., had a block height set, are blank in height map M2. This indicates that such blocks B contain outliers, i.e., the player heights set for blocks B that are thought to have a different height from the actual height of block B have been reset.

[0070] Next, the map generation unit 121 smoothes the height map M2 using a weighted average (S23). More specifically, for a block B for which a block height is set, the map generation unit 121 resets the block height to the weighted average of the height of the block and the height of the block B adjacent to the block B.

[0071] FIG. 7 is a diagram illustrating the weights set for each adjacent block B. FIG. 7 shows a block Bs whose height is to be reset, and nine blocks Ba (Ba1, Ba2, ...) adjacent to block Bs. As shown in FIG. 7, a weight of "2" may be used for block Bs whose height is to be reset, and a weight of "1" may be used for block Ba. The map generation unit 121 calculates the height of the block to be reset for block Bs by a weighted average using such weights.

[0072] The map generating unit 121 performs weighted averaging by setting the weight of block B, for which no block height is set, to 0.

[0073] The weights may be set arbitrarily and are not limited to the example shown in Fig. 7. Furthermore, the weighted average may further use the height of blocks B in a wider range, for example, the height of the block Bs whose height is reset and the height of the block B two blocks away.

[0074] Furthermore, for a block B for which the block height has not been set, the block height is calculated using the block heights set for each adjacent block B. Hereinafter, a block B for which the block height has not been set will be referred to as an "unset block Bn."

[0075] For example, the map generating unit 121 may calculate the average value of the heights of the blocks set in the blocks B adjacent to the unset block Bn as the height of the block to be set in the unset block Bn.

[0076] It is also possible that block heights have not been set for all blocks B adjacent to an unset block Bn. In this case, the map generation unit 121 may set the block height set for the block B that is closest to the unset block Bn and for which a block height has been set as the block height of the unset block Bn. If there are multiple blocks B that are closest to the unset block Bn and for which a block height has been set, the average value of the heights of the multiple blocks may be set as the block height of the unset block Bn.

[0077] Fig. 8 is a diagram showing an example of a height map obtained by smoothing the height map M2 shown in Fig. 6. The height map M3 shown in Fig. 8 is a height map generated by performing a weighted average process on each block of the height map M2 and setting the block height of the unset block Bn. In the height map M3, as in the height maps M1 and M2, the heights represented in the height map M3 are represented by a legend G.

[0078] In height map M3, height map M2 is smoothed by weighted averaging. For example, in height maps M1 and M2, block B5 was set to a height of approximately -20 cm. In height map M3, block B5 is smoothed by resetting its height to a higher value than before by taking a weighted average of the set block height and the block height set for the adjacent block B.

[0079] Furthermore, in the height map M3, by setting the block height for the unset block Bn, the block height is set for all blocks B in the height map M3.

[0080] For example, in height maps M1 and M2, the player height was not acquired for block B3, so no block height was set for block B3 and it was blank. In height map M3, the block height is set by calculating the average value of the block heights set for blocks B adjacent to block B3.

[0081] In addition, in the height map M2, due to the removal of outliers, no block height was set for block B4, and it was left blank. In the height map M3, it can be seen that the block height is set by calculating the average value of the block heights set for blocks B adjacent to block B4.

[0082] According to the steps S22 and S23 described above, the deviation between the actual block height and the block height in the height map M1 due to an error or the like when generating the motion data is corrected.

[0083] The height of each block B indicated by each height map M (height maps M1 to M3) may be adjusted by an administrator of the virtual space, etc. More specifically, the height map M may be transmitted to an external terminal, and an editing screen for the height map M may be displayed on the external terminal. By operating the editing screen, the administrator may specify a block B whose height is to be changed, or may change the height of the specified block B. The height may be changed by inputting a value, or may be changed intuitively by moving the block B shown together with the reference plane in the height direction.

[0084] The generation of the height map M has been described above. Up to this point, an example has been described in which the player height, which indicates the distance between the player and the reference plane in the motion data acquired by the communication unit 110, is used to set the height of the blocks in the height map M. That is, up to this point, an example has been described in which the reference plane in the height map M where the height of the blocks is 0 cm (hereinafter also referred to as the "map reference plane") matches the reference plane in the motion data. However, the map reference plane in the height map M may be set to a plane different from the reference plane in the motion data.

[0085] For example, the map reference plane may be set to be based on the height of an object fixed within the court, such as the height of a goal net installed within the court.

[0086] If the map reference plane is set to a plane different from the reference plane in the motion data, the player height included in the motion data can be converted to the player height from the map reference plane according to the positional relationship between the set map reference plane and the reference plane in the motion data. By using the converted player height, the map generation unit 121 can set the height of each block obtained by dividing the map reference plane, with the map reference plane being 0 cm.

[0087] Furthermore, the map base plane may be set based on the height of an object set in advance by the user, or may be set based on priority information set in advance by the user. The type of priority information is not particularly limited. For example, it may be expressed as a number such as 0 to 10, or may be expressed as text such as HIGH, MIDDLE, and LOW. Typically, there are multiple candidates for the object in real space that will serve as the map base plane. Therefore, the user may set priority information for each candidate object in advance, and the object with the highest priority may be set as the map base plane, thereby more closely reflecting the user's intentions.

[0088] (S30) Recreating virtual space The spatial processing unit 122 generates an image that reproduces the player in the virtual space based on the smoothed height map M3 generated by the map generation unit 121 and the player's motion data acquired by the communication unit 110. Note that the spatial processing unit 122 may generate an image that reproduces the player in the virtual space based on the height map M1 or the height map M2 generated by the map generation unit 121.

[0089] For example, in generating the virtual space, the spatial processing unit 122 reproduces an undulating court as a flat surface in the virtual space. Hereinafter, the surface reproduced in the virtual space will also be referred to as the "reproduced surface." The reproduced surface may coincide with the map base surface. In this case, the height of each block represented by the height map M3 represents the difference between the position of each position on the court in the virtual space and the reproduced surface. The following mainly describes an example in which the reproduced surface coincides with the map base surface.

[0090] The spatial processing unit 122 corrects the position of the player in the virtual space represented based on the motion data, more specifically, the height of the player in the virtual space, in accordance with the height of the block set in block B corresponding to the position of the player to be placed in the virtual space, as represented by the height map M3.

[0091] The spatial processing unit 122 corrects the position of the player's feet by moving them toward the map reference plane in the virtual space, i.e., the reproduction plane, by the amount of the undulations corresponding to the positions of the player's feet. More specifically, as explained with reference to Fig. 1, the spatial processing unit 122 corrects the position of the player's feet F by the amount of the difference between the real-space corresponding plane RF (the plane corresponding to the court in the virtual space) and the reproduction plane VF, i.e., by the amount of the height of each block represented by the height map M3, and reproduces the position in the virtual space.

[0092] When correcting the position of a player's feet, the positions of other parts of the player (such as hands) may also be corrected by the same amount according to the corrected position of the feet, thereby generating a more natural-looking image of the virtual space.

[0093] Fig. 9 is a diagram illustrating the virtual space generated by the spatial processing unit 122. The upper part of Fig. 9 shows an image Ib of a virtual space V that reproduces players in real space when no correction is performed by the spatial processing unit 122. At positions corresponding to the players reproduced in the image Ib of the virtual space V, the height of the court in real space is lower than the map reference surface. Therefore, in the image Ib of the virtual space V, the feet of player P1 and the hands of player P2, which are actually in contact with the court, are reproduced as being sunken into the court.

[0094] The lower part of FIG. 9 shows the image Ic of the virtual space V, which reproduces the players in real space after correction by the spatial processing unit 122. The spatial processing unit 122 corrects the position of each part of the player according to the height of the block set for each block at the position where the player reproduced in the image Ic of the virtual space V is located, as represented by the height map M3. This generates the image Ic of the virtual space V, which reflects the positional relationship between the player and the court in real space, i.e., the height of the player relative to the court in real space. This configuration allows the players to be reproduced naturally in the image Ic of the virtual space V. For example, in the image Ic of the virtual space V, the feet of player P1 and the hands of player P2 are reproduced so that they do not sink into the court but are in contact with it.

[0095] The above describes an example of correction by the spatial processing unit 122 when the reproduction surface and the map reference surface coincide. However, the spatial processing unit 122 may use a surface that does not coincide with the map reference surface as the reproduction surface. In this case, the spatial processing unit 122 may correct the positions of each part of the player by the amount of the difference between the map reference surface and the reproduction surface at each position on the reproduction surface. The reproduction surface is not limited to a flat surface, but may also be an undulating surface.

[0096] Freely setting the reproduction surface increases the degree of freedom in expressing virtual space. For example, when a correction process that uses a surface that does not coincide with the map base surface as the reproduction surface is applied to reproducing performers on a stage, such as a dance, play, or music concert, in a virtual space, it becomes possible to express a more complex stage that differs from the stage in real space. This can enhance the enjoyment of users viewing the video in virtual space. For example, in a music concert, it is expected that complex-shaped objects are set on the stage where the performers appear, or that there are objects with steps, such as stairs, connecting the stage to the audience. Even in such cases, by appropriately setting the reproduction surface to the stage or stairs, and correcting the positions of the performers and audience by the amount of the difference between the map base surface and each reproduction surface, the positions of the performers' and audience's feet and hands can be properly depicted without sinking into the surface of the stage or stairs. This makes it possible to provide users with a more natural experience.

[0097] The above description deals with an example in which the spatial processing unit 122 uses the height map M3 to recreate an undulating court as a flat surface in virtual space. However, the spatial processing unit 122 may recreate an undulating court in virtual space by reflecting the shape of the undulations indicated by the height map M3 on a surface in virtual space that corresponds to the map reference surface of the height map M3. In this case, the spatial processing unit 122 recreates players in virtual space without correcting each part of the player. This allows the appearance in real space to be recreated more faithfully in virtual space.

[0098] Furthermore, up to this point, an example has been described in which the spatial processing unit 122 generates a virtual space including a player using a height map M generated based on the motion data of the player reproduced in the virtual space. However, the spatial processing unit 122 may generate a virtual space using a height map M generated using motion data different from the motion data of the player reproduced in the virtual space. For example, when reproducing the state of a match in real time in a virtual space, the height map M may be generated based on player motion data acquired in another match that was previously played on the court where the match is being played.

[0099] Other matches played in the past on the court where the match is being played may be, for example, matches played in the same season as the match being reproduced in the virtual space. Generally, in outdoor sports such as soccer, the turf is often replaced or repairs are made between seasons. Therefore, there is relatively little change in the topography of the court during the same season. Therefore, a natural virtual space can be generated by generating a virtual space based on a height map M generated based on player motion data acquired during such matches.

[0100] Furthermore, multiple height maps M generated based on motion data acquired from multiple games played in the same season may be used to calculate an average value of block height for each block B. By using such an average value as the block height, the accuracy of the block height increases, allowing a more natural virtual space to be generated.

[0101] However, there may be cases where motion data acquired from games played in the same season, such as the first game of the season, does not exist. In such cases, a height map M generated based on motion data acquired from games played in the immediately preceding season may be used. Even in this case, a more natural virtual space can be generated compared to, for example, using a height map that simulates the undulations of each part of the court based on measurement results of the heights of characteristic parts of the court.

[0102] Furthermore, although an example has been described in which the height map M is generated based on the motion data of players, the height map M may also be generated based on the motion data of other objects present on the court. For example, the height map M may also be generated based on the motion data of a soccer ball. Note that if the soccer ball motion data includes the position of the center of the soccer ball, the height of the soccer ball may be obtained from the size of the soccer ball. Then, the height map M may be generated using the height of the soccer ball in the same way as the player heights. Furthermore, the motion data of the match referee may be used in the same way as the player motion data.

[0103] Furthermore, the positions of other objects present on the court, as represented by their motion data, may be corrected based on the height map M3, thereby reproducing the other objects in the virtual space. For example, the position of a soccer ball may be corrected based on the height map M3 and reproduced in the virtual space.

[0104] <4. Hardware Configuration> An embodiment of the present disclosure has been described above. Next, an example of the hardware configuration of the server 10 according to the embodiment of the present disclosure will be described with reference to FIG.

[0105] The above-described processing by the server 10 can be realized by one or more information processing devices. Fig. 10 is a block diagram showing an example hardware configuration of an information processing device 900 that realizes the server 10 according to an embodiment of the present disclosure. Note that the information processing device 900 does not necessarily have to have all of the hardware configuration shown in Fig. 10. Furthermore, some of the hardware configuration shown in Fig. 10 may not be present in the server 10.

[0106] 10, the information processing device 900 includes a CPU 901, a ROM (Read Only Memory) 903, and a RAM 905. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. Instead of or in addition to the CPU 901, the information processing device 900 may have a processing circuit such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an ASIC (Application Specific Integrated Circuit).

[0107] The CPU 901 functions as an arithmetic processing unit and control unit, and controls all or part of the operations within the information processing device 900 in accordance with various programs recorded in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 927. The ROM 903 stores programs and calculation parameters used by the CPU 901. The RAM 905 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during the execution. The CPU 901, the ROM 903, and the RAM 905 are interconnected by a host bus 907 constituted by an internal bus such as a CPU bus. Furthermore, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.

[0108] The input device 915 is a device operated by a user, such as a button. The input device 915 may include a mouse, a keyboard, a touch panel, a switch, a lever, or the like. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device 929 such as a mobile phone that supports operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs the signal to the CPU 901. The user operates the input device 915 to input various data to the information processing device 900 and to instruct processing operations.

[0109] The input device 915 may also include an imaging device and a sensor. The imaging device is a device that captures real space and generates a captured image using an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) and various components such as a lens for controlling the formation of a subject image on the imaging element. The imaging device may capture a still image or a moving image.

[0110] The sensors include various types of sensors such as a distance measurement sensor, an acceleration sensor, a gyro sensor, a geomagnetic sensor, a vibration sensor, an optical sensor, and a sound sensor. The sensors acquire information about the state of the information processing device 900 itself, such as the attitude of the housing of the information processing device 900, and information about the surrounding environment of the information processing device 900, such as the brightness and noise around the information processing device 900. The sensors may also include a GPS (Global Positioning System) sensor that receives a GPS signal and measures the latitude, longitude, and altitude of the device.

[0111] The output device 917 is configured with a device capable of visually or audibly notifying the user of acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker or headphones. The output device 917 may also include a PDP (Plasma Display Panel), a projector, a hologram, a printer, or the like. The output device 917 outputs the results obtained by processing by the information processing device 900 as video such as text or images, or as sound such as voice or audio. The output device 917 may also include a lighting device that brightens the surroundings.

[0112] The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900. The storage device 919 is configured, for example, by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs and various data executed by the CPU 901, as well as various data acquired from the outside.

[0113] The drive 921 is a reader / writer for a removable recording medium 927 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into or externally attached to the information processing device 900. The drive 921 reads information recorded on the attached removable recording medium 927 and outputs the information to the RAM 905. The drive 921 also writes information to the attached removable recording medium 927.

[0114] The connection port 923 is a port for directly connecting a device to the information processing device 900. The connection port 923 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. The connection port 923 may also be an RS-232C port, an optical audio terminal, or an HDMI (registered trademark) (High-Definition Multimedia Interface) port. By connecting an external device 929 to the connection port 923, various types of data can be exchanged between the information processing device 900 and the external device 929.

[0115] The communication device 925 is, for example, a communication interface configured with a communication device for connecting to the network 931. The communication device 925 may be, for example, a communication card for a wired or wireless local area network (LAN), Bluetooth (registered trademark), Wi-Fi (registered trademark), or WUSB (Wireless USB). The communication device 925 may also be a router for optical communication, a router for an asymmetric digital subscriber line (ADSL), or a modem for various types of communication. The communication device 925 transmits and receives signals, for example, between the Internet and other communication devices using a predetermined protocol such as TCP / IP. The network 931 connected to the communication device 925 is a network connected by wire or wirelessly, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0116] <5. Supplementary Information> Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0117] For example, in the above embodiment, the map reference plane and the reference plane in the motion data are set along the ground, and a height map representing the undulations of the ground is generated. However, the present technology is not limited to such an example. For example, the map reference plane and the reference plane in the motion data may be set along a wall. In this way, a map representing the undulations of the wall is generated instead of the height map M in the above embodiment. For example, by generating a map using motion data of a bouldering athlete, it is possible to generate a map representing the undulations of a climbing wall. Using such a map makes it possible to naturally reproduce a bouldering athlete in a virtual space.

[0118] It is also possible to create a computer program for causing the hardware, such as the CPU, ROM, and RAM, built into the server 10 to perform the functions of the server 10. A computer-readable storage medium storing the computer program is also provided.

[0119] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0120] The following configurations also fall within the technical scope of the present disclosure. (1) an acquisition unit that acquires a distance between a reference plane set in real space and the object, the distance being obtained from the movement of the object; a generation unit that generates a map used to reproduce an object in a virtual space, the map representing undulations relative to the reference plane, based on the distance acquired by the acquisition unit; and Equipped with Information processing system. (2) the distance is a first distance, The generation unit Dividing the reference surface into regions of a predetermined size; setting the shortest first distance among the plurality of first distances acquired over the area as a second distance that is a distance representing the undulations of the area; The information processing system according to (1) above. (3) The generation unit determining whether each of the plurality of second distances that have been set satisfies a predetermined condition; The information processing system according to (2), wherein, when the second distance satisfies a predetermined condition, a setting of the second distance for the area corresponding to the second distance is changed. (4) The information processing system according to (3), wherein the predetermined condition includes that the second distance is equal to or greater than a first value or equal to or less than a second value. (5) The information processing system described in (3), wherein the specified condition includes that the difference between the second distance and the average value of the second distances of each area adjacent to the area is greater than or equal to a third value. (6) The information processing system according to (5), wherein the third value is set according to the degree of inclination of the undulations relative to the reference surface. (7) The information processing system described in any one of (2) to (6), wherein the generation unit resets the second distance for the area in which the second distance is set to a weighted average of the second distance and the second distance of an area adjacent to the area in question. (8) The information processing system described in any one of (2) to (7), wherein the generation unit sets the average value of the second distances of the areas adjacent to the area where the second distance is not set as the second distance of the area. (9) The information processing system includes: a processing unit that corrects a position of the target in the virtual space using the map generated by the generation unit, The information processing system according to any one of (1) to (8) above. (10) The information processing system according to (9), wherein the processing unit corrects the position of the object in the virtual space according to the undulations corresponding to the position of the object to be placed in the virtual space, as represented by the map. (11) The information processing system according to (10), wherein the processing unit performs a correction to move the position of the object toward the reference plane in the virtual space by an amount corresponding to an undulation corresponding to the position of the object. (12) The information processing system includes: a processing unit that reflects the shape of the undulations on a surface in the virtual space corresponding to the reference surface, The information processing system according to any one of (1) to (11) above. (13) the reference plane is a plane set based on real space, The distance between the reference plane and the object represents the height of the object from the reference plane. The information processing system according to any one of (1) to (12) above. (14) The information processing system according to (13), wherein the reference plane is a plane specified by two axes of three-dimensional coordinate axes that are set when obtaining motion data from the movement of the object. (15) the target is a part of the user's body existing in the real space, The reference plane is a plane set based on the location where the user will play. The information processing system according to any one of (1) to (14) above. (16) the body part is the user's foot; The information processing system according to (15) above. (17) the reference plane is set based on the object in real space that has the highest priority; The information processing system according to any one of (1) to (16) above. (18) the generation unit generates one or more of the maps based on the distances corresponding to objects in one or more games played in the past at locations in real space where the objects in the virtual space are reproduced, the locations being different from the objects in the virtual space whose positions are corrected by the processing unit; the processing unit corrects the position of the object in the virtual space according to an undulation corresponding to the position of the object in the virtual space represented by the one map generated by the generation unit, or an average value of undulations corresponding to the position of the object in the virtual space represented by the multiple maps; The one or more games are games that took place in the same season or the season immediately preceding the season in which the game played in the real space reproduced by the processing unit takes place. The information processing system according to any one of (9) to (11) above. (19) Computer, an acquisition unit that acquires a distance between a reference plane set in real space and the object, the distance being obtained from the movement of the object; a generation unit that generates a map used to reproduce the object in a virtual space, the map representing undulations relative to the reference plane based on the distance between the reference plane and the object acquired by the acquisition unit; and A program that functions as a (20) Obtaining a distance between a reference plane set in real space and the object, the distance being obtained from the movement of the object; generating a map to be used for reproducing the object in a virtual space, the map representing the undulations relative to the reference plane, based on the acquired distance between the reference plane and the object; 2. A computer-implemented information processing method, comprising: [Explanation of symbols]

[0121] 10 Servers 110 Communications Department 120 control section 121 Map Generation Unit 122 Spatial Processing Section 130 Storage section Block B M Height Map

Claims

1. an acquisition unit that acquires a distance between a reference plane set in real space and the object, the distance being obtained from the movement of the object; a generation unit that generates a map used to reproduce an object in a virtual space, the map representing undulations relative to the reference plane, based on the distance acquired by the acquisition unit; and Equipped with Information processing system.

2. the distance is a first distance, The generation unit Dividing the reference surface into regions of a predetermined size; setting the shortest first distance among the plurality of first distances acquired over the area as a second distance that is a distance representing the undulations of the area; The information processing system according to claim 1 .

3. The generation unit determining whether each of the plurality of second distances that have been set satisfies a predetermined condition; The information processing system according to claim 2 , wherein, when the second distance satisfies a predetermined condition, a setting of the second distance for the area corresponding to the second distance is changed.

4. The information processing system according to claim 3 , wherein the predetermined condition includes that the second distance is equal to or greater than a first value or equal to or less than a second value.

5. The information processing system according to claim 3 , wherein the predetermined condition includes a difference between the second distance and an average value of the second distances of the regions adjacent to the region being equal to or greater than a third value.

6. The information processing system according to claim 5 , wherein the third value is set in accordance with the degree of inclination of the undulations relative to the reference surface.

7. 3. The information processing system according to claim 2, wherein the generation unit resets the second distance for the region in which the second distance is set to a weighted average of the second distance and the second distance of a region adjacent to the region in question.

8. The information processing system according to claim 2 , wherein the generation unit sets an average value of the second distances of areas adjacent to the area in which the second distance is not set as the second distance of the area in which the second distance is not set.

9. The information processing system includes: a processing unit that corrects a position of the target in the virtual space using the map generated by the generation unit, The information processing system according to claim 1 .

10. The information processing system according to claim 9 , wherein the processing unit corrects the position of the target in the virtual space in accordance with an undulation corresponding to the position of the target to be placed in the virtual space, which is represented by the map.

11. The information processing system according to claim 10 , wherein the processing unit performs a correction to move the position of the target toward the reference plane in the virtual space by an amount corresponding to an undulation corresponding to the position of the target.

12. The information processing system includes: a processing unit that reflects the shape of the undulations on a surface in the virtual space corresponding to the reference surface, The information processing system according to claim 1 .

13. the reference plane is a plane set based on real space, The distance between the reference plane and the object represents the height of the object from the reference plane. The information processing system according to claim 1 .

14. The information processing system according to claim 13 , wherein the reference plane is a plane specified by two axes of three-dimensional coordinate axes that are set when obtaining motion data from the movement of the object.

15. the target is a part of the user's body existing in the real space, The reference plane is a plane set based on the location where the user will play. The information processing system according to claim 1 .

16. the body part is the user's foot; 16. The information processing system according to claim 15.

17. the reference plane is set based on the object in real space that has the highest priority; The information processing system according to claim 1 .

18. the generation unit generates one or more of the maps based on the distances corresponding to objects in one or more games played in the past at locations in real space where the objects in the virtual space are reproduced, the locations being different from the objects in the virtual space whose positions are corrected by the processing unit; the processing unit corrects the position of the object in the virtual space according to an undulation corresponding to the position of the object in the virtual space represented by the one map generated by the generation unit, or an average value of undulations corresponding to the position of the object in the virtual space represented by the plurality of maps; the one or more games are games that took place in the same season or the season immediately preceding the season in which the games played in the real space reproduced by the processing unit are played; The information processing system according to claim 9 .

19. Computer, an acquisition unit that acquires a distance between a reference plane set in real space and the object, the distance being obtained from the movement of the object; a generation unit that generates a map used to reproduce the object in a virtual space, the map representing undulations relative to the reference plane based on the distance between the reference plane and the object acquired by the acquisition unit; and A program that functions as a

20. Obtaining a distance between a reference plane set in real space and the object, the distance being obtained from the movement of the object; generating a map to be used for reproducing the object in a virtual space, the map representing the undulations relative to the reference plane, based on the acquired distance between the reference plane and the object; 2. A computer-implemented information processing method, comprising:

Images