IMAGE GENERATION DEVICE, IMAGE GENERATION METHOD, AND PROGRAM

Abstract

To more precisely express a depth of a virtual object superimposed on a body of a player.SOLUTION: An image generating apparatus 100 according to an embodiment of the present invention comprises: a player recognition unit 102 for recognizing a body of a player; a viewpoint acquiring unit 103 for acquiring viewpoint information including a viewpoint position and a viewpoint direction; a mesh generating unit 105 for generating a mesh structure of the player which reflects a skeleton of the body of the player, on the basis of a recognition result; and an image generating unit 110 for generating an image by rendering the mesh structure and a virtual object of the player viewed in the viewpoint direction of the viewpoint information from the viewpoint position thereof to superimpose the rendered virtual object on the rendered mesh structure of the player.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an image generating device, an image generating method, and a program. [Background technology]

[0002] Currently, a user wears a head-mounted display (HMD) connected to a game console on his / her head and operates a controller while looking at the screen displayed on the HMD to play a game. When wearing an HMD, the user sees nothing but the image displayed on the HMD, which increases the sense of immersion in the world of the image and has the effect of further enhancing the entertainment value of the game. In addition, if virtual reality (VR) images are displayed on the HMD and the user wearing the HMD rotates his / her head, a 360-degree virtual space that can be seen all around is displayed, which further increases the sense of immersion in the image and improves the operability of applications such as games.

[0003] There are also video see-through HMDs that can capture images of the outside world using a camera mounted on the HMD and display them on the display panel. Video see-through HMDs can also generate and display augmented reality (AR) images by superimposing virtual world objects generated by computer graphics (CG) on images of the outside world captured by the camera. Augmented reality images differ from virtual reality, which is separated from the real world, in that the real world is extended with virtual objects, allowing users to experience the virtual world while being aware of the connection with the real world. Summary of the Invention [Problem to be solved by the invention]

[0004] When displaying an image in which a virtual object is superimposed on an image of a player playing a game, the virtual object that should be in the background may overwrite a part of the player's body that should be in the foreground. This may result in an inability to accurately express depth. As a result, there is no sense of unity between the player's body and the virtual object, and the image becomes unnatural.

[0005] In view of the above problems, one objective of the present disclosure is to provide an image generating device, an image generating method, and a program capable of providing an image in which the depth of a player's body and a virtual object superimposed on the player's body is more accurately represented. [Means for solving the problem]

[0006] In order to solve the above problems, an image generation device of one aspect of the present disclosure includes a player recognition unit that recognizes a player's body, a viewpoint acquisition unit that acquires viewpoint information including a viewpoint position and a viewpoint direction, a mesh generation unit that generates a mesh structure of the player reflecting a skeleton of the player's body based on the recognition result, and an image generation unit that renders the mesh structure of the player and a virtual object when viewed from the viewpoint position in the viewpoint direction in the viewpoint information, and generates an image by superimposing the rendered virtual object on the rendered player's mesh structure.

[0007] Another aspect of an image generation method of the present disclosure includes the steps of recognizing a player's body, acquiring viewpoint information relating to a viewpoint position and a viewpoint direction, generating a player mesh structure reflecting a skeleton of the player's body based on the recognition result, and rendering the player's mesh structure and a virtual object when viewed from the viewpoint position in the viewpoint direction in the viewpoint information, and generating an image by superimposing the rendered virtual object on the rendered player's mesh structure.

[0008] A program according to yet another aspect of the present disclosure is a program for causing a computer to execute the steps of: recognizing a player's body; acquiring viewpoint information relating to a viewpoint position and viewpoint direction; generating a mesh structure of a player reflecting a skeleton of the player's body based on the recognition result; rendering the mesh structure of the player and virtual objects as viewed from the viewpoint position in the viewpoint direction in the viewpoint information, and generating an image by superimposing the rendered virtual objects on the rendered mesh structure of the player.

[0009] Any combination of the above components, and conversion of the present disclosure into a method, device, system, computer program, data structure, recording medium, etc. are also valid aspects of the present disclosure. Effect of the Invention

[0010] According to the present disclosure, it is possible to provide an image in which the depth of a player's body and a virtual object superimposed on the player's body is more accurately represented. [Brief description of the drawings]

[0011] [Figure 1] 1 is an overall schematic diagram of an image generating system according to a first embodiment. [Diagram 2] FIG. 13 is a diagram illustrating an example of how the embodiment is used by a user. [Diagram 3] FIG. 1 is a configuration diagram of an image generating device according to a first embodiment. [Figure 4A] FIG. 1 is a diagram illustrating a skeleton. [Figure 4B] FIG. 1 is a diagram illustrating a mesh structure. [Figure 4C] FIG. 13 is a diagram showing an example in which a virtual object is superimposed on a mesh structure of a player. [Diagram 5] FIG. 2 is a diagram showing an example of a captured image. [Figure 6]6 is a diagram showing an example of an image in which a virtual object is superimposed on the photographed image of FIG. 5. FIG. [Figure 7] FIG. 1 is a diagram showing an example of occlusion represented by a conventional method. [Figure 8] FIG. 4 is a diagram showing an example in which occlusion is expressed by the technique of the first embodiment. [Figure 9] FIG. 11 is a configuration diagram of an image generating device according to a second embodiment. [Figure 10A] The mesh structure of the player is shown superimposed on a photographed image of the real world. [Figure 10B] 10B is a diagram showing an example in which virtual objects of legs of a mechanical structure are superimposed on the legs of the mesh structure of the player in FIG. 10A. FIG. [Figure 10C] FIG. 13 is a diagram illustrating an example of a process for distorting space. [Figure 10D] FIG. 10D is a diagram showing an example in which a virtual object is superimposed after the processing of FIG. 10C. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] First Embodiment Fig. 1 is an overall schematic diagram of an image generation system according to a first embodiment. The image generation system 10 in Fig. 1 includes an image generation device 100. An example of the image generation device 100 is a game machine. The image generation device 100 is connected to a head mounted display (HMD) 15, a camera 19, and an image providing device 21. The number of HMDs 15 connected to the image generation device 100 is not limited to the example in Fig. 1.

[0013] The image generating device 100 includes a control unit 11, a storage unit 12, and an interface unit 13. The control unit 11 includes a processor and executes various information processing by executing programs stored in the storage unit 12. The storage unit 12 includes a memory device such as a RAM, and stores the programs executed by the control unit 11 and data processed by the programs.

[0014] The interface unit 13 is an interface for data communication between the HMD 15, the camera 19, and the image providing device 21. The image generating device 100 is connected to the HMD 15, the camera 19, and the image providing device 21 in a wired or wireless manner via the interface unit 13. Specifically, the interface unit 13 includes a multimedia interface such as HDMI (registered trademark) (High-Definition Multimedia Interface), which is a standard for a communication interface that transmits video and audio as digital signals.

[0015] The HMD 15 is a viewing device worn on the user's head to view still images and videos displayed on a display and listen to audio and music output from headphones. The HMD 15 is provided with a gyro sensor and an acceleration sensor. Using these sensors, the HMD 15 measures the position information of the head of the user wearing the HMD 15 and the rotation angle and inclination of the head. The HMD 15 is equipped with a camera unit (not shown). Using the camera unit, the HMD 15 can capture the outside world from the user's viewpoint while the user is wearing the HMD 15. The HMD 15 supplies the captured image to the image generating device 100. The HMD 15 displays an image generated by the image generating device 100 based on the captured image on a display.

[0016] The camera 19 is installed so as to be able to capture images of the game player and surrounding objects such as floors, walls, etc. The camera 19 supplies the captured images of the real space to the image generating device 100.

[0017] The image providing device 21 is configured to be able to receive the generated image from the image generating device 100 and provide the received image to a user. The image providing device 21 is configured to be able to display the image received from the image generating device 100. In addition, the image providing device 21 is configured to be able to upload the received image to a video distribution server (not shown) as a content file.

[0018] The image generating device 100 may be connected to an input unit (not shown) for inputting operation information by a user. The image generating device 100 may have a communication interface for performing data communication with the outside via a communication network.

[0019] FIG. 2 shows an example of the use of this embodiment by a user. In the example of FIG. 2, user A is a player who plays a game and wears the HMD 15. In the example of FIG. 2, user B is a spectator who watches user A play in the game and watches the play image of user A displayed by the image providing device 21. In addition, in the example of FIG. 2, the camera 19 is arranged so as to photograph user A from the front of user A. However, this arrangement is not limited to this, and the camera 19 may be arranged so as to photograph user A from any position such as directly to the side, directly behind, or diagonally to the side of user A. As will be described later, this embodiment makes it possible to more accurately express occlusion between the player's body and a virtual object. Therefore, for example, as shown in FIG. 2, this embodiment is suitable for a case where a spectator watches a player's gameplay from a third-person viewpoint different from the player's viewpoint. In addition, this embodiment is suitable for a case where a player of a game photographs his / her own gameplay with the third-person viewpoint camera 19 and distributes the gameplay to other users via the image providing device 21.

[0020] Fig. 3 is a configuration diagram of the image generating device of this embodiment. The image generating device 100 includes a player recognition unit 102, a captured image acquisition unit 101, a viewpoint acquisition unit 103, a skeleton generation unit 104, a mesh generation unit 105, an image generation unit 110, and an HDMI transmission / reception unit 106. The image generation unit 110 includes a rendering unit 111, a superimposition unit 112, a post-processing unit 113, and a distortion processing unit 114. Fig. 3 is a block diagram focusing on functions, and these functional blocks can be realized in various forms using only hardware, only software, or a combination of both (the same applies to Fig. 9 described later).

[0021] Hereinafter, this embodiment will be described by taking as an example a case where an image from a third-person viewpoint is generated. Specifically, as in the example of Fig. 2, a case where an image from the viewpoint of a camera 19 is generated so that a user B can watch a game play of a user A will be described. It is assumed that an image of the same game play (a game play of a user A) generated from each viewpoint is supplied to the HMD 15 and image providing device 21 of the user A (player).

[0022] The captured image acquisition unit 101 acquires captured images of the real space. In this embodiment, the captured images of the real space are acquired from the camera 19 via the HDMI transmission / reception unit 106. The captured image acquisition unit 101 supplies each of the acquired captured images to the player recognition unit 102, the viewpoint acquisition unit 103, and the superimposition unit 112.

[0023] The player recognition unit 102 recognizes the body of the player. Specifically, the player recognition unit 102 recognizes the position and posture of each body part of the player's body based on the position and posture of each body part of the user A in the captured image acquired by the captured image acquisition unit 101. For example, the player recognition unit 102 recognizes the player's body by estimating the position and posture of each body part of the player using a trained machine learning model based on the captured image of the camera 19. For example, the player recognition unit 102 can recognize the player's body using an open pose or the like. The player recognition unit 102 supplies the recognition result to the skeleton generation unit 104.

[0024] The viewpoint acquisition unit 103 acquires viewpoint information including the viewpoint position and line of sight direction of the camera 19 based on the captured image supplied from the camera 19 via the captured image acquisition unit 101. The viewpoint acquisition unit 103 supplies each piece of acquired viewpoint information to the rendering unit 111. The viewpoint information of the camera 19 is an example of viewpoint information of a spectator that is different from the viewpoint of a player.

[0025] The skeleton generation unit 104 generates a skeleton of the player's body based on the recognition result of the player recognition unit 102. For example, the skeleton generation unit 104 generates the skeleton by estimating the position and posture of each part of the player's skeleton using a trained machine learning model based on the recognition result of the position and posture of each body part of the player's body. For example, the skeleton generation unit 104 can generate the skeleton using an open pose or the like.

[0026] FIG. 4A shows a skeleton generated by the skeleton generating unit 104. The example of FIG. 4A shows a case where the user A is viewed from the viewpoint (front view) of the camera 19 in FIG. 2 (the same applies to the following FIG. 4B and FIG. 4C). The skeleton 30 in FIG. 4A has a plurality of nodes 31 indicating each end and each joint of the player's body. Specifically, the skeleton 30 includes a head node 31a, a chest node 31b, a waist node 31c, a shoulder node 31d, an elbow node 31e, a wrist node 31f, a hand node 31g, a knee node 31h, an ankle node 31i, and a foot node 31j of the player. Adjacent nodes 31 are connected by bones 32. In this embodiment, the skeleton generating unit 104 generates the skeleton 30 by estimating the position and orientation of each node 31 corresponding to the player's body part with respect to a reference position and orientation (for example, an initial position and orientation at the start of the game). The estimated position and orientation data of each node 31 is stored in the storage unit 12. Therefore, it is possible to accurately reflect the positional relationship in the depth direction for each part of the player's body. The skeleton generation unit 104 supplies the generated skeleton to the mesh generation unit 105.

[0027] The mesh generation unit 105 generates a mesh structure of the player that reflects the generated skeleton. For example, the mesh generation unit 105 generates a mesh structure 33 of the player by modeling the skeleton with a mesh structure based on the shape of the player in the acquired photographed image of the real space (see FIG. 4B). The modeling of the mesh structure of the player is performed by a known method. The mesh generation unit 105 supplies the generated mesh structure of the player to the rendering unit 111.

[0028] The rendering unit 111 renders the player's mesh structure and the virtual object when viewed in the line of sight from the viewpoint position of the viewpoint information according to the viewpoint information of the camera 19. Specifically, the rendering unit 111 renders the virtual object and stores the color value in the pixel buffer 121, and renders the player's mesh structure, for example, in white (RGB(255,255,255)) or gray (RGB(128,128,128)) and stores the rendered value in the pixel buffer 121. The rendering unit 111 renders virtual objects that can be attached to at least a part of the player's mesh structure, such as the player's avatar, costume, and accessories, as virtual objects to be superimposed on the player's mesh structure.

[0029] When the rendering unit 111 renders the player's mesh structure, the depth value (mesh depth value) of the player's mesh structure is written to a depth buffer for the player's mesh structure (called a "mesh depth buffer") 122. If there is another virtual object in front of the player's mesh structure, then either the mesh depth value is not written to that pixel in the mesh depth buffer 122, or the mesh depth value is overwritten and erased when the other virtual object is rendered. As a result, an area is generated that contains only the portion where the player's mesh structure is drawn.

[0030] When the rendering unit 111 renders a virtual object, the depth value of the virtual object (scene depth value) is written to a depth buffer for virtual space rendering (called the "scene depth buffer") 123, and the anteroposterior relationship between virtual objects is determined. For pixels where no virtual object is rendered, no specific depth value is written in the scene depth buffer 123, and the scene depth value is infinity (undefined).

[0031] Furthermore, the rendering unit 111 renders an expression related to light by a virtual object to be superimposed on the mesh structure of the player. Examples of the expression related to light of a virtual object include a shadow cast by a virtual object on the mesh structure of the player, a reflection of a virtual object on the mesh structure of the player, semi-transparency of a virtual object, and a lighting expression on the mesh structure of the player by a virtual light source. For example, shadow mapping can draw a shadow or reflection using a method of projecting a depth map from a light source onto a plane or a method such as ray tracing. By superimposing a semi-transparent image of a shadow or reflection of a virtual object on the mesh structure of the user, it is possible to express a shadow or reflection of a virtual object on the mesh structure of the user. Since the mesh structure of the user is rendered in a solid white color, it can be distinguished from an area where a shadow or reflection is drawn. In this way, an expression related to light by a virtual object superimposed on the mesh structure of the player is given to the image.

[0032] The rendering unit 111 provides the rendered mesh structure of the player and the rendered virtual object to the superimposition unit 112.

[0033] The superimposing unit 112 superimposes the captured image on the mesh structure of the player rendered by the rendering unit 111. Specifically, the superimposing unit 112 superimposes the captured image on an area where the scene depth value is infinite and an area where the mesh depth value is written. The area where the mesh depth value is written is superimposed on the captured image while leaving color information of the shadow of the virtual object and lighting expression (reflection, etc.). The superimposing unit 112 superimposes the virtual object on the mesh structure of the player on which the captured image is superimposed.

[0034] Please refer to Fig. 4C. The superimposing unit 112 generates the player's avatar 34 by superimposing the virtual object of the avatar so as to be pasted on the mesh structure of the player. Here, in the example of Fig. 4C, the entire body of the player is superimposed by the virtual object of the avatar, but this is not limited thereto, and a part of the player's body may be superimposed by the virtual object. In this case, the mesh structure of the player on which the photographed image is superimposed becomes visible from places other than the part superimposed by the virtual object.

[0035] The superimposing unit 112 supplies the superimposed image to a post-processing unit 113 .

[0036] The post-processing unit 113 executes a process for applying light-related expression by a virtual object to the superimposed image. The post-processing unit 113 also executes a process for applying a drawing effect according to at least one of the position and the posture of the mesh structure of the player. The drawing effect includes, for example, a motion blur according to a change in the position of the mesh structure of the player, and an aura rising from the mesh structure of the player according to the posture of the mesh structure of the player. In addition to these, the post-processing unit 113 may perform post-processing such as depth of field adjustment, tone mapping, and anti-aliasing to make the image look natural and smooth. The post-processing unit 113 supplies the image that has been subjected to the post-processing to the distortion processing unit 114.

[0037] The distortion processing unit 114 performs a process of distorting the post-processed image by modifying the image in accordance with the distortion caused by the optical system of the camera 19. The distortion processing unit 114 supplies the image that has been subjected to the distortion process to the HDMI transmitting / receiving unit 106.

[0038] In this manner, the image generation unit 110 renders the mesh structure and virtual object of the player for each of users A and B when viewed in the viewpoint direction from the viewpoint position of each of users A and B. The image generation unit 110 generates an image by superimposing the rendered virtual object on the rendered mesh structure.

[0039] The HDMI transmission / reception unit 106 transmits the image generated by the image generation unit 110 to the image providing device 21 according to HDMI. The image transmitted from the image generating device 100 is displayed on the display of the image providing device 21. This allows user B to see an image on which a virtual object is superimposed on the player. This image may be an AR image on which a video see-through image is superimposed as the background image of the player. Alternatively, this image may be a VR image on which a virtual object showing a virtual space is superimposed as the background image of the player.

[0040] Now, with reference to Figs. 5 to 8, an image in which a virtual object is superimposed on an image of a player playing a game will be described.

[0041] Fig. 5 shows an example of an image captured by the camera 19. The image in Fig. 5 is an image of a user A playing a game captured by the camera 19. In the example in Fig. 5, the user A stands in front of the camera 19, and the face 400 of the user A is mainly captured in the image.

[0042] Fig. 6 shows an example of an image in which a virtual object is superimposed on the captured image of Fig. 5. As shown in Fig. 6, the area surrounding user A's face 400 is superimposed with a virtual object 500 wearing a bear headgear. This allows user B to view an image in which the virtual object 500 wearing a bear headgear is drawn around user A's face 400 via image providing device 21.

[0043] FIG. 7 is a diagram showing an example of occlusion expressed by a conventional method. For comparison, an example in which occlusion and depth are not appropriately expressed will be described with reference to FIG. 7. The example in FIG. 7 shows a state in which user A is trying to hide his face 400 with his hand 600. When user A brings his hand 600 in front of his face 400 so as to hide his face 400 with his hand 600, the hand 600 is reflected in the captured image at a position on the front side of the face 400 of user A in the drawing. At this time, the virtual object 500 of a bear headgear that should be on the back side in FIG. 7 may be superimposed on a part of the hand 600 of user A that should be on the front side in FIG. 7. This is because the positional relationship between the virtual object 500 of the bear headgear and a part of the hand 600 of user A may not be correctly determined. Therefore, in the conventional method, inappropriate occlusion occurs in which a part of the hand 600 of user A that should be visible is hidden by the virtual object 500 of the bear headgear. As a result, the depth is not properly expressed, and the sense of unity between the player's body and the virtual object is lost, resulting in an unnatural image. Therefore, it is necessary to correctly express the positional relationship between each part of the player's body and the image superimposed on the player.

[0044] FIG. 8 is a diagram showing an example of occlusion represented by the method of this embodiment. The image generating device 100 of this embodiment generates a mesh structure of a player reflecting the skeleton of the player's body, and superimposes a virtual object on the mesh structure of the player. As described above, the skeleton accurately reflects the positional relationship including the depth of each part of the player's body. Therefore, according to this embodiment, it is possible to more appropriately represent occlusion between the player's body and the virtual object superimposed on the mesh structure of the player. As a result, the depth is appropriately represented, a sense of unity is obtained between the player's body and the virtual object, and a natural image can be provided.

[0045] In this embodiment, the viewpoint information includes the viewpoint position and viewpoint direction of a spectator different from the player. With this configuration, it is possible to provide the spectator with an image from the viewpoint of a third party (spectator) in which the depth of the player is appropriately expressed.

[0046] In this embodiment, the image generating unit 110 generates an image so as to add an expression related to light by a virtual object superimposed on the mesh structure of the player. Conventionally, it has not been possible to accurately reflect the expression related to light by a virtual object on the body of a player. According to this configuration, since the mesh structure of a player generated based on a skeleton accurately reflects the positional relationship of each part of the player's body, it is possible to provide an image in which the expression related to light by a virtual object is more accurately reflected on the body of the player.

[0047] In this embodiment, the image generation unit 110 generates an image so as to impart a rendering effect according to at least one of the position and posture of the mesh structure of the player. With this configuration, the mesh structure of the player appropriately reflects the positional relationship of each part of the player's body, so that rendering effects such as motion blur can be more appropriately expressed.

[0048] In this embodiment, a photographed image is superimposed on a mesh structure of a player, and a virtual object is superimposed on the mesh structure of the player on which the photographed image is superimposed. Therefore, with this configuration, it is possible to more accurately represent occlusion between the part of the player's body on which the virtual object is superimposed and other parts of the player's body that reflect the player's real-life appearance.

[0049] <Second embodiment> The second embodiment of the present disclosure will be described below. In the drawings and description of the following embodiment, the same or equivalent components and members as those in the first embodiment are denoted by the same reference numerals. Descriptions that overlap with the first embodiment will be omitted as appropriate, and the description will focus on the configurations that differ from the first embodiment.

[0050] 9 is a configuration diagram of an image generating device according to the second embodiment. The image generating device 100 according to this embodiment further includes a depth acquisition unit 107 and a real space depth buffer 124. In this embodiment, an AR image on which a video see-through image is superimposed is provided as a background image for the player.

[0051] The depth acquisition unit 107 acquires depth information of the real space. In this embodiment, the depth information of the real space is acquired from the camera 19. The depth information of the real space may be acquired using a depth sensor using, for example, an infrared pattern, structured light, or TOF (Time Of Flight). The depth acquisition unit 107 supplies the acquired depth information to the mesh generation unit 105.

[0052] The mesh generation unit 105 of this embodiment generates a mesh structure of the real space including a mesh structure of the player. Specifically, the mesh generation unit 105 generates the mesh structure of the real space by modeling the real space with a mesh structure based on the depth information of the real space. The modeling of the mesh structure of the real space is performed by a known method. The mesh structure of the real space includes walls, floors, ceilings, stationary objects, etc. of the real space.

[0053] The rendering unit 111 renders the mesh structure of the real space, for example, in white (RGB(255,255,255)), and stores the rendered result in the pixel buffer 121. When the rendering unit 111 renders a real object, the depth value of the real object (called a "real space depth value") is written to a real space depth buffer 124 for real space rendering. The real space depth value is used when determining the anteroposterior relationship between real objects. The real space depth value is generated based on the depth information of the real space.

[0054] According to this embodiment, it is possible to appropriately express the positional relationship of the mesh structure in real space including the mesh structure of the player. As a result, it is possible to more appropriately express the occlusion between the virtual object superimposed on the mesh structure of the player and the object (such as a ball) existing around it. As a result, it is possible to more accurately express the depth of the mesh structure in real space.

[0055] <Third embodiment> The third embodiment of the present disclosure will be described below.

[0056] FIG. 10A shows a mesh structure of a player on which a photographed image of real space is superimposed. Consider a case where a virtual object of a leg of a mechanical structure (see FIG. 10B) is superimposed on the leg of the mesh structure of the player in FIG. 10A. As shown in FIG. 10B, if the superimposed virtual object of the leg of the mechanical structure is smaller than the leg of the mesh structure, the leg of the mesh structure may protrude from the virtual object of the leg of the mechanical structure. In this case, the leg of the mesh structure and the virtual object of the leg of the mechanical structure do not fit together, which tends to look unnatural.

[0057] Therefore, in the case where a portion of the mesh structure of the player on which a virtual object is superimposed extends beyond the superimposed virtual object, the post-processing unit 113 of this embodiment performs processing to distort the space around the skeleton in that portion so that the superimposed portion does not extend beyond the virtual object. Here, processing is performed to distort this space in at least one of the horizontal and vertical directions. This processing will be described below with reference to Figures 10C and 10D.

[0058] Please refer to FIG. 10C. In the process of FIG. 10C, first, the space around the legs of the mesh structure of the player is distorted along the bones 32 (see FIG. 4) that respectively connect the waist node 31c, the knee node 31h, the ankle node 31i, and the foot node 31j of the skeleton. In this embodiment, a process is performed to attract pixels in the space around the legs toward the bones 32. The closer to the bones, the greater the amount of attraction of pixels. This makes it possible to make the legs of the mesh structure of the player thinner.

[0059] Please refer to Fig. 10D. After this processing, the virtual object of the leg of the mechanical structure is superimposed on the leg of the mesh structure of the player. As shown in Fig. 10D, only the virtual object of the leg of the mechanical structure is visible on the leg of the player.

[0060] With this configuration, even when a virtual object that is small compared to parts of the player's body is superimposed, the virtual object can be made to fit the mesh structure of the player, making it possible to reduce the sense of incongruity caused by the difference in size between the player's mesh structure and the virtual object.

[0061] In this embodiment, the case where the mesh structure of the player protrudes from the virtual object on which it is superimposed is taken as an example, but the present invention is not limited to this. For example, if there is a very heavy object on a part of the mesh structure of the player (e.g., on the head), the space around the player may be distorted so as to crush the part of the mesh structure of the player. Also, for example, if the player's avatar itself is set as a very heavy avatar (e.g., an alien avatar like a giant rock), the space near the floor on which the avatar is standing may be distorted. In this way, it is possible to make the mesh structure of the player look smaller in accordance with the characteristics of the virtual object.

[0062] <Modification> At least a part of the functions of the image generating device 100 may be implemented in the HMD 15 or the image providing device 21. Alternatively, at least a part of the functions of the image generating device 100 may be implemented in a server connected to the image generating device 100 via a network.

[0063] The image generating device 100 may further be connected to a server via a network. In this case, the server may provide the image generating device 100 with an online application such as a game in which multiple users can participate via the network. The HMD 15 or the image providing device 21 may be connected to a computer or a mobile terminal instead of the image generating device 100.

[0064] In the above embodiment, the user A is a player who plays the game, and the user B is a spectator who watches the play of the user A in the game, but this is not limited to the above. For example, both the users A and B may be players who play the game.

[0065] In the above embodiment, the player recognition unit 102 recognizes the player's body based on the captured image supplied from the camera 19, but is not limited to this. For example, the player's body may be recognized by detecting the positions and postures of the user's body parts using a tracker.

[0066] In the above embodiment, an example has been shown in which the user A playing the game wears the HMD 15, but the HMD 15 does not have to be provided. In this case, for example, an image of the image generating device 100 may be supplied to a separately provided display, and the user A may play the game based on the image displayed on this display.

[0067] In the above embodiment, an example using the camera 19 and the image providing device 21 has been described, but these do not have to be used. In this case, for example, the user B also wears the HMD 15, and an image captured by the HMD 15 of the user B is supplied to the image generating device 100 instead of an image captured by the camera 19, and an image generated by the image generating device 100 based on the viewpoint information of the HMD 15 is displayed on the HMD 15.

[0068] 2, the camera 19 and the image providing device 21 are separate bodies, but they may be configured as one body. In this case, for example, a personal computer having a photographing function and a display function may be used.

[0069] In the above embodiment, the photographed image is superimposed on the player's virtual object, but it is also possible to superimpose only the virtual object on the mesh structure without superimposing the photographed image.

[0070] The processes for the light-related expression and the rendering effects in the post-processing section may not be executed as necessary.

[0071] In the above explanation, examples of post-processing include depth of field adjustment, tone mapping, and anti-aliasing, but distortion processing, simple enlargement / reduction, trapezoidal transformation, and the like can also be called post-processing.

[0072] The present disclosure has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing step, and that such modifications are also within the scope of the present disclosure. [Explanation of symbols]

[0073] 10 image generation system, 15 HMD, 100 image generation device, 101 captured image acquisition unit, 102 player recognition unit, 103 viewpoint acquisition unit, 104 skeleton generation unit, 105 mesh generation unit, 106 HDMI transmission / reception unit, 107 depth acquisition unit, 110 image generation unit, 111 rendering unit, 112 superimposition unit, 113 post-processing unit, 114 distortion processing unit, 121 pixel buffer, 122 mesh depth buffer, 123 scene depth buffer, 124 real space depth buffer.

Claims

1. a player recognition unit that recognizes the body of a player; a viewpoint acquisition unit for acquiring viewpoint information including a viewpoint position and a viewpoint direction; a mesh generation unit that generates a mesh structure of a player reflecting a body skeleton of the player based on a recognition result of the player recognition unit; an image generation unit that generates an image by rendering a mesh structure of the player and a virtual object that can be attached to at least a part of the mesh structure when viewed in the viewpoint direction from the viewpoint position in the viewpoint information, and pasting and superimposing the rendered virtual object on the rendered mesh structure of the player; 13. An image generating device comprising:

2. a captured image acquisition unit for acquiring a captured image of a real space; A depth acquisition unit that acquires depth information of the real space, 2. The image generating device according to claim 1, wherein the image generating unit generates the image based on the captured image and the depth information so as to represent occlusion between an object existing in the real space and the virtual object superimposed on a mesh structure of the player.

3. The image generating device according to claim 1 , wherein the image generating section generates the image so as to impart a drawing effect according to at least one of a position and an orientation of the mesh structure of the player.

4. The image generating device according to claim 1 , wherein the viewpoint information includes a viewpoint position and a viewpoint direction of a spectator different from a viewpoint of the player.

5. A photographed image acquisition unit for acquiring a photographed image of a real space is further provided, 5. The image generating device according to claim 1, wherein the image generating unit generates the image by superimposing the photographed image of the player on a mesh structure of the player, and superimposing a virtual object on the mesh structure of the player on which the photographed image is superimposed.

6. The image generating device according to claim 1 , wherein the image generating unit generates the image so as to impart an expression relating to light by the virtual object superimposed on a mesh structure of the player.

7. The image generating device according to claim 1 , wherein the image generating unit generates the image so as to represent occlusion between the body of the player and the virtual object superimposed on a mesh structure of the player.

8. The image generating device according to claim 1 , wherein the image generating unit generates the image by distorting a space around the skeleton in a portion of a mesh structure of the player on which a virtual object is superimposed.

9. 9. The image generation device according to claim 1, wherein, when a portion of a mesh structure of the player on which a virtual object is superimposed protrudes from the superimposed virtual object, the image generation unit generates the image by distorting a space around the skeleton at the portion so that the portion does not protrude from the superimposed virtual object.

10. 10. The image generating device according to claim 1 , wherein the image generating unit generates the image by superimposing the virtual object on a mesh structure of the player when viewed from a viewpoint of another player different from the player.

11. Recognizing a body of a player; Obtaining viewpoint information relating to a viewpoint position and a viewpoint direction; generating a mesh structure of the player reflecting a skeleton of the player's body based on a recognition result of the recognizing step; generating an image by rendering a mesh structure of the player and a virtual object attachable to at least a part of the mesh structure when viewed in the viewpoint direction from the viewpoint position in the viewpoint information, and pasting and superimposing the rendered virtual object on the rendered mesh structure of the player; 13. An image generating method comprising:

12. Recognizing a body of a player; Obtaining viewpoint information relating to a viewpoint position and a viewpoint direction; generating a mesh structure of the player reflecting a skeleton of the player's body based on a recognition result of the recognizing step; generating an image by rendering a mesh structure of the player and a virtual object attachable to at least a part of the mesh structure when viewed in the viewpoint direction from the viewpoint position in the viewpoint information, and pasting and superimposing the rendered virtual object on the rendered mesh structure of the player; A program for causing a computer to execute the following.

Images