Three-dimensional virtual character generation apparatus, three-dimensional virtual character generation method, and three-dimensional virtual character generation program product

Abstract

The present invention provides a device and method for easily generating a high-quality three-dimensional virtual character based on captured image data. The device includes a base model generation unit (1), a shape information generation unit (2), a two-dimensional image generation unit (3), a feature point extraction unit (4), a feature point addition unit (5), a skeleton correspondence determination unit (6), a skeleton adjustment unit (7), and a surface adjustment unit (8). The base model generation unit (1) generates a general base model. The shape information generation unit (2) generates three-dimensional surface shape information of an object based on captured data of the object, such as a person, which is the generation target of the three-dimensional virtual character. The two-dimensional image generation unit (3) converts the three-dimensional surface shape information into a two-dimensional image. The feature point extraction unit (4) extracts feature points from the two-dimensional image. The feature point addition unit (5) adds the extracted feature points to the three-dimensional surface shape information. The skeleton correspondence determination unit (6) determines the correspondence between the feature points added to the three-dimensional surface shape information and the skeleton structure of the general base model. The skeleton adjustment unit (7) adjusts the three-dimensional distribution of the skeleton structure of the general base model to match the three-dimensional distribution of the feature points. The surface adjustment unit (8) adjusts the surface shape of the general base model to match the three-dimensional shape of the surface of the object included in the three-dimensional shape information.

Description

TECHNICAL FIELD

[0001] The present application relates to a three-dimensional virtual character generation device and the like that creates three-dimensional computer graphics data based on captured data of a person or the like as an object. BACKGROUND

[0002] In recent years, with the improvement in the processing capacity of electronic computers such as computers and the like, the development of games and video contents using three-dimensional computer graphics has been promoted. That is, a well-known computer game series that was expressed by two-dimensional animation in the past has been reproduced using 3DCG, and in addition, a movie adapted from a classic fairy tale has also been produced using 3DCG and has been a hit worldwide, and now three-dimensional computer graphics has been widely used as a de facto standard in the expression method of video contents.

[0003] It has been proposed to use a realistic three-dimensional computer graphics that uses a person or the like that actually exists as a model, as a form of development of three-dimensional computer graphics. It uses a virtual character (a character as a doppelganger) created by imitating a person who plays a game as a character of a computer game or the like or an opponent person who plays a game together, and by using the virtual character, it is possible to enjoy a game with a more realistic experience by improving the immersion in the world of the work.

[0004] Patent Documents 1 and 2 each disclose an example of a virtual character created by imitating a person who is a player himself or an opponent player, in a computer game that uses a head-mounted display to express a virtual space.

[0005] Patent Document 1: Japanese Laid-Open Patent Publication No. 2019-012509

[0006] Patent Document 2: Japanese Laid-Open Patent Publication No. 2019-139673 SUMMARY

[0007] PROBLEMS TO BE SOLVED BY THE INVENTION

[0008] However, neither of Patent Documents 1 and 2 discloses a method of creating a virtual character that imitates a person such as a player. In actual games, a common method is as follows: for example, a body part is prepared with a body of an ordinary size, and a head part is prepared with a structure in which a face photo is attached to a sphere or an ellipsoid or a structure in which a self-portrait drawing prepared by the user himself or herself is attached by a drawing tool or the like. However, the body size and the head part of the virtual character described above are greatly different from those of an actual player, and it is difficult for a game player to have an immersive feeling in the world of the work.

[0009] On the other hand, it is unrealistic to create a virtual character of a user by a method similar to general three-dimensional computer graphics. The reason is that, when a character is created using three-dimensional computer graphics, a modeling operation of a mesh structure corresponding to human skin or the like is required first, then a skeleton structure operation of a skeleton corresponding to a human skeleton is required, and a skinning operation of a function corresponding to human muscles, i.e., an operation of associating a skeleton and a skin, is required.

[0010] In the above operations, for example, the modeling operation is to generate a three-dimensional shape by synthesizing a user's posture image taken from a plurality of directions, and based on this, it is not impossible to perform a modeling operation related to a surface shape and the like automatically to some extent. However, the operations related to an internal structure, i.e., the skeleton structure operation and the skinning operation, cannot be automated by only the image of the user's posture (surface), and these operations can only be manually performed by a skilled technician for a long time. In view of the fact that such a large amount of time and labor is required, it is unrealistic to perform a character creation operation using general three-dimensional computer graphics for each virtual character of a user.

[0011] The present application has been made in view of the above problems, and has an object to provide an apparatus and a method which can easily generate a high-quality three-dimensional virtual character based on photographed image data.

[0012] -Technical Solution to Solve the Technical Problem-

[0013] To achieve the above object, the three-dimensional virtual character generation device according to the technical solution 1 generates a three-dimensional virtual character representing a three-dimensional surface shape by a vertex group and / or a face group based on the photographed data of the object, and is characterized in that the three-dimensional virtual character generation device includes a surface shape generation unit, a skeleton structure estimation unit, a skeleton correspondence relationship determination unit, a skeleton adjustment unit, a surface adjustment unit, and an image addition unit. The surface shape generation unit generates a three-dimensional surface shape based on the photographed data of the object. The skeleton structure estimation unit converts the three-dimensional surface shape of the object into a two-dimensional image and estimates at least a part of the skeleton structure based on the two-dimensional image. The skeleton correspondence relationship determination unit determines the correspondence relationship between the skeleton structure estimated by the skeleton structure estimation unit and the skeleton structure of a general body virtual character in which the three-dimensional surface shape, the skeleton structure, and the correlation between the two are defined. The skeleton adjustment unit adjusts the position of the skeleton structure of the general body virtual character to match the correspondence relationship determined by the skeleton correspondence relationship determination unit while maintaining the connection relationship between the elements of the skeleton structure. The surface adjustment unit moves the vertex group and / or the face group of the general body virtual character to a position consistent with the vertex group and / or the face group representing the three-dimensional surface shape of the object based on the correspondence relationship between the vertex group and / or the face group representing the three-dimensional surface shape of the object and the vertex group and / or the face group representing the three-dimensional surface shape of the general body virtual character after the position of the vertex group and / or the face group is moved by the surface adjustment unit. The image addition unit adds an image processed from the photographed data of the object to the surface of the general body virtual character after the position of the vertex group and / or the face group is moved by the surface adjustment unit.

[0014] In addition, to achieve the above object, the three-dimensional virtual character generation device according to the technical solution 2 is based on the above invention, and is characterized in that the action information, the shape transformation information, and the information related to the vertex group representing the internal shape are defined in the general body virtual character. The action information is related to the position change of the entire skeleton structure and the linkage mode between the elements of the skeleton structure when performing an action. The shape transformation information defines the transformation mode of the vertex group when transforming from a first shape to a second shape in at least a part of the three-dimensional surface shape. The object subjected to the position movement by the surface adjustment unit does not include the vertex group representing the internal shape.

[0015] Further, in order to achieve the above object, the three-dimensional virtual character generation device according to the technical solution 3, based on the above invention, is characterized in that: the three-dimensional virtual character generation device further comprises a surface area judging unit, a basic body type generating unit and a shape changing unit, the surface area judging unit judges a skin exposed area, a clothing area and a hair area in a three-dimensional surface shape based on the shooting data of the object, the basic body type generating unit removes the clothing part from the clothing area judged by the surface area judging unit and generates a surface shape of a basic body type, the shape changing unit causes the surface shape of the general body virtual character to change in shape according to the shape changing information so as to match the basic body type, the surface adjusting unit does not perform surface adjusting processing on at least the clothing area and the hair area, and generates a three-dimensional virtual character, wherein the three-dimensional virtual character does not have information related to clothing and hair in the shooting data of the object.

[0016] Further, in order to achieve the above object, the three-dimensional virtual character generation device according to the technical solution 4, based on the above invention, is characterized in that: the three-dimensional virtual character generation device comprises a clothing area extracting unit, a clothing body generating unit, a body connecting unit, a general shaping unit, an information transmitting unit and a connection releasing unit, the clothing area extracting unit extracts a surface structure, wherein the surface structure is located on the clothing area in a three-dimensional surface shape based on the shooting data of the object, the clothing body generating unit converts the extracted part into a three-dimensional clothing body, the body connecting unit connects the clothing body to a three-dimensional virtual character based on the shooting data, the general shaping unit causes a product formed by connecting the three-dimensional virtual character and the clothing body to change in shape according to the shape changing information so as to conform to a three-dimensional surface shape of the general body virtual character, the information transmitting unit transmits information related to the relevance of the three-dimensional surface shape and the skeleton structure possessed by the general body virtual character and the shape changing information to the clothing body after the shape changing, and the connection releasing unit releases the connection between the clothing body and the three-dimensional virtual character after the information transmitting unit transmits the information.

[0017] Further, in order to achieve the above object, the three-dimensional virtual character generation method according to the technical solution 5 generates a three-dimensional virtual character representing a three-dimensional surface shape by a vertex group and / or a face group based on the photographed data of the object, and is characterized in that the three-dimensional virtual character generation method includes a surface shape generation process, a skeleton structure estimation process, a skeleton correspondence relationship determination process, a skeleton adjustment process, a surface adjustment process, and an image attachment process, the surface shape generation process generates a three-dimensional surface shape based on the photographed data of the object, the skeleton structure estimation process converts the three-dimensional surface shape of the object into a two-dimensional image and estimates at least a part of the skeleton structure based on the two-dimensional image, the skeleton correspondence relationship determination process determines a correspondence relationship between the skeleton structure estimated by the skeleton structure estimation process and a skeleton structure of a general body virtual character in which a three-dimensional surface shape, a skeleton structure, and a correlation between the two are defined, the skeleton adjustment process adjusts the position of the skeleton structure of the general body virtual character to match the correspondence relationship determined by the skeleton correspondence relationship determination process while maintaining a connection relationship between elements of the skeleton structure, the surface adjustment process moves the vertex group and / or the face group of the general body virtual character to a position consistent with the vertex group and / or the face group representing the three-dimensional surface shape of the object based on a correspondence relationship between the vertex group and / or the face group representing the three-dimensional surface shape of the object and the vertex group and / or the face group representing the three-dimensional surface shape of the general body virtual character after the position of the skeleton structure is adjusted, and the image attachment process attaches an image processed from the photographed data of the object to the surface of the general body virtual character after the position of the vertex group and / or the face group is moved by the surface adjustment process.

[0018] Further, in order to achieve the above object, the three-dimensional virtual character generation method according to the technical solution 6 is based on the above invention, and is characterized in that in the general body virtual character, action information, shape transition information, and information related to a vertex group representing an internal shape are defined, the action information is information related to a position change of the entire skeleton structure and a linkage manner between elements of the skeleton structure when an action is performed, the shape transition information defines a transition manner of the vertex group when transitioning from a first shape to a second shape in at least a part of the three-dimensional surface shape, and the object of the position movement in the surface adjustment process does not include the vertex group representing the internal shape.

[0019] Further, in order to achieve the above object, the three-dimensional virtual character generation method according to the technical solution 7, based on the above invention, is characterized in that the three-dimensional virtual character generation method further comprises a surface region judging procedure, a basic body type generating procedure and a shape changing procedure, the surface region judging procedure judges a skin exposed region, a clothing region and a hair region in a three-dimensional surface shape based on the shooting data of the object, the basic body type generating procedure removes the clothing part from the clothing region judged by the surface region judging procedure and generates a surface shape of a basic body type, the shape changing procedure causes the surface shape of the general base body virtual character to change in shape according to the shape changing information so as to match the basic body type, and in the surface adjusting procedure, at least the clothing region and the hair region are not subjected to surface adjusting processing, and a three-dimensional virtual character is generated, wherein the three-dimensional virtual character does not have information related to clothing and hair in the shooting data of the object.

[0020] Further, in order to achieve the above object, the three-dimensional virtual character generation method according to the technical solution 8, based on the above invention, is characterized in that the three-dimensional virtual character generation method further comprises a clothing region extracting procedure, a clothing base body generating procedure, a base body connecting procedure, a general shape changing procedure, an information transmitting procedure and a connection releasing procedure, the clothing region extracting procedure extracts a surface structure, wherein the surface structure is located in the clothing region on a three-dimensional surface shape based on the shooting data of the object, the clothing base body generating procedure converts the extracted part into a three-dimensional clothing base body, the base body connecting procedure connects the clothing base body to a three-dimensional virtual character based on the shooting data, the general shape changing procedure causes a product formed by connecting the three-dimensional virtual character and the clothing base body to change in shape according to the shape changing information so as to conform to a three-dimensional surface shape of the general base body virtual character, the information transmitting procedure transmits information related to the relevance of the three-dimensional surface shape and the skeleton structure possessed by the general base body virtual character and the shape changing information to the clothing base body after the shape change, and the connection releasing procedure releases the connection between the clothing base body and the three-dimensional virtual character after the information transmitting procedure transmits the information.

[0021] Further, in order to achieve the above object, the three-dimensional virtual character generation program according to the technical solution 9 generates a three-dimensional virtual character representing a three-dimensional surface shape by a vertex group and / or a face group based on the photographed data of the object, characterized in that the three-dimensional virtual character generation program causes a computer to execute a surface shape generation step of generating a three-dimensional surface shape based on the photographed data of the object, a skeleton structure estimation step of converting the three-dimensional surface shape of the object into a two-dimensional image and estimating at least a part of a skeleton structure based on the two-dimensional image, a skeleton correspondence relationship discrimination step of discriminating a correspondence relationship of the skeleton structure estimated by the skeleton structure estimation step and a skeleton structure of a general body virtual character in which a three-dimensional surface shape, a skeleton structure, and a correlation between the two are defined, a skeleton adjustment step of adjusting a position of the skeleton structure of the general body virtual character so as to match the correspondence relationship discriminated by the skeleton correspondence relationship discrimination step while maintaining a connection relationship among elements of the skeleton structure, a surface adjustment step of moving a vertex group and / or a face group of the general body virtual character to a position coinciding with a vertex group and / or a face group representing the three-dimensional surface shape of the object based on a correspondence relationship of the vertex group and / or the face group representing the three-dimensional surface shape of the object and the vertex group and / or the face group representing the three-dimensional surface shape of the general body virtual character after the position of the skeleton structure is adjusted, and an image addition step of adding an image processed from the photographed data of the object to a surface of the general body virtual character after the position of the vertex group and / or the face group is moved by the surface adjustment step.

[0022] Further, in order to achieve the above object, the three-dimensional virtual character generation program according to the technical solution 10 is characterized in that, in the general body virtual character, action information, shape transition information, and information related to a vertex group representing an internal shape are defined, the action information is information related to a position change of the entire skeleton structure and a linkage manner among elements of the skeleton structure when an action is performed, the shape transition information defines a transition manner of a vertex group when a transition from a first shape to a second shape in at least a part of a region of a three-dimensional surface shape, and an object of the position movement in the surface adjustment step does not include the vertex group representing the internal shape.

[0023] Further, in order to achieve the above object, the three-dimensional virtual character generation program according to the technical solution 11, based on the above invention, is characterized in that the three-dimensional virtual character generation program further comprises a surface region judging step, a basic body type generating step, and a shape changing step, the surface region judging step judges a skin exposed region, a clothing region, and a hair region in a three-dimensional surface shape based on the photographed data of the object, the basic body type generating step removes a clothing portion from the clothing region judged by the surface region judging step and generates a surface shape of a basic body type, the shape changing step causes a surface shape of the general body type virtual character to change in shape in accordance with the shape changing information so as to match the basic body type, and in the surface adjusting step, at least the clothing region and the hair region are not subjected to surface adjustment processing, and a three-dimensional virtual character is generated, wherein the three-dimensional virtual character does not have information related to clothing and hair in the photographed data of the object.

[0024] Further, in order to achieve the above object, the three-dimensional virtual character generation program according to the technical solution 12, based on the above invention, is characterized in that the three-dimensional virtual character generation program further comprises a clothing region extracting step, a clothing body generating step, a body connecting step, a general shape changing step, an information transferring step, and a connection releasing step, the clothing region extracting step extracts a surface structure, wherein the surface structure is located on the clothing region in a three-dimensional surface shape based on the photographed data of the object, the clothing body generating step converts the extracted portion into a three-dimensional clothing body, the body connecting step connects the clothing body to a three-dimensional virtual character based on the photographed data, the general shape changing step causes a product of the three-dimensional virtual character connected with the clothing body to change in shape in accordance with the shape changing information so as to conform to a three-dimensional surface shape of the general body type virtual character, the information transferring step transfers information related to the relevance of the three-dimensional surface shape and the skeleton structure possessed by the general body type virtual character and the shape changing information to the clothing body which has changed in shape, and the connection releasing step releases the connection of the clothing body to the three-dimensional virtual character after the information transferring step transfers the information.

[0025] Effects of the Invention

[0026] According to the present invention, the effect that a high-quality three-dimensional computer graphics based on a real shot image can be easily generated can be achieved. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a schematic view showing the configuration of a three-dimensional virtual character generation apparatus according to the first embodiment.

[0028] Figure 2 is a flowchart showing the action of the three-dimensional virtual character generation apparatus according to the first embodiment.

[0029] Figure 3 is a schematic diagram showing the configuration of the three-dimensional virtual character generation apparatus according to the second embodiment.

[0030] Figure 4 is a flowchart showing the action of the three-dimensional virtual character generation apparatus according to the second embodiment.

[0031] Figure 5 is a schematic diagram showing the configuration of the three-dimensional virtual character generation apparatus according to the third embodiment.

[0032] Figure 6 is a flowchart showing the action of the three-dimensional virtual character generation apparatus according to the third embodiment.

[0033] Figure 7 is a schematic diagram showing the configuration of the three-dimensional virtual character generation apparatus according to the modification.

[0034] Figure 8 is a schematic diagram showing the configuration of the three-dimensional virtual character generation apparatus according to the fourth embodiment.

[0035] Figure 9 is a flowchart showing the action of the three-dimensional virtual character generation apparatus according to the fourth embodiment. DETAILED DESCRIPTION

[0036] Hereinafter, the embodiments of the present application will be described in detail with reference to the accompanying drawings. In the following embodiments, examples most suitable for embodying the present application are described, and of course, the contents of the present application should not be construed as being limited to the specific examples shown in the embodiments. As long as the same action and effect are obtained, configurations other than the specific configurations shown in the embodiments are of course included in the technical scope of the present application.

[0037] (First Embodiment)

[0038] First, the three-dimensional virtual character generation apparatus according to the first embodiment will be described. The three-dimensional virtual character generation apparatus according to the first embodiment is configured to generate a three-dimensional virtual character of an object based on photographing data obtained by photographing the object, using a general body on which basic information as a three-dimensional computer graphics is recorded.

[0039] Figure 1 is a schematic diagram showing the configuration of the first embodiment. As shown in the diagram, the three-dimensional virtual character generation apparatus according to the first embodiment includes a photographing data acquisition unit 100, a general body data acquisition unit 200, a three-dimensional virtual character generation unit 300, and a display unit 400. Figure 1As shown, the three-dimensional virtual character generation apparatus according to the first embodiment includes a base model generation section 1 that generates a general base model, a shape information generation section 2 that generates three-dimensional surface shape information relating to a surface shape of an object, such as a person, which is a generation target of a three-dimensional virtual character, based on photographing data of the object, a two-dimensional image generation section 3 that converts the three-dimensional shape information into a two-dimensional image, a feature point extraction section 4 that performs feature point extraction of the two-dimensional image, a feature point addition section 5 that adds the extracted feature points to the three-dimensional surface shape information, a skeleton correspondence relation determination section 6 that determines a correspondence relation between the feature points added to the three-dimensional surface shape information and a skeleton structure of the general base model, a skeleton adjustment section 7 that adjusts a three-dimensional distribution of the skeleton structure of the general base model so as to conform to a three-dimensional distribution of the feature points, a surface adjustment section 8 that adjusts a surface shape of the general base model so as to conform to a three-dimensional shape of a surface of the object included in the three-dimensional shape information, a texture information generation section 9 that generates information relating to a surface texture of the object based on three-dimensional image data of the object, a surface texture imparting section 10 that imparts a texture representation to a surface of the general base model based on the texture information, and a surface image addition section 11 that adds image information generated based on the photographing data of the object to the surface of the general base model.

[0040] The base model generation section 1 generates a general base model, which is a material for generating a three-dimensional virtual character of a generation target object. The general base model is, for example, three-dimensional computer graphics information of a person or the like composed of an average body size, and specifically includes at least three-dimensional shape information relating to a three-dimensional surface shape, skeleton information relating to a skeleton structure for controlling actions and the like, correlation information relating to a correlation between the surface shape and the skeleton structure, and two-dimensional mapping information obtained by two-dimensionally expanding a surface of the three-dimensional surface shape. The base model generation section 1 has a function of generating the above information and generating a general base model that has no individuality in terms of body size, facial features, and the like.

[0041] The surface shape information is information relating to a surface shape of a three-dimensional computer graphics, which corresponds to a shape of a human skin or the like. The form of the information can be a form in which an entire surface is defined as a collection of minute units such as voxels and position information of each of the minute units is recorded, but from the viewpoint of reducing a load on information processing, it is preferable to be a form in which a three-dimensional shape of a surface is represented by performing so-called modeling processing and defining a prescribed number of vertices and a connection mode between the vertices. After the modeling processing, a side connecting vertices to each other is formed based on information relating to the vertices and the connection mode between the vertices, a region surrounded by three or more sides is defined as a face (polygon), and a surface shape is determined by a collection (mesh) of the faces.

[0042] However, the application object of the present application is not limited to this, and the shape information can be used as the surface shape information in the present application as long as it is information including position information of a plurality of vertices and / or a plurality of faces arranged in correspondence with a surface shape. Further, for a part of the plurality of vertices included in the surface shape information, not only position information and information related to a connection mode between vertices, but also information related to a meaning content of the vertex is recorded. For example, for a vertex of a feature point corresponding to a specific position of an eye, a nose, a mouth, and the like and a detailed positional relationship of each part (an outer corner of an eye, a pupil, a tip of a nose, a corner of a mouth, and the like), information such as "corresponding to an outer corner of a right eye" is added. Further, as the position information of the vertex, a preferable form is any one of information related to an absolute position and information related to a relative position with respect to a skeletal structure composed of a joint part and a bone part. In the present embodiment, the position information includes both of the former and the latter, and as the position of the joint part changes, the position, the length, and the like of the bone part change, the position of each vertex also changes while maintaining a relative positional relationship.

[0043] The skeletal information is information related to an internal structure corresponding to a human skeleton and the like and serving as a reference when an action is produced in a three-dimensional computer graphics and the like. The form of the information can be a skeletal structure composed of a bone and a joint having a prescribed thickness and a prescribed size as in a human skeletal structure, but is preferably a form represented by what is called a skeletal part, which is a collection of a joint part (represented by a point) corresponding to a human joint and the like and a bone part (represented by a line) located between the joint parts and corresponding to a human bone. However, the application object of the present application is not limited to the above information form, and can be other forms as long as it is composed of information related to two parts, one of which is a part (collectively referred to as a "joint part" in the present application) that is parallelly / rotationally movable like a joint and also functions as a fulcrum in a relationship with an adjacent part, and the other of which is a part (collectively referred to as a "bone part") that is only parallelly / rotationally movable like a bone.

[0044] The correlation information is information that specifies the correlation between the skeleton information and the surface shape information, and more specifically, is information that specifies to what extent each vertex that forms the surface shape moves in accordance with the movement of the joint portion and the bone portion included in the skeleton structure. If a configuration is adopted in which the surface shape 100% follows the movement of the joint portion and the bone portion, the movement of a character such as a human will also become like a tin robot, and the character will lack realism. Therefore, when generating a three-dimensional computer graphic of a person or the like, it is preferable to previously set information related to the extent to which each portion of the surface shape follows the movement of the nearby bone portion and joint portion. In the present embodiment, as the correlation information with respect to each vertex that constitutes the surface shape information, information in which numerical information that indicates the followability with respect to the nearby bone portion and / or joint portion is set is used. Note that the generation work of the correlation information is called skin processing, weight editing, or the like, and a weight value is generally also used with respect to the correlation information, but the correlation information in the present application is not limited to this, and includes all information that satisfies the above conditions.

[0045] The two-dimensional mapping information is information in which the position information of each vertex that constitutes the surface shape of the general primitive is converted into a two-dimensional coordinate system and is illustrated like a world terrain on a spherical surface is converted into a two-dimensional world map. The two-dimensional mapping information is obtained by expressing the surface shape constituted by the three-dimensional structure on a two-dimensional coordinate system in order to facilitate the production of surface texture, detailed patterns, and the like. The two-dimensional mapping is generally achieved by UV unwrapping, but the two-dimensional mapping information in the present application is not limited to information obtained by UV unwrapping, and includes all information that satisfies the above conditions.

[0046] Next, the shape information generation section 2 will be described. The shape information generation section 2 is used to generate information related to the surface shape of the object, i.e., three-dimensional surface shape information, from the photographed data of the object. Like the information generated by the primitive generation section 1, the surface shape information generated by the shape information generation section 2 is information that is constituted by position information of a plurality of vertices that are arranged in correspondence with the surface shape, and is generated by automatic or semi-automatic analysis processing of three-dimensional image data.

[0047] The two-dimensional image generation section 3 generates a two-dimensional image from the three-dimensional surface shape information generated based on the captured data of the object. The two-dimensional image can also be generated by the same process as the pixel volume generation section 1 to generate two-dimensional mapping information, but in the present embodiment, the two-dimensional image is generated by projecting the three-dimensional surface shape information in one direction. More specifically, in order to facilitate the feature point extraction process by the feature point extraction section 4 to be described later, it is preferable to project the three-dimensional surface shape information in the normal direction of the front view of the person or the like, i.e., the face of the person or the like, to generate a two-dimensional image. However, the two-dimensional image can also be generated by projecting in other directions, as long as the two-dimensional image can recognize the face structure and the shape of the torso portion to some extent. The two-dimensional image can be generated by projecting in one direction, and two kinds of two-dimensional images can be generated by projecting in different directions.

[0048] The feature point extraction section 4 analyzes the positions of the feature points of the face and the skeletal structure from the two-dimensional shape information generated by the two-dimensional image generation section 3. The feature point extraction section 4 determines the positions of the feature points on the internal structure, such as the joints of the neck, shoulders, elbows, wrists, fingertips, waist, knees, ankles, and the like of the person as the object, and the positions of the feature points on the external surface of the face, such as the eyes, nose, mouth, ears, and the like, by using a two-dimensional image analysis technique such as a pose estimation technique. Note that the specific configuration of the feature point extraction section 4 can also employ other image recognition techniques realized by deep learning, mechanical learning, or the like, and a plurality of image recognition techniques can also be used in combination. Also, the same applies to the two-dimensional image as the object of the determination, and the feature point analysis can be performed using not only one two-dimensional image obtained by projecting in one direction, but also other two-dimensional images obtained by projecting in different directions.

[0049] The feature point addition section 5 adds information related to the feature points extracted by the feature point extraction section 4 to the three-dimensional surface shape information related to the surface shape of the object. In the present embodiment, the feature point addition section 5 has a function of adding information related to the positions and meanings of the feature points, such as points corresponding to the outer corners of the eyes, points corresponding to the knee joints, and the like, as information related to the feature points.

[0050] For example, the normal direction of the face of the person or the like in the three-dimensional image data is set as the Z axis, and the two axes orthogonal thereto, such as the direction in which the person stands and the direction of the straight line connecting the outer corners of the eyes, are set as the X axis and the Y axis. After such setting, the X coordinate and the Y coordinate of the feature points determined in the projection image in the positive direction of the Z axis, i.e., the two-dimensional image, directly correspond to the X coordinate and the Y coordinate in the three-dimensional image data.

[0051] Further, when the feature point is located on the outer surface of the three-dimensional surface shape (for example, a feature point of the face), the Z coordinate of the feature point is the same value as the Z coordinate of the point located on the outer surface at the X coordinate, Y coordinate (both places) on the side of the projection direction. According to this principle, for the position of the feature point on the outer surface, the feature point addition section 5 adds, in addition to the X coordinate, Y coordinate in the two-dimensional image as the position information, the Z coordinate of the point on the three-dimensional outer surface shape at the X, Y coordinate as the position information.

[0052] Further, for a feature point on the internal structure indicating a joint or the like, the Z coordinate is expressed as Z1

[0053] The skeleton correspondence relation discriminating section 6 discriminates the correspondence relation of the feature point on the internal structure among the feature points to which the three-dimensional position information is added and the skeleton structure, that is, the joint portion and the bone portion in the general body. Specifically, the skeleton correspondence relation discriminating section 6 discriminates by comparing and collating the position of the feature point with respect to the entire three-dimensional surface shape of the object and the position of the joint portion and the bone portion with respect to the entire surface shape in the general body, to determine whether a certain feature point related to the internal structure corresponds to which constituent element of the skeleton structure in the general body.

[0054] If the feature point corresponding to the constituent element of the skeleton structure in the general body has not been extracted, the skeleton correspondence relation discriminating section 6 estimates the existence and position of the feature point to be corresponded. For example, when the feature point corresponding to the left knee is extracted by the feature point extracting section 4 and the feature point corresponding to the right knee is not extracted, using the fact that the human body has the left-right symmetry in the skeleton structure, after the feature point corresponding to the right knee in the three-dimensional surface shape of the object is extracted, the correspondence relation of the feature point to the skeleton structure in the general body is discriminated. Specifically, for the feature point located at a position away from the center line of the human body like the right knee, the position in the three-dimensional surface shape is decided by assuming that the feature point is located at a position symmetrical to the left knee with respect to the center line.

[0055] The skeleton adjusting section 7 adjusts the positions of the joint sections and the skeleton sections constituting the skeleton information of the general-purpose body so as to match the skeleton information of the general-purpose body with the distribution of the feature points related to the internal structure extracted by the feature point extracting section 4, while maintaining the connection relationship between the constituent elements such as the joint sections and the skeleton sections. The skeleton adjusting section 7 judges the joint sections and the skeleton sections in the body corresponding to the feature points related to the internal structure of the surface shape information of the object, and converts the position information of the joint sections and the skeleton sections into the same position information as the corresponding feature points. For the joint sections and the skeleton sections for which the correspondence relationship with the feature points related to the internal structure of the surface shape information of the object has not been judged, the positions are changed in accordance with the change in the positions of the joint sections and the skeleton sections for which the correspondence relationship has been judged, while maintaining the connection relationship between the joint sections and the skeleton sections in the reference body. For example, when the feature points on the internal structure corresponding to the right shoulder and the right fingertip are extracted by the feature point extracting section 4 and the feature point corresponding to the right elbow is not extracted, the position information of the joint sections corresponding to the right shoulder and the right fingertip is converted into the position information of the corresponding feature points, and the position information of the joint section corresponding to the right elbow located between the right shoulder and the right fingertip is converted in such a manner as to maintain the internal ratio with the right shoulder and the right fingertip.

[0056] Specifically, the skeleton adjusting section 7 compares the information of the feature points related to the internal structure converted into the three-dimensional position information by the feature point extracting section 4 and the skeleton information in the general-purpose body by the feature point attaching section 5, and derives the correspondence relationship therebetween, for example, derives the following correlation: the specific feature points A, B, C,... related to the internal structure correspond to the joint sections 1, 2, 3,... constituting the skeleton information in the general-purpose body, respectively. On this basis, the skeleton adjusting section 7 moves the positions of the skeleton structure in the general-purpose body so as to make the positional relationship of the skeleton structure in the general-purpose body coincide with the positional relationship of the feature points related to the internal structure. In the above example, the position (xl, yl, zl) of the joint section 1, the position (x2, y2, z2) of the joint section 2,... in the general-purpose body are moved to the position (X A , Y A , Z A ) of the corresponding feature point A related to the internal structure, the position (X B , Y B , Z B ) of the feature point B,..., respectively. Along with the movement of the joint sections, the positions and the shapes (lengths) of the skeleton sections formed between the joint sections also change, and the vertices which hold the information on the relative positional relationship between the joint sections and the skeleton sections and form the information related to the surface shape of the general-purpose body also change their positions in accordance with the movement of the joint sections.

[0057] Further, when there are feature points that cannot be extracted by the feature point extraction section 4, the skeleton adjustment section 7 does not delete the joint sections and the bone sections for which there are no corresponding feature points, and also performs position adjustment on the joint sections and the bone sections while maintaining the connection relationship between the joint sections and the bone sections, so as to match the distribution of the extracted feature points. With this function, position conversion processing of the joint sections and the bone sections can be performed without affecting the function of the general body as a computer graphic.

[0058] Through the processing by the skeleton adjustment section 7, the portions of the general body that are derived from the skeleton structure, such as the arm length, the leg length, the torso length, the shoulder width, and the like, become almost identical to the contents in the three-dimensional image data of the object.

[0059] The surface adjustment section 8 is used to change the positions of the vertices and / or the faces that constitute the surface shape of the general body after the position adjustment of the skeleton structure by the skeleton adjustment section 7, so as to make the surface shape of the general body identical to the outer surface shape based on the three-dimensional image data of the object.

[0060] Specifically, the surface adjustment section 8 first compares each vertex that constitutes the surface shape of the general body with each vertex included in the shape information generated by the shape information generation section 2 from the three-dimensional image data of the object, and derives the correspondence relationship that each vertex a, b, c,... included in the surface shape based on the three-dimensional image data corresponds to each vertex a, b, c,... included in the surface shape information of the general body. On this basis, the surface adjustment section 8 moves the positions of the vertices included in the surface shape information of the general body so as to make the positional relationship of the vertices included in the surface shape information of the general body identical to the positional relationship of the vertices of the outer surface shape of the object. In the above example, the positions (x α , y α , z α ) of the vertex a, the positions (x β , y β , z β ) of the vertex b, and the like of the general body are moved to the positions (X a , Y a , Z a ) of the corresponding vertex a, the positions (X b , Y b , Z b ) of the vertex b, and the like of the outer surface shape based on the three-dimensional image data, respectively. Along with the movement of the vertices, the positions and shapes of the edges connecting the vertices, the faces (polygons) constituted by three or more edges, and the collection of faces (meshes) also change.

[0061] The first embodiment adopts the following method to derive the correspondence between the vertices. First, the generic volume and the surface shape information based on the three-dimensional image data are arranged on the same space in a manner that the positional relationship of the skeletal structure is consistent. Second, for each of the faces (polygons) constituting the surface shape of the generic volume, a face (polygon) constituting the surface shape based on the three-dimensional image data located in the normal direction is searched for, and the closest face is determined as the corresponding face. Then, the vertex constituting the corresponding face that is closest to the vertex constituting the face of the generic volume is determined as the corresponding vertex, and this operation is performed for all the vertices.

[0062] Note that when the correspondence of a part of the vertices constituting the surface shape of the generic volume to the vertices constituting the surface shape of the object cannot be determined, the part of the vertices is moved along with the surrounding vertices while maintaining the positional relationship with each other. Through the processing of the surface adjustment section 8, the three-dimensional surface shape in the generic volume becomes almost consistent with the three-dimensional image data of the object.

[0063] Further, as the processing of the surface adjustment section 8, a method of adjusting the position of the face is also effective in addition to the movement of the vertices. For example, for each of the faces (polygons) constituting the surface shape of the generic volume, a face (polygon) constituting the surface shape based on the three-dimensional image data of the object located in the normal direction is searched for, and the closest face is determined as the corresponding face. Then, the face constituting the surface shape of the generic volume is moved to the corresponding face, and the direction of the face is rotated to be consistent with the direction of the corresponding face. Through the above processing, the three-dimensional surface shape in the generic volume can be made to become almost consistent with the three-dimensional image data of the object.

[0064] In the first embodiment, the relevance information indicating the relevance between the joint portions, the bone portions included in the skeletal structure, and the vertices constituting the surface shape is maintained by the processing of the surface adjustment section 8. The joint portions, the bone portions, and the vertices are only moved in position to be consistent with the three-dimensional image data, and do not lose their own identities, and thus the relevance information indicating the above relevance does not change.

[0065] Therefore, at the end of the processing of the surface adjustment section 8, a three-dimensional computer graphics is generated in which the generic volume is consistent with the object in the surface shape while maintaining the skeletal structure, the three-dimensional surface shape, and the relevance information between the skeletal structure and the surface shape in all aspects, and has matching positions of the feature points of the internal structure of the object on the skeletal structure and is almost consistent with the object in the surface shape, and completely has the relevance information between the surface shape and the skeletal structure.

[0066] The texture information generation section 9 generates information on the texture of the surface, i.e., texture information, from three-dimensional image data on the surface generated by photographing a person or the like as an object for generating a three-dimensional virtual person image. Texture also refers to a feature of the appearance of the surface made up of minute irregularities and the like, and is expressed as a pattern or the like. In general, in three-dimensional CG, a surface shape is determined by a model processing in which a surface is formed by connecting vertices to each other by edges, and a region surrounded by three or more edges is defined as a face (polygon), and a surface shape is determined by a collection (mesh) of faces. Since this method approximately expresses a real surface shape, information on texture made up of minute irregularities and the like of a surface is not included. Therefore, information on texture is created separately and added to a surface formed by the model processing to produce a realistic appearance. Note that the information on texture is specifically information on a two-dimensional image added to a surface, and a pattern (e.g., a height map, a normal map) reflecting a concave-convex pattern or the like is formed on a two-dimensional plane to express a concave-convex pattern or the like.

[0067] As a method of generating texture information, it is theoretically possible to generate texture directly on a three-dimensional surface shape. However, in the first embodiment, two-dimensional map information that displays a three-dimensional surface shape in two dimensions is used to generate texture on a two-dimensional map, or texture information prepared in advance on a two-dimensional map for a general body is used as it is.

[0068] The surface texture imparting section 10 imparts texture to the surface of the general body that has been subjected to skeleton adjustment and surface adjustment. By adding texture information generated by the texture information generation section 9 to the surface of the general body, texture made up of minute irregularities and the like can be reproduced on the general body made up of surfaces generated by a model processing. In the first embodiment, for texture information generated on a two-dimensional map of the general body, the texture information is added by performing a process of restoring a two-dimensional structure to a three-dimensional structure. Note that the surface shape of the general body is deformed to the surface shape of the object by vertex movement by the surface adjustment section 8, and the identity of the vertices is maintained before and after the process, so by making the positions of the vertices on the two-dimensional map correspond to the positions after movement by the surface adjustment section 8, texture information can be imparted automatically.

[0069] The surface image attaching section 11 is used to attach a surface image composed of a pattern, color, or the like of an outer surface included in the three-dimensional image data of a person or the like as a subject to the surface of the general body that has been subjected to skeleton adjustment and surface adjustment. The surface shape of the general body that has been subjected to surface adjustment is in agreement with the outer surface and shape surface in the three-dimensional image data, and the surface image attaching section 11 has a function of pasting the surface image included in the three-dimensional image data to the surface of the general body composed of the same shape after appropriately processing the surface image as needed by projection mapping by simulation. In the present first embodiment, the surface image is attached by projecting an RGB image acquired by a camera in a form of projection mapping by simulation from a setting position of the camera for acquiring the three-dimensional image data.

[0070] Next, the flow of the operation of the three-dimensional virtual character generation apparatus according to the present first embodiment will be described with reference to the flowchart of Fig. 6. Figure 2 First, the general body having a surface shape, a skeleton structure, and information indicating the correlation between the two is prepared by the body generation section 1 (step S101). In addition, the three-dimensional surface shape information based on the photographing data of the subject is generated by the shape information generation section 2 (step S102), a two-dimensional image is generated based on the generated three-dimensional surface shape information (step S103), feature point extraction is performed in the two-dimensional image (step S104), and the extracted feature points are mapped to appropriate positions on or inside the three-dimensional surface shape (step S105).

[0071] Then, the positions of the skeleton sections and joint sections of the general body are moved by the skeleton adjustment section 7 so as to match the positions of the corresponding feature points among the feature points located inside the three-dimensional surface shape of the subject (step S106). Also, the vertices and / or faces constituting the three-dimensional surface shape of the subject are compared with the vertices and / or faces constituting the three-dimensional surface shape of the general body by the surface adjustment section 8, and the positions of the vertices and / or faces of the general body are moved so as to agree with the positions of the corresponding vertices and / or faces of the subject (step S107).

[0072] Further, the texture information generated by the texture information generation section 9 is imparted to the surface shape of the general body (step S108), and the image generated based on the photographing data of the subject is attached to the three-dimensional surface shape of the general body (step S109), whereby the three-dimensional virtual character that realistically represents the subject is completed.

[0073] Finally, the advantages of the three-dimensional virtual character generation apparatus according to the first embodiment will be described. First, in the first embodiment, a general body having information (three-dimensional surface shape, skeleton structure, correlation between the two, etc.) required to constitute a virtual character prepared in advance is adjusted in position information of the skeleton structure and the three-dimensional surface shape to conform to the three-dimensional surface shape generated from the captured data of the object, and thereby a three-dimensional virtual character of the object is generated. In the related art, the skeleton structure and the correlation between the skeleton structure and the surface shape are manually constructed from the captured data, and by employing the constitution, the three-dimensional virtual character can be generated extremely easily and in a short time with high precision compared to the related art.

[0074] Further, in the first embodiment, in the feature point extraction process of the three-dimensional surface shape generated from the captured data, the feature points are extracted from a two-dimensional image converted from the three-dimensional surface shape, and then the feature points are projected onto the surface or inside of the three-dimensional surface shape. As the research of image recognition, posture estimation, and the like is continuously developed, the analysis process of the two-dimensional image can extract the feature points related to the three-dimensional surface shape with high precision by employing the method according to the first embodiment. Further, even if the feature points are missed when extracted by the feature point extraction section 4, the presence of the feature points can be appropriately estimated, for example, according to the thumb rule of the left-right symmetry of the human body and the like.

[0075] Further, in the first embodiment, when the skeleton structure of the general body is adjusted by the skeleton adjustment section 7, the position of the skeleton structure of the general body is adjusted to match the distribution of the feature points in the three-dimensional surface shape of the object in the corresponding relationship, with the connection relationship between the constituent elements of the skeleton structure, i.e., the joint portion and the bone portion, maintained. By employing this constitution, the following advantage is obtained: even if only a part of the feature points of the three-dimensional surface shape of the object can be extracted, the three-dimensional virtual character constituted by an appropriate skeleton structure can be generated in a configuration suitable for the three-dimensional surface shape of the object, using the extracted feature points as a clue.

[0076] (Second Embodiment)

[0077] Next, the three-dimensional virtual character generation apparatus according to the second embodiment will be described. In the second embodiment, the constituent elements having the same name and the same reference numerals as those in the first embodiment have the same function as those in the first embodiment, unless otherwise specified.

[0078] As Figure 3As shown, the three-dimensional virtual character generation apparatus according to the second embodiment includes, in addition to the configuration of the first embodiment, a motion information adding section 14 that adds motion information related to the motion of the skeleton structure to the general body, a shape transition information adding section 15 that adds shape transition information related to the change in the three-dimensional surface shape to the general body, an internal surface shape generation section 16 that adds the surface shape of the internal surface that is difficult to express on the external surface in the three-dimensional surface shape to the general body, and a non-adjustment region specifying section 17 that specifies the vertex that is not the position movement object of the surface adjustment section 8 among the vertices of the surface shape of the general body.

[0079] The motion information adding section 14 generates motion information related to the change in position when performing a motion, the linkage manner between the joint sections and the bone sections, and adds the motion information to the general body with respect to the skeleton structure of the general body generated by the body generation section 1. Specifically, the motion information specifies the specific linkage manner when performing a motion among a plurality of joint sections and bone sections in addition to the information that specifies the bending direction, the bending range of each joint section, and the rotation direction, the rotation range of each joint section and / or bone section. The information related to the linkage manner also includes information that specifies how the bone sections connected to the joint sections move and the joint sections move when performing a specified motion, like so-called inverse kinematics setting information, forward kinematics setting information, and the like.

[0080] The motion information is used when the general body including the skeleton structure performs a motion by animation or the like. When there is no motion information, in order to make the general body perform an animated motion, for example, the position designation of all the joint sections and the bone sections needs to be manually performed for each frame of the animation, and the work is extremely cumbersome. In the second embodiment, by adding the motion information in advance, when the general body performs a motion such as "reaching out the right hand to take an apple placed on a table", only the time change of the position of the joint section of the right fingertip (the time change related to the movement from the stable position to the apple) is designated, and the position changes of the joint sections of the right wrist, the right elbow, the right shoulder, and the like related to the joint section and the bone sections therebetween are almost automatically designated, and the general body can perform a natural animated motion.

[0081] The shape transition information adding section 15 generates and adds shape transition information to the general body, the shape transition information being information related to the change in the surface shape, specifically, information related to the position information of each vertex of a first shape and a second shape for forming a part of a region on the surface shape and the change in the position of each vertex in the process of transitioning from the first shape to the second shape.

[0082] The surface shape of the general primitive is defined by the apexes and the connection between the apexes. Therefore, the first shape and the second shape within the defined range are also defined by the position information of each apex within the range and the like. In the shape transition information, each apex is given the specific position information of the first shape and the specific position information of the second shape, and the movement path of each apex when transitioning from the first shape to the second shape is also defined. As the definition related to the movement path, for example, a function of a variable t is set so that when t = 0, the apex is located at the position corresponding to the first shape, when t = 1, the apex is located at the position corresponding to the second shape, and when 0 < t < 1, the apex is located at a defined position between the position of the first shape and the position of the second shape.

[0083] For example, in the region corresponding to the "eye" of the face, the first shape is set to the "closed eyelid state" and the second shape is set to the "open eye state". By the shape transition information addition section 15, the apex of the region corresponding to the "eye" in the surface shape of the general primitive is given the shape transition information composed of the above-mentioned shape and the position information corresponding to the transition between the shapes, and the variable t is appropriately adjusted when performing the animation action, whereby the "eye blinking" can be expressed. Further, in the regions corresponding to the "outer corner of the eye" and the "corner of the mouth" of the face, the first shape is set to the "state of lifting the outer corner of the eye" and the "state of lowering the corner of the mouth", and the second shape is set to the "state of lowering the outer corner of the eye" and the "state of lifting the corner of the mouth", whereby the "smiling" action can be expressed by the transition from the first shape to the second shape.

[0084] Further, not only the face, but also the body shape can be changed by changing the shape of the hand, the leg, and the waist, and the body shape change can be expressed when performing the animation action. In this way, according to the expression mode specification, for each apex belonging to the specified region, the position is specified using a specified variable with respect to the first shape, the second shape, and the intermediate shape therebetween, whereby a more smooth and natural animation action can be realized by controlling the value of the variable.

[0085] Note that, for the convenience of explanation, the above describes an example in which the shape is changed using a single variable t, but the shape transition information is not limited to being composed of only a single variable. The shape transition information can be composed of two or more variables, and independent variables can be set according to the properties of the first shape and the second shape composed of a plurality of modes.

[0086] The inner surface shape generating section 16 is used to generate the shape of the inner surface that is difficult to express on the outer surface on the general body. The inner surface is a surface structure that is continuously connected with the outer surface, and generally refers to a portion that cannot be confirmed from the outside. For example, it refers to a portion that corresponds to the inside of the oral cavity (the portion that is exposed when the mouth is opened), the inside of the nasal cavity, and the inside of the eye socket (a structural portion located under the eyelid). By constructing the inner surface shape on the general body by the inner surface shape generating section 16, for example, by appropriately selecting the first shape and the second shape by the shape conversion information adding section 15, it is possible to display the inner surface shape of the inside of the oral cavity in the state where the mouth is opened when the mouth opening and closing action is performed, and it is possible to achieve a natural video performance.

[0087] The non-adjustment region designating section 17 is used to designate a region that does not belong to the adjustment target of the surface adjustment section 8 in the surface shape of the general body. As also mentioned in the first embodiment, the surface adjustment section 8 performs the following processing: on the basis of making each vertex that constitutes the surface shape of the general body correspond to each vertex that constitutes the three-dimensional surface shape of the object, moving to the position of the corresponding vertex. However, since the three-dimensional surface shape of the object is constituted by the photographing information and the like and only information related to the outer surface is given, for example, for a vertex on the inner surface such as the inside of the oral cavity in the closed mouth state, there is no corresponding vertex on the three-dimensional surface shape of the object, and thus it should not be set as the adjustment target of the surface adjustment section 8. The non-adjustment region designating section 17 has the function of excluding, from the adjustment target of the surface adjustment section 8, a vertex of a region that should not be set as the above adjustment target by designating the vertex. Note that when the surface adjustment section 8 does not perform adjustment, the above vertex, like other vertices, is changed in shape when adjustment is performed by the skeleton adjustment section 7 in accordance with the position change of the corresponding skeleton section and joint section, and after adjustment is performed by the surface adjustment section 8, each vertex of the inner surface is moved or subjected to interpolation processing to match the position change of each vertex that is adjacent to the designated range.

[0088] Next, the action of the three-dimensional virtual character generating apparatus related to the second embodiment will be described with reference to the flowchart of FIG. 8. Figure 4 First, like the first embodiment, the general body is prepared (step S201), the action information is given to the skeleton structure of the general body (step S202), the shape conversion information is added to the three-dimensional surface shape of the general body (step S203), and the inner surface shape is added to the surface shape (step S204).

[0089] Then, like the first embodiment, three-dimensional surface shape information based on the photographed data of the object is generated (step S205), a two-dimensional image is generated based on the generated three-dimensional surface shape information (step S206), feature points are extracted in the two-dimensional image (step S207), and the extracted feature points are mapped to appropriate positions on the surface or inside of the three-dimensional surface shape (step S208). Then, the positions of the skeleton and joint portions of the general base are moved by the skeleton adjustment section 7 so as to match the positions of the corresponding feature points among the feature points located inside the three-dimensional surface shape of the object (step S209).

[0090] Further, the region constituting the internal surface shape or the like is excluded from the object of the surface adjustment by the non-adjustment region specification section 17 (step S210), and on this basis, the position of the vertex of the general base is moved by the surface adjustment section 8 so as to coincide with the position of the corresponding vertex of the object (step S211), like the first embodiment. Then, like the first embodiment, texture information is given to the surface shape (step S212), the photographed data of the object is attached to the three-dimensional surface shape of the general base (step S213), and thus the three-dimensional virtual character of the object is completed.

[0091] Finally, the advantages of the three-dimensional virtual character generation apparatus according to the second embodiment will be described. First, in the second embodiment, the configuration in which the action information and the shape transition information are attached to the general base is adopted, and by this configuration, the advantage that natural animation actions of the generated three-dimensional virtual character can be easily realized is obtained. By attaching the action information, for example, the natural position change manner of the wrist, the wrist, the elbow, the shoulder, or the like can be determined by specifying only the position change of the fingertip, and by attaching the shape transition information, natural expressions in the eye blinking, the smiling, or the like, the body shape change, or the like can be easily realized.

[0092] Further, in the second embodiment, in the surface adjustment processing by the surface adjustment section 8, the internal surface or the like that cannot be expressed on the three-dimensional surface shape of the object is excluded from the adjustment object. As described in the first embodiment, the surface adjustment section 8 automatically derives the correspondence relationship between the vertices and the vertices or between the faces and the faces to perform the surface adjustment processing, and for example, when the vertices and / or the faces constituting the internal surface are also adjusted as objects, a correspondence relationship can be constructed with a part of the external surface of the three-dimensional surface shape of the object or the like that should not originally form a correspondence relationship. In the present second embodiment, in order to avoid the above situation, the following configuration is adopted: the internal surface shape on which a correspondence relationship cannot be obviously constructed and the external surface shape corresponding to the region that cannot be photographed in the photographed data are excluded from the object of the surface adjustment processing in advance. In this way, the advantage that the automatic processing is performed while avoiding the generation of an inappropriate processing result is obtained.

[0093] (Third Embodiment)

[0094] The three-dimensional virtual character generation device according to the third embodiment will now be described. It should be noted that, in the third embodiment, unless otherwise specified, the constituent elements with the same names and symbols as those in the first and second embodiments perform the same functions as those in the first and second embodiments.

[0095] The three-dimensional virtual character image generation device according to this third embodiment prepares a general model for which there are no specified information related to hairstyle and clothing, and generates a basic virtual character image with no limitation on the hairstyle and clothing of the object based on the general model and the shooting data of the object.

[0096] like Figure 5 As shown, the three-dimensional virtual character image generation device according to this third embodiment, based on the configuration of the first embodiment and the second embodiment, further includes a surface area discrimination unit 20 for judging the exposed skin area, hair area and clothing area in the three-dimensional surface shape of the object; a body shape estimation unit 21 for estimating the basic body shape after removing clothing and the like from the clothing area of ​​the object; a shape transformation unit 22 for adjusting the surface shape of the general body to conform to the estimated basic body shape; an unadjusted area specification unit 24 for specifying the clothing area, hair area and inner surface shape to exclude them from the surface adjustment object; a skin color calculation unit 25 for calculating the skin color of the object; a texture information generation unit 26 for generating texture information related to the virtual character image without hair and clothing; and a surface texture assignment unit 27 for assigning texture to all surfaces of the virtual character image based on the generated texture information.

[0097] The surface region discrimination unit 20 analyzes and processes the outer surface shape of three-dimensional image data based on a person or similar object to determine areas where skin is exposed (skin exposure area), areas covered by clothing or other decorations (clothing area), and areas covered by hair (hair area). Specifically, the surface region discrimination unit 20, like the two-dimensional image generation unit 3, converts the three-dimensional image data into a two-dimensional image and uses techniques such as semantic segmentation on the converted two-dimensional image to determine the skin exposure area, clothing area, and hair area in the two-dimensional image. Furthermore, by performing the same processing as that performed by the feature point attachment unit 5, the boundary lines of the areas derived from the two-dimensional image are reproduced on the three-dimensional surface shape, thereby clarifying the skin exposure area, clothing area, and hair area. It should be noted that in the processing of the surface region discrimination unit 20, the two-dimensional image is preferably multiple images obtained by projecting in multiple directions.

[0098] The body shape estimation unit 21 is used to estimate the basic body shape of the person or the like as the subject after the clothes or the like are removed from the clothing region. The three-dimensional image data of the person or the like as the subject is generally captured in a state in which the clothes or the like are worn, and the basic body shape cannot generally be derived from the captured data itself. Therefore, in the third embodiment, the following configuration is adopted: the basic body shape is set through the estimation processing of the body shape estimation unit 21.

[0099] Specifically, the body shape estimation unit 21 excludes the estimated volume corresponding to the clothes or the like in the clothing region after estimating the structure of the clothes or the like in the clothing region, thereby estimating the basic body shape of the person or the like as the subject in the clothing region. Specifically, the body shape estimation unit 21 performs a process of reducing the volume of the clothing region according to the clothing thickness for most of the region covered by the clothes. In addition, the body shape estimation unit 21 performs a process of reducing the volume for the portion near the joint portion in the clothing region (for example, the portion corresponding to the neck, underarm, and crotch pad), and the amount of reduction is the thickness amount obtained by multiplying the clothing thickness by a coefficient larger than 1. This is because the clothing is generally formed in a size with a margin so as not to feel tight when the clothing is bent for the joint portion, and thus it is preferable to be larger than the reduction amount of other portions. As for the specific coefficient value, it is preferable to be previously databased according to the clothing type or the like.

[0100] Note that, as the function of the body shape estimation unit 21, for example, the clothing thickness can be estimated according to the shape change near the boundary between the clothing region and the skin exposure region. Since the shape change near the boundary with the skin exposure region is due to the presence or absence of the clothes or the like, it is preferable to estimate the clothing thickness on the assumption that the volume increase of the clothing region side due to the shape change is caused by the thickness of the clothes or the like.

[0101] In addition, as the function of the body shape estimation unit 21, the relationship between the color, texture, shape, pattern, or the like of the clothes or the like and the thickness of the clothes portion can be additionally stored in a database or the like, and the clothing thickness of the clothes or the like can be estimated by referring to the color tone, texture, specific shape, and pattern of the clothing region of the person or the like as the subject in the database. Specifically, it can be, for example, that when a winter coat is estimated according to the color tone, texture, or the like, the thickness of the coat and the average underwear thickness are estimated as the clothing thickness, and when a T-shirt is estimated, the thickness of the T-shirt is estimated to estimate the basic body shape.

[0102] Further, for a site such as the neck circumference where a skin exposed area and a clothing area (not covered by a decoration such as clothing but partially exposed skin) coexist, the body shape estimation section 21 uses information of the surface shape of the skin exposed area. Specifically, since the basic body shape of the neck circumference is approximately cylindrical with the joint portion (or the bone portion corresponding to the spine) in the inside as the center, it is possible to estimate that the basic body shape of the clothing area is constituted by the surface of an approximate cylinder with the distance from the corresponding joint portion (or bone portion) to the surface of the skin exposed area as the radius. According to this estimation, the body shape estimation section 21 derives the basic body shape of the neck circumference or the like.

[0103] The shape transformation section 22 is used to change the surface shape of the general primitive by using shape transformation information set in advance for the general primitive. As also mentioned in the second embodiment, it is possible to set the shape transformation information in order to change the shape of the hand, leg, waist, or the like, and for each vertex constituting the surface shape corresponding to the above-mentioned portion, a positional change when 0 < t < 1 is respectively prescribed, and the variable t is appropriately adjusted, whereby the surface shape of the general primitive is changed to conform to the basic body shape. In the present third embodiment, for the portion where the basic body shape is estimated in the clothing area, instead of the surface adjustment processing by the surface adjustment section 8, the shape change is performed by the adjustment of the shape transformation information by the shape transformation section 22.

[0104] Note that, in theory, it is possible to perform the shape transformation processing only for the portion corresponding to the clothing area, and for the skin exposed portion, the adjustment is performed by the surface adjustment section 8 as in the first embodiment and the second embodiment. However, in the present embodiment, since the basic body shape of the clothing area and the surface shape of the skin exposed area are continuously formed, there is no reason to adopt different methods for each area in the adjustment of the surface shape of the general primitive. Therefore, in the present third embodiment, the following configuration is adopted: for the surface shape other than the surface shape of the head portion which needs to be finely adjusted, the surface shape of the general primitive is adjusted by the shape transformation section 22.

[0105] The non-adjustment area designating section 24, like the non-adjustment area designating section 17, excludes the inside surface from the object of the surface adjustment, and further excludes the areas corresponding to the hair area and the clothing area from the object of the surface adjustment. Through the area designation by the non-adjustment area designating section 24, the surface adjustment section 8 performs the processing of making each vertex of the general primitive identical to each vertex in the outer surface shape based on the three-dimensional image data of the person or the like which is the object, only for the portion in the skin exposed area corresponding to the head portion in the outer surface.

[0106] The skin color calculating section 25 is used to calculate the skin color of the person who is the object. The color of the skin exposed area in the photographed data of the person who is the object and the like is not always the same, and varies depending on the lighting at the time of photographing and the like. Therefore, the skin color calculating section 25 acquires the color information of the skin exposed area, and supplies the texture information generating section 26 with information relating to the skin color and the like of the portion of the clothing area and the hair area after the clothing and the hair are removed.

[0107] The texture information generating section 26 is used to generate information relating to the surface texture when the surface shape of the clothing area and the hair area is replaced with the basic body shape, and information relating to the skin color of the above-mentioned areas, on the basis of the functions of the texture information generating section 9. The surface texture when replaced with the basic body shape can adopt an existing skin texture, or can regenerate the skin texture from the information of the skin exposed area of the person who is the object and the like. Furthermore, the information relating to the skin color adopts the information generated by the skin color calculating section 25.

[0108] The surface texture imparting section 27 is used to impart the surface texture suitable for the basic body shape to the above-mentioned areas when the surface shape of the hair style area and the clothing area is replaced with the basic body shape, on the basis of the functions of the surface texture imparting section 10. The information relating to the imparted surface texture adopts the information generated by the texture information generating section 26.

[0109] Next, the action of the three-dimensional virtual person image generating apparatus relating to the third embodiment will be described with reference to the flowchart of FIG. 8. Figure 6 First, a general base body having no features in the hair area and the clothing area is prepared (step S301), and like the second embodiment, action information is added to the skeletal structure of the general base body (step S302), shape conversion information is added to the three-dimensional surface shape of the general base body (step S303), and an inner surface shape is added to the surface shape (step S304). Then, like the first embodiment, three-dimensional surface shape information based on the photographed data of the object is generated (step S305), a two-dimensional image is generated from the generated three-dimensional surface shape information (step S306), feature point extraction is performed in the two-dimensional image (step S307), and the extracted feature points are mapped to appropriate positions on the surface or inside of the three-dimensional surface shape (step S308). Then, the position of the skeletal portion and the joint portion of the general base body is moved by the skeletal adjusting section 7 so as to match the position of the corresponding feature point among the feature points located inside the three-dimensional surface shape of the object (step S309).

[0110] Next, the skin exposed area, the hair area, and the clothing area are discriminated in the surface shape of the object by the surface area discriminating section 20 (step S310), and the basic body shape is estimated by the body shape estimating section 21 for the clothing area (step S311).

[0111] Then, the shape transformation section 22 appropriately adjusts the variables of the pre-set shape transformation information for the general body so as to conform to the surface shape of the object after the clothing region is replaced by the basic body shape, thereby changing the three-dimensional surface shape of the general body (step S312). On the other hand, for the head, the surface adjustment section 8 performs adjustment processing on the other part of the general body except for the inner surface shape and the hair region, so that the shape of the general body conforms to the surface shape of the object, and the structure of the general body is maintained for the inner surface shape and the hair region (step S313).

[0112] Further, the skin color calculation section 25 calculates the skin color of the person or the like as the object (step S314), and generates texture information including the texture of the part replaced by the basic body shape using the calculated skin color (step S315). Then, the surface texture imparting section 27 imparts the texture information of the part replaced by the basic body shape and the texture information related to the region other than the clothing region to the surface shape of the general body after the surface shape is adjusted (step S316), and the surface image attaching section 11 performs image attachment to the necessary part such as the face (step S317), thereby generating the basic virtual character image related to the person or the like as the object.

[0113] Finally, the advantages of the three-dimensional virtual character image generation apparatus according to the third embodiment will be described. In the third embodiment, the basic virtual character image having the features of the face, the body, and the like of the object and basically using the hairstyle and the clothing of the general body is generated.

[0114] As an application example of the three-dimensional virtual character image which faithfully reproduces the object, a case where the hairstyle is different from the actual object or the clothing is different from that at the time of shooting can be considered. In the third embodiment, because the hairstyle and the clothing part are adopted as a separate object which is separated from the basic virtual character image, the advantage that the hairstyle and the clothing different from those in the shooting data can be additionally set to the basic virtual character image is obtained.

[0115] (Modified Example)

[0116] The three-dimensional virtual character image generation apparatus according to the modified example is associated with the function of the three-dimensional virtual character image generation apparatus according to the third embodiment, extracts the hair and the clothing and the like worn on the body of the person or the like as the object, and generates the body data composed of a separate three-dimensional computer graphics.

[0117] As Figure 7As shown, the three-dimensional virtual character generation apparatus according to the present modification includes a surface extraction section 29 that extracts surface portions of the hair region and the clothing region from the three-dimensional surface shape of the subject, a three-dimensional structuring section 30 that three-dimensionally structures the extracted surface portions, a boundary interpolation section 31 that interpolates the boundaries (edges) of the surface portions and the structures added by the three-dimensional structuring section 30, and a shape information generation section 32 that reconstructs the surface shape information of the entire three-dimensional structure, on the basis of the constituent elements of the three-dimensional virtual character generation apparatus according to the third embodiment.

[0118] The surface extraction section 29 extracts the surface portions of the hair region and the clothing region judged by the surface region discrimination section 20. The specific configuration can be arbitrary as long as the extraction processing can be performed, and in the present modification, the surface portions included in the hair region and the clothing region are extracted by so-called extraction processing.

[0119] The three-dimensional structuring section 30 three-dimensionally structures the surface portions extracted by the surface extraction section 29. Since the surface portions extracted by the surface extraction section 29 are constituted only by a part of the surface of the three-dimensional computer graphics, they do not have a three-dimensional structure as such. The data processing of this structure is complicated in three-dimensional image processing, and therefore, in the present modification, it is converted into a three-dimensional structure by the three-dimensional structuring section 30. As an example of the three-dimensional structure, a cylindrical three-dimensional structure having a hollow portion can be formed by giving a thickness to the extracted surface portion in the internal direction (negative offset direction), and a columnar three-dimensional structure having no hollow portion can be formed by closing the end portion corresponding to the boundary portion with the other region with a newly set surface structure.

[0120] The boundary interpolation section 31 interpolates the boundaries of the surface portions extracted by the surface extraction section 29 and the structure portions added after the three-dimensional structuring. It is preferable to smooth the boundary shape, for example, by performing curve interpolation processing.

[0121] The shape information generation section 32 generates the surface shape information again with respect to the surface shape of the clothing region after the three-dimensional structuring. The arrangement of the vertices of the surface shape and the like are optimized by the shape information generation section 32.

[0122] After the surface shape is optimized by the shape information generation section 32, the outer surface of the clothing region is two-dimensionally structured by performing UV unwrapping, and the texture information is generated and imparted to the two-dimensionally structured outer surface, and the outer surface is again restored to a three-dimensional structure. Then, the color, pattern, and the like are added to the outer surface in a form in which the RGB image related to the hair portion and the clothing portion is projected by the simulated projection mapping.

[0123] As described above, in the present modification, the part belonging to the hair region and the part belonging to the clothing region are extracted from the three-dimensional surface shape of the object and three-dimensionalized, thereby generating the body data independent of the basic virtual character image.

[0124] (Fourth Embodiment)

[0125] Next, the three-dimensional virtual character image generation apparatus according to the fourth embodiment will be described. Note that, in the fourth embodiment, the components having the same names and the same symbols as those in the first to third embodiments function in the same manner as the components in the first to third embodiments, unless otherwise specified.

[0126] The three-dimensional virtual character image generation apparatus according to the fourth embodiment has a function of grafting the hair part and the clothing part extracted from the three-dimensional surface shape of the object to the basic virtual character image of another character or the like. Specifically, the apparatus has a configuration of performing a prescribed process using the basic virtual character image generated using the three-dimensional surface shape related to a specific character or the like in the third embodiment and the independent body data related to the clothing and the hair of the character or the like generated in the modification, thereby grafting the clothing body and the hair body to the basic virtual character image related to another character.

[0127] As shown in Figure 8 , the three-dimensional virtual character image generation apparatus according to the fourth embodiment includes, on the basis of the configurations of the first to third embodiments, a body connection unit 35 that associates the clothing body and the hair body related to a specific character or the like with the basic virtual character image of the specific character, a general shape unit 36 that changes the basic virtual character image, the clothing body, and the hair body associated by the body data connection unit 35 integrally into the shape of a general body, a relevance information transmission unit 37 that transmits relevance information related to the vertex of the corresponding general body to each vertex constituting the clothing body and the hair body changed into the shape of the general body, a transformation information transmission unit 38 that transmits transformation information related to the vertex of the corresponding general body, and a connection release unit 39 that releases the association of the clothing body and the hair body with the basic virtual character image.

[0128] The body connection unit 35 is used to associate the extracted clothing body and hair body related to a specific character with the basic virtual character image of the same character. Specifically, the body connection unit 35 performs a process of connecting the clothing body and the hair body to the positions corresponding to the clothing region and the hair region of the basic virtual character image, respectively, while maintaining the independence of the clothing body and the hair body. Through this process, the clothing body and the hair body change their own shapes in accordance with the change in the shape of the basic virtual character image while maintaining the independence with respect to the basic virtual character image.

[0129] The general-shaping section 36 functions to change the shape of the basic virtual character, the clothing primitive, and the hair primitive, which are integrated with each other, to the same shape as the general primitive. As described in the third embodiment, the shape conversion section 22 adjusts the variable t of the conversion information for the general primitive to make the basic virtual character conform to the three-dimensional surface shape of the object (the surface shape after the clothing region is replaced by the basic body shape). The general-shaping section 36 has a function of performing a shape change process by performing a process opposite to that of the shape conversion section 22, i.e., by restoring the variable t from the adjusted value to the value of the general primitive, to make the basic virtual character, the clothing primitive, and the hair primitive conform to the shape of the general primitive.

[0130] The correlation information transmission section 37 functions to transmit, to each vertex constituting the clothing primitive and the hair primitive, the correlation information related to the vertex on the corresponding general primitive. As described above, the clothing primitive and the hair primitive are changed in shape by the general-shaping section 36 to conform to the shape of the general primitive, and the vertices on the surface of the general primitive and the vertices of the clothing primitive and the like are in a corresponding relationship with each other. By transmitting the correlation information given to the vertices on the surface of the general primitive to the corresponding vertices, the clothing primitive and the hair primitive can move in accordance with the movement of the skeleton structure as with the surface of the general primitive.

[0131] The conversion information transmission section 38 functions to transmit, to each vertex constituting the clothing primitive and the hair primitive, the conversion information related to the vertex on the corresponding general primitive. As described in the third embodiment, each vertex constituting the surface shape of the general primitive is given conversion information, and the shape conversion section 22 appropriately adjusts the variable t of the conversion information to conform to the basic body shape, thereby generating the basic virtual character. Each vertex constituting the clothing primitive and the hair primitive is also given the same conversion information as each vertex on the corresponding general primitive, and the shape can be changed by the shape conversion section 22 as with the general primitive.

[0132] The connection release section 39 functions to release the connection relationship between the clothing primitive and the hair primitive and the basic virtual character after a series of processes ends. By separating the clothing primitive and the hair primitive, which have a three-dimensional surface shape conforming to the general primitive and are given the same correlation information and conversion information as the general primitive, from the basic virtual character, the clothing primitive and the hair primitive can be used as accessories for other basic virtual characters. That is, when generating a basic virtual character of another object, the three-dimensional surface shape is changed by performing the same adjustment of the variable of the conversion information as that of the basic virtual character, and becomes a state conforming to the shape of the basic virtual character. By attaching the clothing primitive and the hair primitive in this state to the basic virtual character, a three-dimensional virtual character having a different hairstyle and different clothing from those at the time of photographing the object can be realized.

[0133] Next, the operation of the three-dimensional virtual character generation apparatus according to the fourth embodiment will be described with reference to the drawings. Figure 9 First, the association process of integrating the basic virtual character generated in the third embodiment and the modification example, the hair element, and the clothing element is performed by the element data connection section 35 (step S401). Then, the shape change process is performed by the general shape section 36 by adjusting the variable of the transformation information of the three-dimensional surface shape composed of the integrated basic virtual character, the clothing element, and the hair element to coincide with the surface shape of the general element (step S402).

[0134] Then, the correlation information of the general element is transmitted to the hair element and the clothing element (step S403), and after the transformation information of the general element is transmitted (step S404), the connection relationship of the hair element and the clothing element to the basic virtual character is released (step S405). Through the above steps, the hair element and the clothing element can be attached to any basic virtual character as independent accessories, and can be given a hairstyle and a costume different from the photographed data of the object to any basic virtual character.

[0135] - Industrial Applicability -

[0136] The present application can be utilized as a technology for generating a high-quality three-dimensional virtual character based on the photographed data of an object.

[0137] -Explanation of Reference Numerals-

[0138] 1 Element generation section

[0139] 2 Shape information generation section

[0140] 3 Two-dimensional image generation section

[0141] 4 Feature point extraction section

[0142] 5 Feature point addition section

[0143] 6 Skeleton correspondence relationship determination section

[0144] 7 Skeleton adjustment section

[0145] 8 Surface adjustment section

[0146] 9, 26 Texture information generation section

[0147] 10, 27 Surface texture imparting section

[0148] 11 Surface image addition section

[0149] 14 Motion information addition section

[0150] 15 Shape transformation information addition section

[0151] 16 inner surface shape generating section

[0152] 17, 24 non-adjustment region designating section

[0153] 20 surface region discriminating section

[0154] 21 body shape estimating section

[0155] 22 shape converting section

[0156] 25 skin color calculating section

[0157] 29 surface extracting section

[0158] 30 three-dimensional processing section

[0159] 31 interface interpolation processing section

[0160] 32 shape information generating section

[0161] 35 body connecting section

[0162] 36 general shape converting section

[0163] 37 relevance information transmitting section

[0164] 38 conversion information transmitting section

[0165] 39 connection releasing section

Claims

1. A three-dimensional virtual character generation device, which generates a three-dimensional virtual character image based on the photographed data of an object, using vertex groups and / or face groups to represent the three-dimensional surface shape, characterized in that: The three-dimensional virtual character image generation device includes a surface shape generation unit (2), a skeleton structure estimation unit (3, 4, 5), a skeleton correspondence discrimination unit (6), a skeleton adjustment unit (7), a surface adjustment unit (8), and an image attachment unit (11). The surface shape generation unit (2) generates a three-dimensional surface shape based on the photographed data of the object. The skeleton structure estimation units (3, 4, 5) convert the three-dimensional surface shape of the object into a two-dimensional image, and estimate at least a portion of the skeleton structure based on the two-dimensional image. The skeleton correspondence discrimination unit (6) determines the correspondence between the skeleton structure estimated by the skeleton structure estimation unit and the skeleton structure of the general virtual character image. The general virtual character image specifies the three-dimensional surface shape, skeleton structure, and the correlation between the two. While maintaining the connection relationship between the constituent elements of the skeleton structure, the skeleton adjustment unit (7) adjusts the position of the skeleton structure of the general virtual character image to match the correspondence determined by the skeleton correspondence discrimination unit. The surface adjustment unit (8) moves the vertex group and / or face group of the general-purpose virtual character image to a position consistent with the vertex group and / or face group of the three-dimensional surface shape of the object represented by the object, based on the correspondence between the vertex group and / or face group of the three-dimensional surface shape of the general-purpose virtual character image after the position adjustment of the skeleton structure. The image attaching unit (11) attaches an image processed from the object's photographic data to the surface of the general-purpose virtual character image after the position of the vertex group and / or face group has been moved by the surface adjustment unit.

2. The three-dimensional virtual character generation device according to claim 1, characterized in that: The general-purpose virtual character image specifies motion information, shape transformation information, and information related to vertex groups representing internal shapes. The motion information relates to the overall positional changes of the skeleton structure and the linkage between its constituent elements during motion. The shape transformation information specifies the transformation method of the vertex groups when transforming from a first shape to a second shape in at least a portion of a three-dimensional surface shape. The objects whose positions are moved by the surface adjustment unit do not include the vertex group that represents the internal shape.

3. The three-dimensional virtual character generation device according to claim 2, characterized in that: The three-dimensional virtual character generation device also includes a surface area judgment unit (20), a basic body shape generation unit (21), and a shape transformation unit (22). The surface region determination unit (20) determines the exposed skin region, clothing region, and hair region in the three-dimensional surface shape based on the object's shooting data. The basic body shape generation unit (21) removes the clothing portion from the clothing area identified by the surface area determination unit and generates the surface shape of the basic body shape. The shape transformation unit (22) causes the surface shape of the universal virtual character image to change according to the shape transformation information, so as to match the basic body shape. The surface adjustment unit does not perform surface adjustment processing on the clothing area and the hair area, but generates a three-dimensional virtual character image. This three-dimensional virtual character image does not have information related to clothing and hair from the object's shooting data.

4. The three-dimensional virtual character generation device according to claim 3, characterized in that: The three-dimensional virtual character generation device includes a clothing area extraction unit (29), a clothing body generation unit (30), a body connection unit (35), a general shape shaping unit (36), an information transmission unit (37, 38), and a connection release unit (39). The clothing region extraction unit (29) extracts the surface structure, which is located on the clothing region on the three-dimensional surface shape based on the object's shooting data. The clothing body generation unit (30) converts the extracted portion into a three-dimensional clothing body. The body connection unit (35) connects the clothing body to the three-dimensional virtual character image based on the shooting data. The generalized shape unit (36) causes the product formed by connecting the three-dimensional virtual character image and the clothing body to undergo shape change according to the shape transformation information, so as to conform to the three-dimensional surface shape of the generalized virtual character image. The information transmission units (37, 38) transmit information related to the three-dimensional surface shape and skeletal structure of the general virtual character image, as well as the shape transformation information, to the clothing body after the shape change. After the information transmission unit transmits information, the connection release unit (39) releases the connection between the clothing body and the three-dimensional virtual character image.

5. A method for generating a three-dimensional virtual character, which generates a three-dimensional virtual character based on the photographic data of an object, using vertex groups and / or face groups to represent the shape of a three-dimensional surface, characterized in that: The method for generating a three-dimensional virtual character includes a surface shape generation process, a skeleton structure estimation process, a skeleton correspondence determination process, a skeleton adjustment process, a surface adjustment process, and an image attachment process. The surface shape generation process generates a three-dimensional surface shape based on the photographic data of the object. The skeleton structure estimation process converts the three-dimensional surface shape of the object into a two-dimensional image, and estimates at least a portion of the skeleton structure based on the two-dimensional image. The skeleton correspondence determination process determines the correspondence between the skeleton structure estimated by the skeleton structure estimation process and the skeleton structure of the general-purpose virtual character image. The general-purpose virtual character image specifies the three-dimensional surface shape, skeleton structure, and the correlation between the two. The skeleton adjustment process maintains the connection relationships between the constituent elements of the skeleton structure while adjusting the position of the skeleton structure of the general-purpose virtual character image to match the correspondence determined by the skeleton correspondence discrimination process. The surface adjustment process, based on the correspondence between the vertex groups and / or face groups of the three-dimensional surface shape of the represented object and the vertex groups and / or face groups of the three-dimensional surface shape of the general-purpose virtual character image after position adjustment of the represented skeleton structure, moves the vertex groups and / or face groups of the general-purpose virtual character image to a position consistent with the vertex groups and / or face groups of the three-dimensional surface shape of the represented object. The image addition process adds an image processed using the object's photographic data to the surface of the general-purpose virtual character image after the vertex group and / or face group have been moved through the surface adjustment process.

6. The method for generating a three-dimensional virtual character image according to claim 5, characterized in that: The general-purpose virtual character image specifies motion information, shape transformation information, and information related to vertex groups representing internal shapes. The motion information relates to the overall positional changes of the skeleton structure and the linkage between its constituent elements during motion. The shape transformation information specifies the transformation method of the vertex groups when transforming from a first shape to a second shape in at least a portion of a three-dimensional surface shape. The objects whose positions are moved in the surface conditioning process do not include the vertex group that represents the internal shape.

7. The method for generating a three-dimensional virtual character image according to claim 6, characterized in that: The method for generating 3D virtual character images also includes a surface area judgment process, a basic body shape generation process, and a shape transformation process. The surface area determination process identifies the exposed skin area, clothing area, and hair area in the three-dimensional surface shape based on the object's photographed data. The basic body shape generation process removes the clothing portion from the clothing area identified by the surface area judgment process and generates the surface shape of the basic body shape. The shape transformation process, based on the shape transformation information, causes the surface shape of the generic virtual character image to change to match the basic body shape. In the surface adjustment process, at least the clothing area and the hair area are not subjected to surface adjustment processing, and a three-dimensional virtual character image is generated. This three-dimensional virtual character image does not have information related to clothing and hair from the object's shooting data.

8. The method for generating a three-dimensional virtual character image according to claim 7, characterized in that: The method for generating a 3D virtual character also includes a clothing area extraction process, a clothing body generation process, a body connection process, a general shape shaping process, an information transmission process, and a connection unlinking process. The clothing area extraction process extracts the surface structure, which is located on the three-dimensional surface shape of the clothing area based on the object's photographic data. The garment body generation process converts the extracted portion into a three-dimensional garment body. The body connection process connects the clothing body to the three-dimensional virtual character image based on the shooting data. The generalized shaping process, based on the shape transformation information, causes the product formed by connecting the three-dimensional virtual character image and the clothing substrate to undergo shape changes to conform to the three-dimensional surface shape of the generalized virtual character image. The information transmission process transmits information related to the three-dimensional surface shape and skeletal structure of the universal virtual character image, as well as the shape transformation information, to the clothing body after the shape change. The connection termination process terminates the connection between the clothing body and the three-dimensional virtual character image after the information transmission process has transmitted the information.

9. A three-dimensional virtual character generation program product, which generates a three-dimensional virtual character image based on the shooting data of an object, by representing the three-dimensional surface shape through vertex groups and / or face groups, characterized in that: The 3D virtual character generation program product enables the computer to perform the following steps: surface shape generation, skeleton structure estimation, skeleton correspondence determination, skeleton adjustment, surface adjustment, and image attachment. The surface shape generation step generates a three-dimensional surface shape based on the photographed data of the object. The skeleton structure estimation step converts the three-dimensional surface shape of the object into a two-dimensional image, and estimates at least a portion of the skeleton structure based on the two-dimensional image. The skeleton correspondence determination step determines the correspondence between the skeleton structure estimated by the skeleton structure estimation step and the skeleton structure of the general-purpose virtual character image. The general-purpose virtual character image specifies the three-dimensional surface shape, skeleton structure, and the correlation between the two. The skeleton adjustment step maintains the connection relationships between the constituent elements of the skeleton structure while adjusting the position of the skeleton structure of the general-purpose virtual character image to match the correspondence determined by the skeleton correspondence discrimination step. The surface adjustment step, based on the correspondence between the vertex group and / or face group of the three-dimensional surface shape of the represented object and the vertex group and / or face group of the three-dimensional surface shape of the general-purpose virtual character image after position adjustment of the represented skeleton structure, moves the vertex group and / or face group of the general-purpose virtual character image to a position consistent with the vertex group and / or face group of the three-dimensional surface shape of the represented object. The image appending step applies an image processed from the object's photographic data to the surface of the general-purpose virtual character image after the vertex group and / or face group have been moved through the surface adjustment step.

10. The three-dimensional virtual character generation program product according to claim 9, characterized in that: The general-purpose virtual character image specifies motion information, shape transformation information, and information related to vertex groups representing internal shapes. The motion information relates to the overall positional changes of the skeleton structure and the linkage between its constituent elements during motion. The shape transformation information specifies the transformation method of the vertex groups when transforming from a first shape to a second shape in at least a portion of a three-dimensional surface shape. The object being moved in the surface adjustment step does not include the vertex group representing the internal shape.

11. The three-dimensional virtual character generation program product according to claim 10, characterized in that: The aforementioned 3D virtual character generation program also enables the computer to perform surface area judgment steps, basic body shape generation steps, and shape transformation steps. The surface region determination step identifies the exposed skin region, clothing region, and hair region in the three-dimensional surface shape based on the object's captured data. The basic body shape generation step removes the clothing portion from the clothing area identified in the surface area determination step and generates the surface shape of the basic body shape. The shape transformation step, based on the shape transformation information, causes a shape change in the surface shape of the generic virtual character image to match the basic body shape. In the surface adjustment step, at least the clothing area and the hair area are not subjected to surface adjustment processing, and a three-dimensional virtual character image is generated. This three-dimensional virtual character image does not have information related to clothing and hair from the object's shooting data.

12. The three-dimensional virtual character generation program product according to claim 11, characterized in that: The aforementioned 3D virtual character generation program also enables the computer to perform steps such as clothing area extraction, clothing body generation, body connection, general shape shaping, information transmission, and connection unlinking. The clothing region extraction step extracts the surface structure, which is located on the three-dimensional surface shape of the clothing region based on the object's photographic data. The clothing model generation step converts the extracted portion into a three-dimensional clothing model. The body connection step connects the clothing body to the 3D virtual character image based on the shooting data. The generalized shape transformation step, based on the shape transformation information, causes the product formed by connecting the 3D virtual character image and the clothing substrate to undergo shape changes to conform to the 3D surface shape of the generalized virtual character image. The information transmission step transmits information related to the three-dimensional surface shape and skeletal structure of the universal virtual character image, as well as the shape transformation information, to the clothing body after the shape change. The connection release step, after the information transmission step transmits information, releases the connection between the clothing body and the three-dimensional virtual character image.

Images