Information processing system, information processing method, and information processing program

Abstract

To provide an information processing system which can reproduce motion of clothing following motion of a person in the real world, in a virtual space.SOLUTION: An information processing system includes: a generation part for generating a clothing model displayable in a three-dimensional virtual space, on the basis of a plurality of images obtained by photographing clothing from a plurality of directions; an imparting part for imparting adjustment points to predetermined parts of the clothing model; an association part for associating bones corresponding to predetermined parts of the clothing model with the adjustment points of the clothing model, out of a plurality of bones set to a three-dimensional human model made to wear the clothing model; and a simulation part for simulating motion of the clothing model, when the human model wearing the clothing model is operated on the basis of motion data defining continuous operation of the human model, and starting a simulation, from a state in an initial attitude of the human model where the bones associated with the respective adjustment positions are positioned at positions of the respective adjustment points in the clothing model having an initial shape which is a shape when the clothing is photographed.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] In recent years, technological developments related to virtual space have progressed with the development of information and communication technology, and many services utilizing virtual space have been proposed in the fashion world. For example, there are services that allow users to visit virtual stores that reproduce stores existing in the real world in virtual space and browse and purchase products, and services that allow users to purchase clothes, shoes, and the like (virtual items) to be worn by avatars traveling through virtual space. In addition, NFT (Non-Fungible Token) technology makes it possible to prevent counterfeiting and tampering of virtual items, making it possible to trade virtual items while maintaining their inherent value, and the distribution of virtual items is progressing (for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Metaverse Research Institute, “[14 Case Studies] How to Use Metaverse in the Apparel Industry and Its Benefits,” [online], searched May 22, 2023, Internet<URL:https: / / metaversesouken.com / metaverse / fashion-2 / > Summary of the Invention [Problem to be solved by the invention]

[0004] The applicant has a large collection of rare vintage clothing, and is attempting to digitize the clothing while protecting its value through NFTs and circulate it in virtual space. However, when real-world clothing is digitized and worn by an avatar, it is difficult to reproduce the movement of the clothing in the virtual space that follows the movement of a person in the real world. [Means for solving the problem]

[0005] An information processing system according to one embodiment of the present invention includes a generation unit that generates a clothing model that can be displayed in a three-dimensional virtual space based on a plurality of images of the clothing taken from a plurality of directions; an assignment unit that assigns adjustment points to predetermined locations of the clothing model; an association unit that associates bones that correspond to predetermined locations of the clothing model among a plurality of bones set in a three-dimensional human body model on which the clothing model is to be worn with the adjustment points of the clothing model; and a simulation unit that simulates the movement of the clothing model when the human body model wearing the clothing model is moved based on motion data that defines the continuous movement of the human body model, and the simulation unit starts the simulation from a state in which the human body model is in an initial posture in which bones associated with each adjustment point are positioned at the position of each adjustment point on the clothing model in an initial shape, which is the shape of the clothing when photographed.

[0006] In an information processing system according to an embodiment of the present invention, the predetermined locations may be at least three or more points that enable calculation of the opening angle of the sleeve relative to the body of the clothing in the initial shape of the clothing model.

[0007] In an information processing system according to one embodiment of the present invention, a simulation unit operates a human body model on a virtual surface based on motion data, and in a simulation based on weight data included in a clothing model that corresponds to the material of the clothing, determines whether or not a collision occurs between a clothing model worn by the human body model and the virtual surface, and may repeatedly execute a simulation from an initial posture while changing the weight data until the operation based on the motion data is completed without a collision.

[0008] An information processing system according to one embodiment of the present invention further includes a memory unit that stores, in association with the human body model, animation data that displays an animation based on the motion data of a human body model wearing the clothing model, with the movement of the clothing model based on weight data when an operation based on the motion data is completed without any collision, and the memory unit may store animation data for each of a plurality of combinations of a plurality of motion data, a plurality of human body models, and a plurality of clothing models.

[0009] The information processing system according to one embodiment of the present invention may further include a learning unit that generates a learning model that outputs adjustment points and weight data for a predetermined clothing model by machine learning, which inputs conditions related to a clothing model including weight data, adjustment points, a human body model, and motion data in a simulation executed by the simulation unit, and outputs a simulation result based on the conditions.

[0010] An information processing system according to one embodiment of the present invention further includes a virtual space generation unit that generates a virtual space in which a plurality of clothing models are arranged, and an output unit that outputs display data for displaying the virtual space on a user's display device to the display device, and in response to detecting a user's operation of selecting a clothing model in the virtual space, the output unit may transmit animation data to the display device for displaying an animation of a human body model wearing the clothing model selected by the user.

[0011] The information processing system according to an embodiment of the present invention may further include a reception unit that receives arrangement instructions regarding the arrangement of the clothing model in the virtual space from a plurality of administrative users of the virtual space.

[0012] An information processing method executed by an information processing device according to one embodiment of the present invention includes the steps of generating a three-dimensional clothing model based on a plurality of images of clothing photographed from a plurality of directions, assigning adjustment points to predetermined locations of the clothing model, associating bones corresponding to predetermined locations of the clothing model among bones set in a three-dimensional human body model on which the clothing model will be worn with the adjustment points of the clothing model, and simulating movement of the clothing model when the human body model wearing the clothing model is moved based on motion data defining continuous movements of the human body model, and the simulating step starts the simulation from a state in which the human body model is in an initial posture in which bones associated with each adjustment point are positioned at the position of each adjustment point on the clothing model in an initial shape, which is the shape of the clothing when photographed.

[0013] An information processing program according to one embodiment of the present invention provides an information processing device that realizes the following functions: generating a three-dimensional clothing model based on a plurality of images of clothing photographed from a plurality of directions; assigning adjustment points to predetermined locations of the clothing model; associating bones corresponding to predetermined locations of the clothing model among bones set in a three-dimensional human body model wearing the clothing model with the adjustment points of the clothing model; and simulating movement of the clothing model when the human body model wearing the clothing model is moved based on motion data defining the continuous movement of the human body model, wherein the simulation function starts a simulation from a state in which the human body model is in an initial posture in which bones associated with each adjustment point are positioned at the position of each adjustment point on the clothing model in an initial shape, which is the shape of the clothing when photographed. Effect of the Invention

[0014] According to one embodiment of the present invention, it is possible to provide an information processing system or the like that can reproduce the movement of the clothing that follows the movement of a person in the real world as the movement of the clothing that follows the movement of the avatar in a virtual space when clothing that exists in the real world is digitized and worn by an avatar. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic diagram of the configuration of an information processing system according to one embodiment of the present invention. [Diagram 2] FIG. 2 is an example of a functional block diagram of an information processing system according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a flowchart showing an example of an information processing method according to an embodiment of the present invention. [Figure 4] 4(a) to (c) are diagrams for explaining an outline of an information processing system according to one embodiment of the present invention. [Diagram 5] 5(a) and (b) are diagrams for explaining an outline of an information processing system according to one embodiment of the present invention. [Figure 6] FIG. 6 is a flowchart showing an example of an information processing method according to an embodiment of the present invention. [Figure 7] FIG. 7 is an example of a data table stored in an information processing system according to an embodiment of the present invention. [Figure 8] FIG. 8 is a schematic diagram showing an example of a virtual space realized in an information processing system according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the invention (also referred to as the present invention) according to the present disclosure will be described with reference to the drawings. Note that the drawings are merely examples, and the present invention is not limited to those shown in the drawings. For example, the illustrated server (information processing device), block diagram of a user terminal (communication device), flow chart, data table, display screen of the user terminal, etc. are merely examples, and the present invention is not limited to these.

[0017] In addition, in this specification, "virtual space" refers to a virtual information space that does not physically exist and is constructed on an information processing device (computer) or a network, and includes the so-called "metaverse," a space in which a user can interact with others by operating an avatar. An "avatar" is a character object that anthropomorphizes a person, animal, living thing, object, etc., and may also be called a "digital human (virtual human)." In the following, an avatar will be an example of a person, and may also be called a "human body model."

[0018] In this specification, "clothing" refers to anything worn by a person or avatar, and includes pants, skirts, dresses, all-in-ones, shirts, T-shirts, jackets, coats, swimsuits, etc., regardless of whether the clothing is Western or Japanese. Furthermore, "clothing" may include accessories such as hats, scarves, gloves, ties, belts, etc., accessories such as necklaces, earrings, watches, glasses, masks, etc., footwear such as shoes, sandals, geta, etc., and masks as hygiene products.

[0019] <System configuration> Fig. 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. The information processing system 600 is an information and communication system that provides a clothing model that is a digitalized version of clothing that exists in the real world (hereinafter also referred to as "real clothing") and that can be worn by a human body model, and that makes it possible to reproduce the movement of the clothing in the real world in a virtual space. The information processing system 600 may also be an information and communication system that provides a digital museum in which a plurality of real clothing items are digitized and exhibited in a state in which the stereoscopic three-dimensional (3D) shapes can be viewed.

[0020] The information processing system 600 includes at least a server 100, a database server 400, and a user's communication terminal (user terminal) 200 (200A, 200B). The server 100 is connected to the communication terminals 200A, 200B via a network 500, and transmits and receives various information to and from the communication terminal 200, and executes various processes related to the information processing system. Although only one server 100 is shown in FIG. 1, the present invention is not limited to this, and multiple servers may be present. The server 100 is a distributed server system that cooperates by communicating via a network, and may include an edge server or may be a so-called cloud server. That is, the server 100 is not limited to a physical server, and may include a virtual server. The functions or processes of each functional unit of the server 100 may be realized by machine learning or AI (Artificial Intelligence) to the extent that they can be realized.

[0021] Network 500 may include a wireless network or a wired network, such as a wireless LAN (WLAN), a wide area network (WAN), integrated service digital networks (ISDNs), wireless LANs, code division multiple access (CDMA), long term evolution (LTE), LTE-Advanced, fourth generation (4G), fifth generation (5G), sixth generation (6G) and subsequent mobile communication systems, or a combination thereof.

[0022] The user terminals 200A and 200B may be communication terminals of users who use the virtual space service provided by the information processing system 600. In Fig. 1, only two user terminals are shown, 200A and 200B, for users A and B, respectively. However, the number of user terminals may be the same as the number of users who use the virtual space service, and when no particular distinction is made, the user terminals will simply be referred to as user terminals 200.

[0023] The user terminal 200 may be installed with an application (hereinafter, also referred to as an "app") for using the virtual space service provided by the information processing system 600, and may be capable of transmitting and receiving various information between the user terminal 200 and the server 100. Note that it is not essential to install an application in the user terminal 200, and the user terminal 200 and the server 100 may transmit and receive various information via a web browser. Note that, although FIG. 1 shows a smartphone as the user terminal 200A and a notebook computer as the user terminal 200B, the user terminal 200 may be any terminal that can realize the functions described in each embodiment described hereinafter. For example, the user terminal 200 may be a computer (e.g., a tablet), a handheld computer device (for example, but not limited to, a PDA (personal digital assistant), a wearable terminal (such as a glasses-type device or a watch-type device), or an HMD (Head Mount Display).

[0024] The database server 400 stores various information (data) used in the information processing system 600. Although only one database server 400 is shown in FIG. 1, separate from the server 100, the database server 400 may be integrated into the server 100. That is, the database server 400 may be a volatile memory or a non-volatile memory of the server 100. The database server 400 may be composed of a plurality of storage devices. The database server 400 may be connected to the server 100 through a dedicated internal network different from the network 500, or may be connected to the server 100 via the network 500.

[0025] <User device> 2 is a configuration diagram of a user terminal 200 according to an embodiment of the present invention. The hardware configuration and functional configuration of the user terminal 200 according to an embodiment of the present invention will be described with reference to FIG.

[0026] (1) Hardware configuration of user terminal The user terminal 200 includes a control unit 210 , a communication unit 220 , a display unit 230 , an input / output unit 240 , and a storage unit 270 .

[0027] The control unit 210 is typically a processor, and is realized by a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), etc. The control unit 210 reads out a program stored in the storage unit 270, and executes a code or instruction included in the read out program, thereby executing the functions and methods shown in each embodiment.

[0028] The storage unit 270 stores various programs and various data required for the operation of the user terminal 200. The storage unit 270 may include, for example, a flash memory or the like, and may also include a memory (such as a RAM (Random Access Memory), a ROM (Read Only Memory)) that provides a working area for the control unit 210.

[0029] The communication unit 220 may be implemented as hardware such as a NIC (Network Interface Card), communication software, or a combination of these, and may transmit and receive various data to and from the server 100 via the network 500. The communication may be performed either wired or wirelessly, and any communication protocol may be used as long as the communication between the two units is possible. When the communication unit 220 is configured as a physically structured circuit, it may be expressed as a communication circuit.

[0030] Display unit 230 is a monitor that displays data according to the display data written to the frame buffer, and may be, for example, a touch panel, a touch display, or the like.

[0031] The input / output unit 240 may include an input device for inputting various operations to the user terminal 200, and an output device for outputting the processing results processed by the user terminal 200. The input / output unit 240 may be an integrated input device and an output device, or may be separated into an input device and an output device. The input device may be realized by any of all types of devices that can accept an input operation from the user 20 and transmit information related to the input to the control unit 210, or a combination thereof. The input device may include, for example, a touch panel, a touch display, a camera, and a microphone. The output device may output the processing results processed by the control unit 210. The output device may include, for example, a display, a touch panel, a speaker, etc.

[0032] (2) Functional configuration of user terminal The user terminal 200 includes a control unit 210. The control unit 210 controls communication with the server 100 via the network 500 by the communication unit 220, and executes transmission and reception of various information. The control unit 210 also controls the display of data on the display unit 230, and for example, causes the display unit 230 to display a screen corresponding to information transmitted from the server 100. Furthermore, the control unit 210 controls the transmission of various information with an external device via the input / output unit 240, and for example, transmits various information to each functional unit in response to a user's input operation received by an input device, and transmits information from each functional unit to an output device (not shown), such as a touch panel, a monitor, or a speaker.

[0033] <Server> Fig. 2 is a configuration diagram of the server 100 according to one embodiment of the present invention. The hardware configuration and functional configuration of the server (information processing device) 100 according to one embodiment of the present invention will be described with reference to Fig. 2.

[0034] (1) Server hardware configuration The server 100 includes, as hardware components, a control unit 110, a communication unit 120, an input / output unit 130, and a storage unit 170. Although not shown, the server 100 may also include components that are included in a general server.

[0035] The control unit 110 is typically a processor, and may be a central processing unit (CPU), an MPU, etc. The control unit 110 may execute the functions and methods described in each embodiment by reading a program stored in the storage unit 170 and executing a code or instruction included in the read program.

[0036] The storage unit 170 stores (contains) various programs and various data required for the operation of the server 100. For example, the storage unit 170 may store data regarding the movement of a clothing model that follows the movement of a human body model wearing the clothing model, as will be described in detail later. The storage unit 170 may include, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage unit 170 may also include a memory that provides a working area for the control unit 110.

[0037] The communication unit 120 is implemented as hardware such as a NIC, communication software, or a combination of these, and transmits and receives information between the server 100 and other devices.

[0038] (2) Functional configuration of the server The server 100 includes a generating unit 111, an assigning unit 112, an associating unit 113, a simulation unit 114, an output unit 115, a learning unit 116, and a receiving unit 117 as functions realized by the control unit 110. Note that in Fig. 2, functional units that are not essential to each embodiment described hereinafter may be omitted. Furthermore, the functions or processes of each functional unit may be realized by machine learning or AI to the extent that they can be realized.

[0039] Hereinafter, one embodiment of the present invention will be described with reference to FIGS. 3 to 7 along with the processing of each functional unit of the server 100.

[0040] 3 is an example of a flowchart showing processing by the server 100 according to an embodiment. First, the generation unit 111 generates a clothing model that can be displayed in a three-dimensional virtual space based on a plurality of images of clothing captured from a plurality of directions (step S11). Here, "clothing model that can be displayed in a three-dimensional virtual space" may refer to displaying a clothing model so that the user can visually recognize it three-dimensionally and stereoscopically.

[0041] As described above, in one embodiment of the present invention, the clothing model is generated by photogrammetry based on a plurality of images of the clothing taken from a plurality of directions. Note that "photogrammetry" is a technique for taking images of a subject from various angles, analyzing and integrating the digital images to create a three-dimensional 3DCG (3 Dimensional Computer Graphics) model. The method of taking images from a plurality of directions is not particularly limited, and a large number of (e.g., 200) shooting units (cameras) may be regularly installed so as to surround the subject to be photographed, and a plurality of images may be taken, or a plurality of images may be taken by rotating a turntable on which the subject is placed, while taking images with a camera. Alternatively, a plurality of images may be taken by not moving the subject to be photographed, but moving the camera. Note that an existing method may be used for generating a three-dimensional model from a plurality of images.

[0042] 4(a) is an example of a clothing model generated by the generation unit 111. The generation unit 111 may generate the clothing model 10 after erasing, through image processing, any object used to fix the clothing, such as a torso or mannequin, that was used when photographing the clothing. The clothing model 10 may include polygon meshes, texture information, and weighting data based on the material of the clothing.

[0043] The assigning unit 112 assigns an adjustment point to a predetermined location of the clothing model 10 (step S12). The "adjustment point" may be a virtual point (marker) used for aligning the clothing model with the human body model, which will be described later. FIGS. 5(a) and 5(b) are diagrams for explaining the predetermined location and the adjustment point. In one embodiment, the predetermined location may be at least three or more points that enable calculation of the opening angle of the sleeve with respect to the body of the clothing in the initial shape of the clothing model 10. The initial shape refers to the shape of the clothing at the time of photographing. As will be described in detail later, when the clothing model is worn by the human body model and the movement of the clothing that follows the movement of the human body model is simulated, a phenomenon occurs in which the clothing model slips off the human body model. According to one embodiment of the present invention, the possibility of the clothing slipping off during the simulation can be reduced by using the adjustment points assigned to the predetermined locations of the clothing model.

[0044] 5(a) and 5(b) are schematic diagrams for explaining the cases of a dress and a T-shirt as clothing models, respectively. As shown in FIG. 5(a), when the clothing model 10 is a dress, the adjustment points may be assigned to the neck part M10, the right wrist part M11, the left wrist part M12, the waist part M13, the right leg part M14, and the left leg part M15. As shown in FIG. 5(b), when the clothing model 10' is a T-shirt, the adjustment points may be assigned to the neck part M20, the right cuff part M21, the left cuff part M22, and the waist part M23. The adjustment points may be designated based on the operation of an administrator user from an administrator user terminal (not shown) operated by an administrator user who provides a virtual space service. Alternatively, the assigning unit 112 may automatically assign adjustment points to any clothing model using a learning model that is generated in advance by learning the correspondence between the clothing model and the adjustment points assigned to the clothing model. The assigned adjustment points may be corrected by a user, or the correction may be performed by a program.

[0045] As described above, the adjustment points may be set according to the type of clothing. For example, adjustment points may be assigned to the neck, right wrist, and left wrist for a T-shirt, to the waist, right leg, and left leg for a skirt, to the neck, right wrist, left wrist, waist, right leg, and left leg for a dress, and to the waist, right knee, left knee, right leg, and left leg for pants. Adjustment points may also be applied to clothing without sleeves, and adjustment points may be set to the shoulders and waist for a tank top, and to the sides and waist for a tube top. When the adjustment points are specified by the administrative user terminal, the adjustment points to be set may be notified by the administrative user terminal according to the type of clothing model.

[0046] The associating unit 113 associates, among the bones set in the three-dimensional human body model on which the clothing model 10 is to be worn, bones corresponding to predetermined locations on the clothing model 10 with adjustment points on the clothing model 10 (step S13). FIG. 4(b) shows the relationship between the human body model 21 and the bones 22. Generally, in three-dimensional animation (3D animation), a bone indicates an element that is a unit of movement for moving a three-dimensional model, and animation is generated by the three-dimensional model deforming to follow the movement of the bone. When creating animation of the human body, joints 23 are defined in the same way as for the human body, and bones 22 are connected to each other by the joints 23, so that the bones can be bent or rotated at the joints 23.

[0047] In the example shown in FIG. 5(a), the associating unit 113 associates the adjustment points assigned to the neck, right wrist, left wrist, waist, right leg, and left leg in the clothing model 10 with bones corresponding to the neck, right wrist, left wrist, waist, right leg, and left leg in the human body model 21. Also, in the example shown in FIG. 5(b), the associating unit 113 associates the adjustment points assigned to the neck, right cuff, left cuff, and waist in the clothing model 10' with bones corresponding to the neck, right elbow, left elbow, and waist in the human body model 21. Here, "associating" may refer to setting the adjustment points so that the positions of the bones match. As a result, the adjustment points move following the movement of the bones.

[0048] The simulation unit 114 simulates the movement of the clothing model 10 when the human body model 21 wearing the clothing model 10 is moved based on the motion data (step S13). The motion data may be data that defines continuous movements of the human body model 21. The simulation of the movement of the clothing model 10 may be a so-called cloth simulation, which is a physical calculation related to the physical laws that act on cloth.

[0049] The process of the server 100 according to an embodiment of the present invention will be described below with reference to the flowchart of Fig. 6. The simulation unit 114 starts the simulation from a state in which the human body model 21 is in an initial posture in which bones associated with each adjustment point are initially positioned at the positions of each adjustment point in the clothing model of the initial shape, which is the shape of the clothing when photographed (step S21). This will be described with reference to Figs. 4 and 5.

[0050] The clothing model 10 in Fig. 5 is an initial shape, and the simulation unit 114 may deform the posture of the human body model 21 so that adjustment points of the clothing model 10 in the initial shape in Fig. 5 match bones of the human body model 21 associated with the adjustment points. Here, "deformation" may include changing the lengths of parts of the human body model 21, such as the arms, shoulder width, and sitting height. The simulation unit 114 also calculates the shoulder angle θ of the clothing model 10 in the initial shape shown in Fig. 5 by the following formula. Here, the shoulder angle θ may be the opening angle of the sleeve with respect to the body of the clothing in the initial shape of the clothing model 10.

[0051]

number

[0052] In the above equation, α is the vector between the neck and the wrist, and β is the vector between the neck and the waist.

[0053] The simulation unit 114 may start the cloth simulation in a state where the arms of the human body model 21 are transformed to a position (initial posture) where the opening angle from the torso is θ, and then the human body model 21 is made to wear the clothing model 10. The simulation unit 114 operates the human body model 21 on a virtual surface based on the motion data, and performs a simulation based on weight data included in the clothing model 10 according to the material of the clothing (step S22). For example, as shown in FIG. 4(c), the simulation unit 114 may simulate the change over time of the clothing model when the human body model 21 walks on the ground in the virtual space. At this time, the simulation unit 114 performs the cloth simulation based on the weight set in the polygon mesh of the clothing according to the type of fabric (silk, denim, linen, etc.).

[0054] The simulation unit 114 may determine whether or not a collision has occurred between the clothing model 10 worn by the human body model 21 and a virtual surface (ground) while the human body model 21 is performing an action based on the motion data (step S23). The occurrence of a collision means that the clothing model 10 has slipped off the human body model 21. If the determination in step S23 is YES, the simulation unit 114 may change the weight data set for the clothing model 10 (step S24). The clothing model 10 may slip off, for example, depending on the relationship between the shoulder width of the human body model 10 and the depth of the neckline of the clothing model 10. If the clothing model 10 slips off, the simulation unit 114 may change the weight of the clothing model 10 in a direction to control the slipping off. For example, the simulation unit may partially increase the weight of the clothing model 10. That is, the simulation unit 114 may prevent the clothing model 10 from slipping off by partially changing (increasing the weight) in the clothing model 10 in addition to the weight for reproducing gravity, which is generally set in the cloth simulation. The parts of the clothing model 10 whose weights are changed may be, for example, shoulders, collars, etc., but are not limited thereto. The weights may be lightened. The simulation unit 114 may execute the cloth simulation of step S22 again using the weight data changed in step S24.

[0055] If no collision occurs between the clothing model 10 and the ground during the simulation (NO in step S23) and the movement of the human body model 21 based on the motion data is completed (i.e., the simulation is successful), the simulation unit 114 associates animation data with the human body model 21 to display an animation based on the motion data of the human body model 21 wearing the clothing model 10, with the clothing model 10 moving according to the weight data when the movement based on the motion data is completed without a collision, and stores the animation data in the database server 400 (step S25). The animation data may include a change over time in the polygon mesh of the clothing model 10 (a change over time in the shape of the clothing model 10) when the human body model 21 is made to walk while wearing the clothing model 10.

[0056] The simulation unit 114 may perform the above-mentioned simulation for multiple combinations of multiple motion data, multiple human body models, and multiple clothing models. Then, animation data when the simulation is successful may be stored in the database server 400. That is, the database server 400 may store the animation data for each of multiple combinations of multiple motion data, multiple human body models, and multiple clothing models, and FIG. 7 shows an example of an animation data table stored in the database server 400. Note that the figure is an example, and the data stored in the database server 400 may be less or more than this, and the data format is not limited to that shown in the figure.

[0057] In the animation data table TB10, for each identification information for identifying each human body model (in the example shown in the figure, "avatar_X", "avatar_Y", "avatar_Z"), animation data associated with identification information for multiple clothing models (cloth_1, cloth_2, cloth_3) and identification information for motion data (motion_1, motion_2) are stored in such a manner that they can be identified by the identification information (ID: Identifier) ​​of the animation data. For example, animation data identified by animation ID "animation_X11" indicates the change over time in the shape of the clothing model when human body model "avatar_X" wears clothing model "cloth_1" and moves according to motion data "motion_1". Animation data identified by animation ID "animation_X22" indicates the change over time in the shape of the clothing model when human body model "avatar_X" wears clothing model "cloth_2" and moves according to motion data "motion_2".

[0058] The user terminal 200 can receive from the user a selection operation of a human body model, clothing, and motion for which animation is desired, and transmit the selection operation to the server 100 as an animation request. The output unit 115 of the server 100 can select animation data corresponding to the request from the database server 400, and output the selected animation data to the user terminal 200 via the communication unit 120. The display unit 230 of the user terminal 200 can display an animation in which the clothing worn by the avatar 21 moves in accordance with the walking of the avatar 21, as shown in FIG. 4(c), based on the animation data received from the server 100.

[0059] In this way, according to one embodiment of the present invention, clothing data, which is three-dimensional data of actual clothing, is generated by photogrammetry, and the manner in which the clothing moves in accordance with the movement of the human body model can be reproduced in animation. At this time, by starting the simulation by matching the shape of the human body model to the clothing data, it is possible to reduce the possibility of the phenomenon of the clothing slipping off the human body model and improve the probability of successful cloth simulation.

[0060] According to one embodiment of the present invention, the learning unit 116 may input conditions related to a clothing model including weight data, adjustment points, a human body model, and motion data in a simulation executed by the simulation unit 114, and generate a learning model that outputs adjustment points and weight data for a predetermined clothing model by machine learning that outputs a simulation result based on the conditions.

[0061] This reduces the probability of a simulation failure and reduces the computational load.

[0062] <Clothing model display> As described above, the applicant has a large collection of rare vintage clothing, and provides a museum (digital museum) in a virtual space that displays these clothing. FIG. 8 is an example of a digital museum displayed on the user terminal 200. As shown in FIG. 8, the digital museum 40 may be a virtual space in which a plurality of clothing models 41 are arranged, and a user may be able to visit the digital museum via an avatar. That is, the server 100 may further include a virtual space generating unit (not shown) and generate display information for displaying a virtual space in which the clothing models are arranged. Note that the above-mentioned animation of the selected clothing model may be started on the user terminal 200 when the clothing model 41 is selected by the user's avatar.

[0063] The reception unit 117 of the server 100 may receive arrangement instructions regarding the arrangement of the clothing models 41 in the virtual space (digital museum) 40 from multiple administrative users of the virtual space. That is, the server 100 may connect multiple user terminals 200 to the same virtual space 40 simultaneously in response to requests from the user terminals 200 of the multiple administrative users. This may mean that the avatars of multiple administrative users exist in the same virtual space 40 simultaneously.

[0064] In one embodiment of the present invention, data for displaying a plurality of clothing models 41 is stored in, for example, a database server 400, and an administrative user may access the database server 400 and place the clothing models 41 in the virtual space 40. Note that "placing in the virtual space" may mean setting position coordinates of the clothing model 41 in a coordinate system set for the virtual space. This allows the user to visually recognize the clothing model 41 as if it were placed in the virtual space 40.

[0065] The clothing model 41 arranged in the virtual space 40 may be operable (accessible) by the administrative user. For example, according to one embodiment of the present invention, the avatar of the administrative user may be able to carry the clothing model 41 in the virtual space 40. That is, the reception unit 117 of the server 100 receives an instruction to arrange the clothing model 41 in response to an operation instruction inputted by the input / output unit 240 in the user terminal 200 of the administrative user 100. In response to the arrangement instruction, the output unit 115 of the server 100 may output display information to the user terminal 200 of another administrative user present in the virtual space 40, which displays a moving image showing the administrative user's avatar carrying the clothing model 41. Note that the avatars in the virtual space 40 may be able to converse with each other by voice chat or text chat using existing technology, and the administrative users may be able to exchange opinions with each other about the arrangement of the clothing model 41.

[0066] Different authorities may be given to different administrative users, and administrative users who can access the database server 400 and move the clothing model 41 within the virtual space 40 may be restricted. This makes it possible to prevent the same clothing model 41 from being duplicated.

[0067] As described above, according to one embodiment of the present invention, since collaborative work by a plurality of users is possible, a virtual space can be efficiently designed.

[0068] Note that objects that can be placed in the virtual space 40 are not limited to clothing models. For example, living things such as animals, fish, and insects, plants such as grass and flowers, and crafts and artworks such as paintings, ceramics, and sculptures may be placed in the virtual space 40 in response to a user's instruction.

[0069] Although the present invention has been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these modifications and corrections are included in the scope of the present invention. For example, the functions included in each component, each step, etc. can be rearranged so as not to cause logical contradictions, and multiple components, steps, etc. can be combined into one or divided. In addition, the configurations shown in the above embodiments may be appropriately combined.

[0070] For example, in the above description, weight data is adjusted when a simulation fails, but the parameters to be adjusted are not limited to this and may be the size of a polygon mesh, a bone to which an adjustment point is associated, or the like.

[0071] In the above, an animation for a combination of a human body model, a clothing model, and motion data is stored in advance, and the animation data is output according to a user's selection. However, a cross simulation may be performed in which a human body model (user avatar) of the user himself / herself is generated and the user avatar is made to wear a clothing model.

[0072] The programs of the embodiments of the present disclosure may be provided in a state stored in a storage medium readable by an information processing device. The storage medium can store the programs in a "non-transitory tangible medium." The programs include, for example, software programs and control programs. When each functional unit of the server is realized by software, the server functions as a generating unit 111, an assigning unit 112, an associating unit 113, a simulating unit 114, an output unit 115, a learning unit 116, and a receiving unit 117 by the processor executing the programs loaded on the memory.

[0073] The storage medium may, where appropriate, include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs), application specific ICs (ASICs), etc.), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable storage media, or any suitable combination of two or more of these. The storage medium may, where appropriate, be volatile, non-volatile, or a combination of volatile and non-volatile.

[0074] Furthermore, the program of the present disclosure may be provided to a server via any transmission medium capable of transmitting the program (such as a communication network or broadcast waves).

[0075] In addition, each embodiment of the present disclosure may be realized in the form of a data signal embedded in a carrier wave in which a program is embodied by electronic transmission. The program of the present disclosure may be implemented using, for example, a script language such as JavaScript (registered trademark) or Python, C language, Go language, Swift, Koltin, Java (registered trademark), or the like. [Explanation of symbols]

[0076] 100 Server (information processing device) 110 Control section 111 Generation part 112 Granting Department 113 Association section 114 Simulation Department 115 Output section 116 Learning Department 117 Reception 120 Communications Department 130 Input / output section 170 Storage section 200 User terminal (communication device) 210 Control section 220 Communications Department 230 Display section 240 Input / output section 270 Storage section 400 Database Server 500 Network 600 Information Processing Systems 10,10′ Clothing model 21 Human Body Model 22 Bone 23 Joints M10~M15,M20~M23 adjustment point 40 Virtual Space 41 Clothing Model

Claims

1. a generation unit that generates a three-dimensional clothing model based on a plurality of images obtained by photographing the clothing from a plurality of directions; an adjustment point adding unit that adds an adjustment point to a predetermined location of the clothing model; an associating unit that associates bones corresponding to predetermined locations of the clothing model among bones set in a three-dimensional human body model on which the clothing model is to be worn with adjustment points of the clothing model; a simulation unit that simulates a movement of the clothing model when the human body model wearing the clothing model is moved based on motion data that defines a continuous movement of the human body model; Equipped with the simulation unit starts the simulation from a state in which the human body model is in an initial posture in which bones associated with each adjustment point are positioned at the positions of the adjustment points in the clothing model in an initial shape, which is a shape of the clothing at the time of photographing the clothing; Information processing system.

2. The predetermined locations are at least three or more points that enable calculation of an opening angle of the sleeve relative to the body of the clothing in the initial shape of the clothing model. The information processing system according to claim 1 .

3. the simulation unit operates the human body model on a virtual surface based on the motion data, determines whether or not a collision occurs between the clothing model worn by the human body model and the virtual surface in the simulation based on weight data included in the clothing model and corresponding to a material of the clothing, and repeatedly executes the simulation from the initial posture while changing the weight data until the movement based on the motion data is completed without a collision occurring. The information processing system according to claim 1 .

4. a storage unit that stores, in association with the human body model, animation data for displaying an animation based on the motion data of the human body model wearing the clothing model, with the clothing model moving according to weight data when the action based on the motion data is completed without the occurrence of the collision; the storage unit stores the animation data for each of a plurality of combinations of a plurality of motion data, a plurality of human body models, and a plurality of clothing models; The information processing system according to claim 3 .

5. a learning unit that generates a learning model that outputs adjustment points and weight data for a predetermined clothing model by machine learning using input of conditions related to a clothing model including weight data, adjustment points, a human body model, and motion data in a simulation executed by the simulation unit, and outputs a simulation result based on the conditions; 5. The information processing system according to claim 4.

6. a virtual space generating unit that generates a virtual space in which a plurality of the clothing models are arranged; an output unit that outputs display data for displaying the virtual space on a user's display device to the display device; Further equipped with the output unit, in response to detecting a selection operation of the clothing model in the virtual space by the user, transmits to the display device animation data for displaying an animation of a human body model wearing the clothing model selected by the user.

6. The information processing system according to claim 5.

7. a reception unit that receives arrangement instructions regarding the arrangement of the clothing model in the virtual space from a plurality of administrative users of the virtual space; The information processing system according to claim 6 , further comprising:

8. An information processing method executed by an information processing device, generating a three-dimensional clothing model based on a plurality of images of the clothing taken from a plurality of directions; adding adjustment points to predetermined locations of the clothing model; a step of associating bones corresponding to predetermined positions of the clothing model, among bones set in a three-dimensional human body model on which the clothing model is to be worn, with adjustment points of the clothing model; a step of simulating a movement of the clothing model when the human body model wearing the clothing model is moved based on motion data defining a continuous movement of the human body model; Including, the simulating step starts the simulation from a state in which the human body model is in an initial posture in which bones associated with each adjustment point are positioned at positions of the adjustment points in the clothing model of an initial shape, which is a shape of the clothing at the time of photographing the clothing; Information processing methods.

9. In the information processing device, A function for generating a three-dimensional clothing model based on multiple images of the clothing taken from multiple angles; A function of adding adjustment points to predetermined locations of the clothing model; a function of associating bones corresponding to predetermined locations of the clothing model, among bones set in a three-dimensional human body model on which the clothing model is to be worn, with adjustment points of the clothing model; a function of simulating a movement of the clothing model when the human body model wearing the clothing model is moved based on motion data that defines a continuous movement of the human body model; Realize this, The function of simulating starts the simulation from a state in which the human body model is in an initial posture in which bones associated with each adjustment point are positioned at the positions of the adjustment points in the clothing model of an initial shape, which is the shape of the clothing at the time of photographing the clothing; Information processing program.

Images