Metaverse character implementation system and metaverse object implementation device

Abstract

A metaverse character implementation system and a metaverse object implementation device are disclosed. The metaverse character implementation system comprises: a motion capture device for generating user capture data by tracking the motion of a user interacting with an object; a proximity detection device for generating proximity detection data by detecting whether the user is in proximity to the object; and a metaverse object implementation device for generating metaverse character image data on the basis of the user capture data and the proximity detection data.

Description

Metaverse character implementation system and metaverse object implementation device

[0001] The present invention relates to a metaverse character implementation system and a metaverse object implementation device. Specifically, the present invention relates to a metaverse character implementation system and a metaverse object implementation device that implement a metaverse character that interacts with real objects.

[0002]

[0003] Metaverse characters are being created through a variety of methods. Recently, a widely used method is to detect the movements of an actor with multiple markers attached and generate a metaverse character skeleton. By rigging the generated skeleton to a pre-created character image, the character's movements are reflected in the actual actor's movements, further enhancing the vividness of metaverse characters.

[0004] However, when actors interact with real objects, implementing metaverse characters interacting with objects in the metaverse virtual space presents several challenges. Specifically, due to differences in the characteristics of actors and metaverse characters, objects frequently end up being placed in locations other than the body parts interacting with the actors.

[0005] Therefore, there is a need for a metaverse character implementation system and device that ensures that the body parts where actors and objects interact are aligned with the body parts where metaverse characters and objects interact in the metaverse virtual space.

[0006]

[0007] The object of the present invention is to provide a metaverse character implementation system that realistically implements the interaction between a metaverse character and an object by correcting a metaverse character implemented in a metaverse virtual space based on the movements of an actor interacting with an actual object.

[0008] Another object of the present invention is to provide a metaverse object implementation device that realistically implements a metaverse character and virtual object implemented in a metaverse virtual space based on the movement of an actor interacting with a real object and the movement of the real object.

[0009] The objectives of the present invention are not limited to those mentioned above. Other objectives and advantages of the present invention not mentioned above can be understood through the following description and will be more clearly understood through the embodiments of the present invention. Furthermore, it will be readily apparent that the objectives and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims.

[0010]

[0011] A metaverse character implementation system according to some embodiments of the present invention for solving the above problem includes a motion capture device that generates user capture data by tracking the movement of a user interacting with an object, an access detection device that generates access detection data by detecting whether the user approaches the object, and a metaverse object implementation device that generates metaverse character image data based on the user capture data and the access detection data.

[0012] In addition, the metaverse object implementation device includes a skeleton generation unit that generates skeleton data representing the skeleton and movement of a metaverse character using the user capture data, and the skeleton generation unit can generate corrected skeleton data by correcting the skeleton data based on the access detection data when the access detection data is input.

[0013] In addition, the skeleton generation unit can generate the corrected skeleton data by correcting at least one of the size of the metaverse character, the position and angle of a portion of the skeleton of the metaverse character.

[0014] In addition, the metaverse object implementation device may further include a retargeting unit that generates character rigging data by combining the appearance of the metaverse character with the skeletal data, and a metaverse character implementation module that generates the metaverse character image data by rendering the character rigging data.

[0015] In addition, the device may further include a photographing device that photographs the user and the object to generate photographed image data, and the device for implementing the metaverse object may further include an image synthesis module that synthesizes the photographed image data and the metaverse character image data to generate synthesized image data.

[0016] In addition, the metaverse object implementation device further includes a character characteristic processing unit that receives character setting data for the metaverse character and generates character characteristic data including numerical values ​​for the skeleton of the metaverse character, and the skeleton generation unit can generate the skeleton data using the character characteristic data.

[0017] In addition, the approach detection device includes a first detection device attached to the surface of the object, and when the distance between the first detection device and the first part of the user is less than or equal to a reference distance, the approach detection data can be generated.

[0018] In addition, the metaverse object implementation device generates corrected skeletal data that corrects the second part of the metaverse character corresponding to the first part of the user when the access detection data is generated, and can correct the positional relationship between the object and the second part so that it corresponds to the positional relationship between the object and the first part.

[0019] According to some embodiments of the present invention for solving the above problem, a metaverse object implementation device includes a skeleton generation unit that generates skeleton data representing the skeleton and movement of a metaverse object based on user capture data that tracks the movement of a user and object capture data that tracks the movement of an actual object, a retarget unit that generates character rigging data by combining the appearance of the metaverse object with the skeleton data, and a metaverse character implementation module that renders the character rigging data to generate metaverse object image data, wherein the skeleton generation unit generates corrected skeleton data by correcting the skeleton data based on approach detection data generated when the user and the actual object approach each other.

[0020] In addition, the skeleton generation unit may generate the corrected skeleton data by correcting at least one of the skeleton data of the metaverse character and the skeleton data of the virtual object so that the interaction between the user and the real object and the interaction between the metaverse character generated based on the user capture data and the virtual object generated based on the object capture data correspond to each other.

[0021]

[0022] The metaverse character implementation system and metaverse object implementation device of the present invention generate proximity detection data when an actor approaches an object, and can use this data to calibrate the metaverse character. This allows for more realistic interactions between metaverse characters and objects in the metaverse virtual space.

[0023] Additionally, the metaverse character implementation system and metaverse object implementation device of the present invention can implement virtual objects corresponding to real objects interacting with actors in the metaverse virtual space. This allows for more natural interactions between metaverse characters and objects in the metaverse virtual space.

[0024] In addition to the above-described contents, the specific effects of the present invention are described together with the specific matters for carrying out the invention below.

[0025]

[0026] FIG. 1 is a conceptual diagram illustrating a metaverse character implementation system according to some embodiments of the present invention.

[0027] Figure 2 is a block diagram for explaining in detail the image processing module of Figure 1.

[0028] Figures 3 and 4 are exemplary diagrams for explaining the approach detection device of Figure 1.

[0029] FIGS. 5 to 8 are exemplary diagrams for explaining a metaverse character implementation method performed in the metaverse object implementation device of FIG. 1.

[0030] Fig. 9 is an example diagram for explaining synthetic image data generated in the metaverse object implementation device of Fig. 1.

[0031] FIG. 10 is a flowchart illustrating a metaverse character implementation method performed in a metaverse object implementation device according to some embodiments of the present invention.

[0032] Figure 11 is a flowchart for explaining in detail the character rigging data generation step of Figure 10.

[0033] Figure 12 is a flowchart for explaining in detail the synthetic image data generation step of Figure 10.

[0034] FIG. 13 is a conceptual diagram illustrating a metaverse character implementation system according to some embodiments of the present invention.

[0035] Figure 14 is a block diagram for explaining in detail the image processing module of Figure 13.

[0036] Fig. 15 is an example diagram for explaining a metaverse character implementation method performed in the metaverse object implementation device of Fig. 13.

[0037] Fig. 16 is a flowchart for explaining a metaverse character implementation method performed in the metaverse object implementation device of Fig. 13.

[0038] Figure 17 is a flowchart for explaining in detail the character rigging data and virtual object data generation steps of Figure 16.

[0039] Figure 18 is a flowchart for explaining in detail the method for generating synthetic image data of Figure 16.

[0040]

[0041] The terms and words used in this specification and claims should not be interpreted based on their general or dictionary meanings. In accordance with the principle that inventors can define the concepts of terms and words to best describe their inventions, they should be interpreted in a way that is consistent with the technical concept of the present invention. Furthermore, the embodiments described in this specification and the configurations depicted in the drawings are merely examples of how the present invention can be realized and do not fully represent the technical concept of the present invention. Therefore, it should be understood that various equivalents, modifications, and applicable examples may exist as of the time of filing.

[0042] The terms first, second, A, B, etc. used in this specification and claims may be used to describe various components, but the components should not be limited by these terms. These terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term "and / or" includes any combination of a plurality of related listed items or any item among a plurality of related listed items.

[0043] The terminology used in this specification and claims is for the purpose of describing specific embodiments only and is not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. It should be understood that terms such as "comprise" or "have" in this application do not preclude the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification.

[0044] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0045] Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless expressly defined in this application.

[0046] In addition, each configuration, process, procedure or method included in each embodiment of the present invention may be shared within a scope that is not technically inconsistent with each other.

[0047]

[0048] Hereinafter, with reference to FIGS. 1 to 9, a metaverse character implementation system according to some embodiments of the present invention will be described.

[0049] FIG. 1 is a conceptual diagram illustrating a metaverse character implementation system according to some embodiments of the present invention.

[0050] Referring to FIG. 1, a metaverse character implementation system according to some embodiments of the present invention includes a motion capture device (100), an approach detection device (200), a photographing device (300), and a metaverse object implementation device (400).

[0051] A motion capture device (100) can generate user capture data (UC) by tracking the movements of a user interacting with an object. Specifically, the motion capture device (100) can detect the user's movements by attaching sensors and / or markers to the user's body surface, and generate the user capture data (UC). In other words, the user capture data (UC) can be data representing the movements of the user's skeleton.

[0052] For example, the motion capture device (100) may be implemented as an optical, magnetic, and / or inertial motion capture device. The optical motion capture device may include a marker and a camera. The optical motion capture device may be a device that captures the movement of a user who has a marker attached to him / her with one or more cameras, and generates user capture data (UC) by calculating the three-dimensional coordinates of the marker attached to the user through triangulation.

[0053] A magnetic motion capture device may be a device that generates user capture data (UC) by attaching a sensor capable of measuring a magnetic field to the user's joints, etc., and then calculating the amount of change in the magnetic field of each sensor near a magnetic field generating device to measure movement.

[0054] An inertial motion capture device may be a device that attaches an inertial sensor including an acceleration sensor, a gyro sensor, and a geomagnetic sensor to a user's joints, etc., to read the user's movement, rotation, and / or direction to generate user capture data (UC). However, the optical, magnetic, and inertial motion capture devices described above are merely examples of how a motion capture device (100) according to some embodiments of the present invention can be implemented, and the embodiments are not limited thereto.

[0055] For example, the motion capture device (100) may use image processing technology utilizing artificial intelligence, or may generate user capture data (UC) in parallel with an optical, magnetic, and / or inertial motion capture device. In the following description, for convenience of explanation, it is assumed that the motion capture device (100) is an optical motion capture device including a marker and a camera.

[0056] The proximity detection device (200) can detect whether a user approaches an object and generate proximity detection data (Acc). The proximity detection device (200) can be attached to the surface of the object. For example, the proximity detection device (200) can be in the form of a button, and can generate proximity detection data (Acc) when a user presses the button. Alternatively, the proximity detection device (200) can measure the distance from the object to the user without contact with the user, such as with an infrared sensor, an ultrasonic sensor, a LiDAR sensor, or a radar sensor, to generate proximity detection data (Acc). The proximity detection device (200) can further include an inertial sensor, and can generate proximity detection data (Acc) by detecting a change in the position of the object using the inertial sensor.

[0057] As another example, the proximity detection device (200) may include a first detection device attached to the surface of an object and a second detection device attached to the body surface of a user. In this case, the first detection device may include a transmitter that generates a signal, and the second detection device may include a receiver that receives a signal. Accordingly, when the second detection device receives a signal generated by the first detection device, proximity detection data (Acc) may be generated. In this case, the proximity detection data (Acc) may include not only whether the user approaches the object but also the distance between the object and the user. As another example, the first detection device may be attached to the body surface of a user, and the second detection device may be attached to the surface of the object.

[0058] The photographing device (300) can generate photographed image data (RVd) that captures the movements of a user interacting with an object and the movements of the object. That is, the photographing device (300) can capture the movements of the user to generate user capture data (UC) and generate photographed image data (RVd). In an embodiment, the photographing device (300) can generate photographed image data (RVd) to enable analysis of three-dimensional movements using an RGB-D camera (depth camera).

[0059] The metaverse object implementation device (400) can receive user capture data (UC) from a motion capture device (100) and access detection data (Acc) from an access detection device (200). In addition, the metaverse object implementation device (400) can receive shooting image data (RVd) from a shooting device and receive character setting data (CSet) for a metaverse character from the outside. At this time, the character setting data (CSet) can include information on at least one of the character's image, body structure, and body size.

[0060] The metaverse object implementation device (400) may include a communication module (410), an image processing module (430), a metaverse character implementation module (470), and an image synthesis module (490).

[0061] The communication module (410) can receive data from outside the metaverse object implementation device (400) and transmit the received data to each module within the metaverse object implementation device (400). Specifically, the communication module (410) can transmit user capture data (UC), access detection data (Acc), and character setting data (CSet) to the image processing module (430). In addition, the communication module (410) can transmit captured image data (RVd) to the image synthesis module (490).

[0062] The image processing module (430) can generate character rigging data (CRig) based on at least one of user capture data (UC), proximity detection data (Acc), and character setting data (CSet). Here, the character rigging data (CRig) is data that combines the user's movements with the image of the metaverse character, and can be corrected using the proximity detection data (Acc). A detailed description of the image processing module (430) will be described later with reference to FIG. 2.

[0063] The metaverse character implementation module (470) can receive character rigging data (CRig) from the image processing module (430). The metaverse character implementation module (470) can render the character rigging data (CRig) to generate character image data (CVd). In this case, the character image data (CVd) may be image data expressing the specific shape and movement of the metaverse character.

[0064] The image synthesis module (490) can receive captured image data (RVd) and character image data (CVd). The image synthesis module (490) can generate composite image data (SVd) by replacing the area where the user (User) was captured in the captured image data (RVd) with a metaverse character. At this time, the image synthesis module (490) can generate composite image data (SVd) by using a metaverse character whose rendering has been completed in the character image data (CVd).

[0065] Although not illustrated in the drawing, the metaverse implementation system may further include a database for storing data used to implement metaverse characters. The database may be located within the metaverse object implementation device (400), but the present invention is not limited thereto.

[0066]

[0067] Figure 2 is a block diagram for explaining in detail the image processing module of Figure 1.

[0068] Referring to FIG. 2, the image processing module (430) may include a character characteristic processing unit (431), a skeleton generation unit (433), and a retargeting unit (435).

[0069] The character characteristic processing unit (431) can receive character setting data (CSet). The character characteristic processing unit (431) can derive character image data (CImg) and character characteristic data (CC) for the metaverse character from the character setting data (CSet). Here, the character image data (CImg) can include a rough image of the metaverse character. For example, the character image data (CImg) can include an image of the frontal shape of the metaverse character. Alternatively, the character characteristic data (CC) can include numerical data for the skeleton of the metaverse character. For example, the character characteristic data (CC) can include at least one of the height of the metaverse character, the positions of bones and joints, the length of each bone, and the range of motion of each joint.

[0070] The skeleton generation unit (433) can receive user capture data (UC), approach detection data (Acc), and character characteristic data (CC). The skeleton generation unit (433) can determine the basic skeleton of the metaverse character based on the character characteristic data (CC). Next, the skeleton generation unit (433) can determine how the movement of the skeleton changes according to the movement of the user from the user capture data (UC), and apply this to the skeleton of the metaverse character to generate skeleton data (Skl). For example, if the user capture data (UC) shows that the user is crossing his arms, the skeleton generation unit (433) can generate skeleton data (Skl) that arranges the skeleton of the metaverse character so that the metaverse character has crossed arms.

[0071] The skeleton generation unit (433) may include an access data processing unit (433a). The access data processing unit (433a) may analyze access detection data (Acc). The skeleton generation unit (433) may generate corrected skeleton data by correcting the skeleton data (Skl) based on the access detection data (Acc) analyzed by the access data processing unit (433a). When corrected skeleton data is generated, the skeleton generation unit (433) may replace the corrected skeleton data with the skeleton data (Skl). The correction of the skeleton data (Skl) will be described later with reference to FIGS. 5 to 8.

[0072] The retarget unit (435) can receive skeleton data (Skl) from the skeleton generation unit (433) and character image data (CImg) from the character characteristic processing unit (431). At this time, if the skeleton generation unit (433) receives approach detection data (Acc), the skeleton data (Skl) received by the retarget unit (435) may be corrected skeleton data. On the other hand, if the skeleton generation unit (433) does not receive approach detection data (Acc), the skeleton data (Skl) received by the retarget unit (435) may be uncorrected skeleton data.

[0073] The retarget unit (435) can generate character rigging data (CRig) by combining skeletal data (Skl) and character image data (CImg). In other words, the retarget unit (435) can generate character rigging data (CRig) that moves the metaverse character to correspond to the skeletal data (Skl) by combining the skeletal arrangement and movement of the skeletal data (Skl) with the appearance of the metaverse character. Here, the character rigging data (CRig) may be data about a metaverse character with an appearance before being rendered moving according to the skeletal data (Skl).

[0074]

[0075] Figures 3 and 4 are exemplary diagrams for explaining the approach detection device of Figure 1.

[0076] Referring to FIG. 3, the marker (Mrk) can be attached to the body surface of the user. The marker (Mrk) can be attached mainly to both ends of the user's joints, muscles, and bones to easily detect the movement of the user's skeleton.

[0077] Meanwhile, the approach detection device (200) may be attached to the surface of an object (Obj) and / or the body surface of a user (User). For example, referring to FIG. <3A>, the approach detection device (200) may be attached to the surface of an object (Obj) and may not be attached to the body surface of a user (User). In this case, the approach detection device (200) may detect at least one of contact with a user (User), a distance from the user (User), and a change in the position of the object (Obj), thereby generating approach detection data (Acc).

[0078] As another example, referring to FIG. <3B>, the proximity detection device (200) may include a first detection device (200_a) attached to the surface of an object (Obj) and a second detection device (200_b) attached to the body surface of a user (User). In this case, the first detection device (200_a) may transmit a signal. When the distance between the object (Obj) and the user (User) is less than or equal to a reference distance, the second detection device (200_b) may receive the signal and generate proximity detection data (Acc). Additionally, the second detection device (200_b) may also derive the distance between the first detection device (200_a) and the second detection device (200_b) based on the intensity of the signal.

[0079] As another example, referring to FIG. <3C>, the proximity detection device (200) may include a first detection device (200_a) attached to the surface of an object (Obj), a second detection device (200_b1) attached to one side of a user's body, and a third detection device (200_b2) attached to the other side of the user's body. At this time, the third detection device (200_b2) may operate in the same manner as the second detection device (200_b1). FIG. <3C> illustrates a case where the second detection device (200_b1) is attached to the user's left wrist, and the third detection device (200_b2) is attached to the user's right wrist. However, this is an example, and the attachment locations and numbers of the second sensing device (200_b1) and the third sensing device (200_b2) may vary depending on the body part of the user (User) interacting with the object (Obj).

[0080]

[0081] Referring to FIG. 4, approach detection data (Acc) may be generated according to the distance between the first detection device (200_a) and the second detection device (200_b). As shown in FIG. <4A>, when the distance between the first detection device (200_a) and the second detection device (200_b) is greater than the reference distance (r1), the second detection device (200_b) may not detect the signal transmitted by the first detection device (200_a). In this case, approach detection data (Acc) may not be generated. Conversely, as shown in FIG. <4B>, when the distance between the first detection device (200_a) and the second detection device (200_b) is within the reference distance (r1), the second detection device (200_b) may detect the signal transmitted from the first detection device (200_a). In this case, approach detection data (Acc) may be generated.

[0082]

[0083] FIGS. 5 to 8 are exemplary diagrams for explaining a metaverse character implementation method performed in the metaverse object implementation device of FIG. 1.

[0084] FIG. 5 illustrates a case where the access detection device (200) is attached only to the surface of an object (Obj) and not to the body surface of a user (User). At this time, for convenience of explanation, the access detection device (200) is described as an example in which the access detection device (200) generates access detection data (Acc) when a user (User) presses a button of the access detection device (200).

[0085] Referring to FIG. <5A>, when a user (User) presses a button of an approach detection device (200), the approach detection device (200) can generate approach detection data (Acc). At this time, the metaverse object implementation device (400) can identify the positions of a first reference marker (Mrk1a) that interacts with the approach detection device (200), a second reference marker (Mrk2a) that is different from the first reference marker (Mrk1a), and a third reference marker (Mrk3a) among markers (Mrk) attached to the body surface of the user (User) from the user capture data (UC). The second reference marker (Mrk2a) and the third reference marker (Mrk3a) may be markers attached to a body part of the user (User) that does not directly interact with an object (Obj). In FIG. <5A>, since the user interacts by extending his left arm and picking up an object (Obj), the markers located at the ankle area, which are unrelated to the interaction with the object (Obj), are set as the second reference marker (Mrk2a) and the third reference marker (Mrk3a).

[0086] The metaverse object implementation device (400) can check the skeletal arrangement of the metaverse character corresponding to the first to third reference markers (Mrk1a to Mrk3a), and correct the skeletal arrangement of the metaverse character using the approach detection data (Acc). At this time, the first reference marker (Mrk1a) may correspond to the first corresponding marker (Mrk1a') of the metaverse character. Similarly, the second reference marker (Mrk2a) may correspond to the second corresponding marker (Mark2a') of the metaverse character, and the third reference marker (Mrk3a) may correspond to the third corresponding marker (Mrk3a') of the metaverse character. Here, the first to third corresponding markers (Mrk1a' to Mrk3a') indicate specific points of the skeleton of the metaverse character, and the first to third corresponding markers (Mrk1a' to Mrk3a') are not shown in the metaverse character image.

[0087] Specifically, the access data processing unit (433a) can derive a triangle having the first reference marker (Mrk1a), the second reference marker (Mrk2a), and the third reference marker (Mrk3a) as vertices. In other words, the access data processing unit (433a) can derive a first straight line (x1) passing through the first reference marker (Mrk1a) and the second reference marker (Mrk2a), a second straight line (x2) passing through the first reference marker (Mrk1a) and the third reference marker (Mrk3a), and a third straight line (x3) passing through the second reference marker (Mrk2a) and the third reference marker (Mrk3a). Subsequently, the access data processing unit (433a) can derive a first height (h1) which is a distance between the third reference marker (Mrk3a) and the third straight line (x3).

[0088] The skeleton generation unit (433) can derive a straight line passing through the second corresponding marker (Mrk2a') and the third corresponding marker (Mrk3a'), and derive a second height (h2) which is the distance from the first corresponding marker (Mrk1a') to the straight line. Then, the skeleton generation unit (433) can correct the skeleton arrangement of the metaverse character (Char) so that the first height (h1) and the second height (h2) are the same. That is, the skeleton generation unit (433) can change the position of the first corresponding marker (Mrk1a') to correct the skeleton data (Skl) so that the left hand of the metaverse character (Char) faces the microphone, thereby generating corrected skeleton data.

[0089] That is, in the conventional method of implementing a metaverse character (Char) so that the posture of the metaverse character (Char) is the same as that of the user (User), there was a limitation that the action of the user (User) picking up an object (Obj) was implemented as an action unrelated to the object (Obj) in the metaverse character (Char), as shown in FIG. <5B>. The present invention can appropriately implement the appearance of a metaverse character (Char) interacting with an object (Obj) by correcting skeletal data (Skl) by utilizing approach detection data, as shown in FIG. <5C>.

[0090]

[0091] Fig. 6 illustrates a case where the proximity detection device (200) is attached to the surface of an object (Obj) and the left arm of a user (User). That is, the first detection device (200_a) may be attached to the surface of the object (Obj), and the second detection device (200_b) may be attached to the left arm of the user (User).

[0092] Referring to FIG. <6A>, when a user (User) approaches an object (Obj) and a first distance (y1) between a first detection device (200_a) and a second detection device (200_b1) is within a reference distance (r1 of FIG. 4), approach detection data (Acc) may be generated. The approach detection data (Acc) generated at this time may include data regarding the first distance (y1). In addition, as in FIG. <6B>, approach detection data (Acc) may also be generated when a user (User) approaches closer to the object (Obj). The approach detection data (Acc) generated at this time may include data regarding the second distance (y2).

[0093] The metaverse object implementation device (400) can check the first device corresponding position (200_a') of the metaverse character image corresponding to the first detection device (200_a) and the second detection device (200_b) of the user (User) and the second device corresponding position (200_b') of the metaverse character (Char). The first device corresponding position (200_a') indicates a specific point of the metaverse character image, and the second device corresponding position (200_b') indicates a specific point on the skeleton of the metaverse character (Char), and is not shown in the metaverse character image.

[0094] The metaverse object implementation device (400) can check the skeletal arrangement of the metaverse character corresponding to the fourth and fifth reference markers (Mrk1b, Mrk2b) and correct the skeletal arrangement of the metaverse character using the approach detection data (Acc). At this time, the fourth reference marker (Mrk1b) may correspond to the fourth corresponding marker (Mrk1b') of the metaverse character. Similarly, the fifth reference marker (Mrk2b) may correspond to the fifth corresponding marker (Mark2b') of the metaverse character. Here, the fourth and fifth corresponding markers (Mrk1b', Mrk2b') indicate specific points of the skeleton of the metaverse character, and the fourth and fifth corresponding markers (Mrk1b', Mrk2b') are not shown in the metaverse character image.

[0095] Additionally, the second detection device (200_b) may be placed between the fourth detection marker (Mrk1b) located on the user's left wrist and the fifth detection marker (Mrk2b) located on the user's left elbow.

[0096] Specifically, the access data processing unit (433a) can derive the position of the first detection device (200_a) based on the first distance (y1) and the second distance (y2). For example, the access data processing unit (433a) can determine a first circle with a radius of the first distance (y1) centered around the second detection device (200_b) in FIG. <6A>, and can determine a second circle with a radius of the second distance (y2) centered around the second detection device (200_b) in FIG. <6B>. Then, the access data processing unit (433a) can derive the position of the first detection device (200_a) by deriving the point of contact between the first circle and the second circle. At this time, the first device corresponding position (200_a') can be set to the derived position of the first detection device (200_a).

[0097] The skeleton generation unit (433) can correct the skeleton arrangement of the metaverse character (Char) based on the derived position of the first detection device (200_a) so that the distance between the first detection device (200_a) and the second detection device (200_b), and the distance between the first device corresponding position (200_a') and the second device corresponding position (200_b') of the metaverse character (Char) correspond to each other. For example, when the distance between the first device corresponding position (200_a') and the second device corresponding position (200_b') is a third distance (y3) different from the second distance (y2), the skeleton generation unit (433) can correct the skeleton arrangement of the metaverse character (Char) so that the second distance (y2) and the fourth distance (y4) correspond to each other. At this time, the position of the metaverse character (Char) may be additionally corrected as necessary. Additionally, the first device corresponding position (200_a') is the same before and after correction, and only the position and / or skeletal arrangement of the metaverse character (Char) may be different.

[0098] Through this, the skeleton generation unit (433) can correct the skeleton arrangement of the metaverse character (Char) so that the left arm of the metaverse character (Char) faces the first device corresponding position (200_a'). Specifically, the skeleton generation unit (433) can correct the position of the marker located on the skeleton where the second detection device (200_b) is placed. That is, the skeleton generation unit (433) can move the positions of the fourth corresponding marker (Mrk1b') and the fifth corresponding marker (Mrk2b') to correct the skeleton data (Skl) so that the left hand of the metaverse character (Char) faces the microphone, thereby generating corrected skeleton data. The skeleton generation unit (433) can not only change the position of the marker to the corresponding marker, but also adjust the specific angle between the skeletons.

[0099] In the conventional method, when a metaverse character (Char) is implemented so that the posture of the metaverse character (Char) is the same as that of the user (User), there was a limitation that the action of the user (User) reaching out toward an object (Obj) was implemented as an action unrelated to the object (Obj) in the metaverse character (Char), as shown in FIG. <6C>. The present invention can implement an action of the metaverse character (Char) reaching out toward an object (Obj) even before the user (User) comes into contact with the object (Obj), as shown in FIG. <6D>, thereby expressing the interaction between the metaverse character (Char) and the object (Obj) more naturally.

[0100]

[0101] Fig. 7 illustrates a case where the proximity detection device (200) is attached to the surface of an object (Obj), the right arm of a user (User), and the left arm of a user (User). That is, the first detection device (200_a) can be attached to the surface of an object (Obj), the second detection device (200_b1) can be attached to the right arm of a user (User), and the third detection device (200_b2) can be attached to the left arm of a user (User).

[0102] The first detection device (200_a) of FIG. 7 can operate in the same manner as the first detection device (200_a) of FIG. 6. Similarly, the second detection device (200_b1) and the third detection device (200_b2) of FIG. 7 can operate in the same manner as the second detection device (200_b) of FIG. 6.

[0103] At this time, the second sensing device (200_b1) may be placed between the sixth sensing marker (Mrk1b) located on the user's right wrist and the seventh sensing marker (Mrk2b) located on the user's right elbow. Similarly, the third sensing device (200_b2) may be placed between the eighth sensing marker (Mrk3b) located on the user's left wrist and the ninth sensing marker (Mrk4b) located on the user's left elbow.

[0104] The metaverse object implementation device (400) can check the third device corresponding position (200_b2') of the metaverse character (Char) corresponding to the user's third sensing device (200_b2). The third device corresponding position (200_b2') indicates a specific point on the skeleton of the metaverse character (Char) and is not shown in the metaverse character image.

[0105] The metaverse object implementation device (400) can check the skeletal arrangement of the metaverse character corresponding to the sixth to ninth reference markers (Mrk1b to Mrk4b) and correct the skeletal arrangement of the metaverse character using the approach detection data (Acc). At this time, the sixth to ninth reference markers (Mrk1b to Mrk4b) may correspond to the sixth to ninth corresponding markers (Mrk1b' to Mrk4b') of the metaverse character, respectively. Here, the sixth to ninth corresponding markers (Mrk1b' to Mrk4b') indicate specific points of the skeleton of the metaverse character, and the sixth to ninth corresponding markers (Mrk1b' to Mrk4b') are not shown in the metaverse character image.

[0106] Referring to FIG. <7A>, a user (User) may extend his / her right arm toward an object (Obj). At this time, the distance between the first detection device (200_a) and the second detection device (200_b1) may be within a reference distance (r1 in FIG. 4). On the other hand, the distance between the first detection device (200_a) and the third detection device (200_b2) may be further than the reference distance (r1). Accordingly, the approach detection data (Acc) may include data on the approach of the second detection device (200_b1), but may not include data on the approach of the third detection device (200_b2).

[0107] In this case, in the same manner as in Fig. 6, the access data processing unit (433a) can derive the position of the first sensing device (200_a) using the access detection data (Acc). In addition, in the same manner as in Fig. 6, the skeleton generation unit (433) can correct the skeleton data (Skl) by changing the positions of the sixth corresponding marker (Mrk1b') and the seventh corresponding marker (Mrk2b') so that the right arm of the metaverse character (Char) faces the first sensing device (200_a).

[0108] On the other hand, since the approach detection data (Acc) does not include data on the approach of the third sensing device (200_b2), correction may not be performed on the left arm on which the third sensing device (200_b2) is placed. Accordingly, as shown in FIG. <7C>, the positions of the eighth corresponding marker (Mrk3b') and the ninth corresponding marker (Mrk4b') are maintained without correction, and the left arm of the metaverse character (Char) can be placed at the same angle as the left arm of the user (User).

[0109]

[0110] Fig. 8 illustrates a case where the proximity detection device (200) is attached to the surface of an object (Obj), the user's arms, and the center of the chin. The first to third detection devices (200_a, 200_b1, 200_b2) may be the same as those described in Fig. 7. The proximity detection device (200) may further include a fourth detection device (200_b3) attached to the center of the user's chin.

[0111] The metaverse object implementation device (400) can check the fourth device corresponding position (200_b3') of the metaverse character (Char) corresponding to the fourth sensing device (200_b3) of the user. The fourth device corresponding position (200_b3') indicates a specific point on the skeleton of the metaverse character (Char) and is not shown in the metaverse character image.

[0112] The metaverse object implementation device (400) can check the skeletal arrangement of the metaverse character corresponding to the 10th and 13th reference markers (Mrk1c to Mrk4c) and correct the skeletal arrangement of the metaverse character using the approach detection data (Acc). At this time, the 10th to 13th reference markers (Mrk1c to Mrk4c) may correspond to the 10th to 13th corresponding markers (Mrk1c' to Mrk4c') of the metaverse character, respectively. Here, the 10th to 13th corresponding markers (Mrk1c' to Mrk4c') indicate specific points of the skeleton of the metaverse character, and the 10th to 13th corresponding markers (Mrk1c' to Mrk4c') are not shown in the metaverse character image.

[0113] Referring to FIG. <8A>, a user (User) is holding an object (Obj) with both hands, and the distances between the first sensing device (200_a) and the second to fourth sensing devices (200_b1 to 200_b3) may all be reference distances (r1). Accordingly, the approach detection data (Acc) may include data on the approach of the second to fourth sensing devices (200_b1 to 200_b3).

[0114] Referring to FIG. <8B>, the positional relationship between the user (User) and the metaverse character (Char) with respect to the object (Obj) is expressed differently due to the size difference. That is, the user (User) is holding the object (Obj) with both hands, but the metaverse character (Char) may not be placed on top of both hands even though the user is holding the object (Obj) with both hands. Therefore, in such cases, it is necessary to correct the object (Obj) so that it is placed on top of the metaverse character (Char)'s both hands.

[0115] The access data processing unit (433a) can derive the position of the first detection device (200_a) using the access detection data (Acc). In addition, the access data processing unit (433a) can also derive the positional relationship between the first detection device (200_a) and the tenth to thirteenth reference markers (Mrk1c to Mrk4c). The access data processing unit (433a) can derive the position of the first detection device (200_a) and the positional relationship between the first detection device (200_a) and the tenth to thirteenth reference markers (Mrk1c to Mrk4c) using the method described above with reference to FIGS. 6 and 7. Subsequently, the access data processing unit (433a) can set the crop area (Bck) based on the object (Obj).

[0116] The skeleton generation unit (433) can correct the skeleton data (Skl) by enlarging the metaverse character (Char) according to the size of the crop area (Bck). At this time, enlarging the metaverse character (Char) is to maintain a constant size ratio between the object (Obj) and the metaverse character (Char) while making the object (Obj) appear enlarged according to the setting of the crop area (Bck).

[0117] Through this, as in FIG. <8C>, the metaverse character (Char) can be implemented as holding an object (Obj) with both hands. At this time, the positional relationship of the first device corresponding position (200_a') and the 10th to 13th corresponding markers (Mrk1c' to Mrk4c') can correspond to the positional relationship of the first sensing device (200_a) and the 10th to 13th reference markers (Mrk1c to Mrk4c) of FIG. <8A>. In addition, by implementing the metaverse character (Char) by enlarging it based on the crop area (Bck), the scene for the metaverse character (Char) can be converted from a full-body shot to a partial shot such as a close-up shot.

[0118] Although not specifically illustrated, the metaverse character implementation method disclosed in FIGS. 5 to 8 can also be performed in reverse order. In other words, when a user (User) moves away from an object (Obj), the skeleton generation unit (433) that was correcting the skeleton data (Skl) may no longer correct the skeleton data (Skl) as opposed to when the user (User) approaches the object (Obj). Accordingly, when a user (User) moves away from an object (Obj), the size, skeleton arrangement, and movement of the metaverse character (Char) can be implemented according to the movement of the user (User) without correction.

[0119]

[0120] Fig. 9 is an example diagram for explaining synthetic image data generated in the metaverse object implementation device of Fig. 1.

[0121] Referring to FIG. 9, the synthetic image data ultimately generated by the metaverse character implementation system of the present invention may embody a real-world background, a real-world character, a metaverse character, and objects interacting with the metaverse character in a single scene. In this case, the synthetic image data may implement a metaverse character corrected using proximity detection data.

[0122] Through this, the present invention can implement a more realistic metaverse character, such as when a real character hands over a microphone to a metaverse character, the metaverse character takes the microphone and conducts an interview.

[0123]

[0124] Hereinafter, with reference to FIGS. 10 to 12, a method for implementing a metaverse character performed in a metaverse object implementation device of the present invention will be described.

[0125] FIG. 10 is a flowchart for explaining a metaverse character implementation method performed in a metaverse object implementation device according to some embodiments of the present invention, and FIG. 11 is a flowchart for explaining in detail the character rigging data generation step of FIG. 10. FIG. 12 is a flowchart for explaining in detail the synthetic image data generation step of FIG. 10. Parts overlapping with the above contents are omitted or simplified.

[0126] Referring to FIG. 10, the metaverse object implementation device (400) receives user capture data (UC) from the motion capture device (100) (S100).

[0127] A motion capture device (100) can generate user capture data (UC) by tracking the movements of a user interacting with an object (Obj). For example, the motion capture device (100) can track the movements of a user by tracking a plurality of markers attached to the body surface of the user. The metaverse object implementation device (400) can receive user capture data (UC) in real time or non-real time.

[0128] Next, the metaverse object implementation device (400) receives access detection data (Acc) from the access detection device (200) (S200).

[0129] The approach detection device (200) can detect whether a user approaches an object (Obj) and generate approach detection data (Acc). The approach detection device (200) can be attached to the surface of the object (Obj) and / or the user (User), or can be built into the housing of the object (Obj). A plurality of approach detection devices (200) can be attached to the surface of the user (User), so that which part of the user's body approaches the object (Obj) can also be included in the approach detection data (Acc). The metaverse object implementation device (400) can receive the approach detection data (Acc) in real time or non-real time.

[0130] Additionally, the metaverse object implementation device (400) receives the shooting image data (RVd) from the shooting device (300) (S300).

[0131] The photographing device (300) can capture images of a user and an object (Obj) to generate captured image data (RVd). At this time, not only the user and the object (Obj), but also characters, backgrounds, props, etc. other than the user (User) can be captured and included in the captured image data (RVd). The metaverse object implementation device (400) can receive the captured image data (RVd) from the photographing device (300) in real time or non-real time.

[0132] Next, the metaverse object implementation device (400) generates character rigging data (CRig) for the metaverse character using at least one of character setting data (CSet), user capture data (UC), and access detection data (Acc) (S400).

[0133] In detail, referring to FIG. 11, the metaverse object implementation device (400) generates skeletal data using character characteristic data (CC) and user capture data (UC) (S410).

[0134] Specifically, referring to FIG. 2, the character characteristic processing unit (431) can derive character characteristic data (CC) and character image data (CImg) from character setting data (CSet). Here, the character characteristic data (CC) can include numerical values ​​for the skeleton of the metaverse character (Char). For example, the character characteristic data (CC) can include the length and thickness of each bone of the metaverse character, the range of motion of the joints, etc.

[0135] Meanwhile, the skeleton generation unit (433) can generate skeleton data (Skl) representing the skeleton and movement of the metaverse character (Char) using character characteristic data (CC) and user capture data (UC). At this time, the skeleton data (Skl) may be data that combines the posture and movement of the user (User) obtained from the user capture data (UC) with the basic skeleton of the metaverse character (Char) created based on the character characteristic data (CC).

[0136] Next, the metaverse object implementation device (400) determines whether access detection data (Acc) is received (S430). If access detection data (Acc) is received, the metaverse object implementation device (400) corrects the skeleton data (Skl) based on the access detection data (Acc) to generate corrected skeleton data (S450). Next, the metaverse object implementation device (400) generates character rigging data (CRig) based on the character image data (CImg) and the skeleton data (Skl) (S470).

[0137] When the approach detection data (Acc) is received, the approach data processing unit (433a) can derive the positional relationship between the object (Obj) and the user (User) using the approach detection data (Acc) and the user capture data (UC). At this time, if the positional relationship between the object (Obj) and the metaverse character (User) in the uncorrected skeleton data (Skl) is different from the derived positional relationship, the skeleton generation unit (433) can generate corrected skeleton data by correcting the skeleton data (Skl). Subsequently, the retarget unit (435) can generate character rigging data (CRig) based on the corrected skeleton data and the character image data (CImg). Here, the character rigging data (CRig) may be data in which the appearance, skeleton, and movement of the metaverse character are combined.

[0138] On the other hand, if access detection data (Acc) is not received, the metaverse object implementation device (400) performs step S470.

[0139] In this case, since the approach detection data (Acc) is not received, the skeleton generation unit (433) does not correct the skeleton data (Skl). Subsequently, the retarget unit (435) can generate character rigging data (CRig) based on the uncorrected skeleton data (Skl) and character image data (CImg).

[0140] Next, the metaverse object implementation device (400) generates synthetic image data based on the captured image data (RVd) and character rigging data (S500).

[0141] In detail, referring to FIG. 12, the metaverse object implementation device (400) renders character rigging data (CRig) to generate character image data (CVd) (S510).

[0142] Specifically, referring to FIG. 1, the metaverse character implementation module (470) can receive character rigging data (CRig). Subsequently, the metaverse character implementation module (470) can render the character rigging data (CRig) to generate character image data (CVd). Here, the character image data (CVd) may be image data that implements the specific appearance of the metaverse character to be displayed in the synthetic image data (SVd), such as the brightness of the metaverse character (Char) and the angle at which the metaverse character (Char) is implemented.

[0143] Next, the metaverse object implementation device (400) generates synthetic image data (SVd) using character image data (CVd) and shooting image data (RVd) (S530).

[0144] Specifically, referring to FIG. 1, the image synthesis module (490) can generate composite image data (SVd) by synthesizing the captured image data (RVd) and the character image data (CVd). At this time, the composite image data (SVd) may be final image data in which a metaverse character (Char) is placed in place of the user's (User) appearance location. A metaverse character (Char) corrected based on approach detection data (Acc) may be implemented in the composite image data (SVd).

[0145] Meanwhile, although step S300 is depicted as being performed after step S200 in FIG. 10, step S300 may be performed before step S100, before step S200, or before step S500.

[0146]

[0147] Hereinafter, with reference to FIGS. 13 to 15, a metaverse character implementation system according to some embodiments of the present invention will be described.

[0148] Figure 13 is a conceptual diagram illustrating a metaverse character implementation system according to some embodiments of the present invention. Any portions that overlap with the previously described content are omitted or simplified.

[0149] Referring to FIG. 13, the motion capture device (100) of the metaverse character implementation system according to some embodiments of the present invention can further generate object capture data (OC) that tracks the movement of an actual object. In this case, the actual object may be the aforementioned object (Obj).

[0150] The image processing module (430) can generate virtual object data (VOD) using object capture data (OC). Here, the virtual object data (VOD) is generated based on the movement of an actual object tracked in the object capture data (OC), and may be data on at least one of the size, approximate shape, and movement of a virtual object implemented to interact with a metaverse character (Char).

[0151] Additionally, the image processing module (430) can further utilize object capture data (OC) to generate character rigging data (CRig). This will be described later with reference to FIG. 14.

[0152] The metaverse character implementation module (470) can render virtual object data (VOD) to generate object image data (OVd). The object image data (OVd) may be image data expressing the specific shape and movement of a virtual object.

[0153] The image synthesis module (490) can generate composite image data (SVd) by synthesizing the captured image data (RVd), the character image data (CVd), and the object image data (OVd). At this time, the image synthesis module (490) can generate composite image data (SVd) by replacing the area where the user (User) is captured in the captured image data (RVd) with a metaverse character (Char) and replacing the area where the actual object is captured with a virtual object. At this time, the image synthesis module (490) can generate composite image data (SVd) by using the metaverse character (Char) whose rendering has been completed in the character image data (CVd) and the virtual object whose rendering has been completed in the object image data (OVd).

[0154]

[0155] Figure 14 is a block diagram for explaining in detail the image processing module of Figure 13. Parts that overlap with the above-described content are omitted or simplified.

[0156] Referring to FIG. 14, the access data processing unit (433a) can analyze the access detection data (Acc) to derive a positional relationship between an actual object and a user (User). The skeleton generation unit (433) can generate corrected skeleton data by correcting the skeleton data (Skl) based on the derived positional relationship. If corrected skeleton data is generated, the skeleton generation unit (433) can replace the corrected skeleton data with the skeleton data (Skl).

[0157] When proximity detection data (Acc) is received, the skeleton generation unit (433) can analyze object capture data (OC) to generate virtual object data (VOD). The skeleton generation unit (433) can generate virtual object data (VOD) by determining at least one of the size, approximate shape, and movement of a virtual object implemented to interact with a metaverse character (Char).

[0158] The retargeting unit (435) can generate object image data (OVd) by combining the approximate appearance of a virtual object with virtual object data (VOD). At this time, the virtual object in the object image data (OVd) may have a low-quality appearance, and detailed elements such as brightness and darkness expression may not be implemented.

[0159]

[0160] Fig. 15 is an example diagram for explaining a metaverse character implementation method performed in the metaverse object implementation device of Fig. 13.

[0161] Fig. 15 illustrates a case where the proximity detection device (200) is attached to the surface of an actual object (Obj) and the center of the user's chin. This may be the same as described in Fig. 8.

[0162] In addition, the motion capture device (100) can generate object capture data (OC) by tracking an object marker (Mrk_obj) attached to the surface of a real object (Obj). Although only one object marker (Mrk_obj) is illustrated in FIG. 15, a plurality of object markers (Mrk_obj) may be attached to the surface of the real object. The object marker (Mrk_obj) of the real object (Obj) may correspond to the object corresponding marker (Mrk_obj') of the virtual object (VO). The metaverse object implementation device (400) can generate synthetic image data in which a virtual object (VO) is implemented at the location of the object corresponding marker (Mrk_obj').

[0163] The metaverse object implementation device (400) can check the skeletal arrangement of the metaverse character corresponding to the 14th and 17th reference markers (Mrk1d to Mrk4d) and correct the skeletal arrangement of the metaverse character using the approach detection data (Acc). At this time, the 14th to 17th reference markers (Mrk1d to Mrk4d) may correspond to the 14th to 17th corresponding markers (Mrk1d' to Mrk4d') of the metaverse character, respectively. Here, the 14th to 17th corresponding markers (Mrk1d' to Mrk4d') indicate specific points of the skeleton of the metaverse character, and the 14th to 17th corresponding markers (Mrk1d' to Mrk4d') are not shown in the metaverse character image.

[0164] Figure <15A> shows the same user (User) posture as Figure <8A>, and Figure <15B> shows the same interaction between the actual object (Obj) and the metaverse character (Char) as Figure <8B>, but the interaction is not properly implemented.

[0165] The access data processing unit (433a) can derive the positional relationship between the object marker (Mrk_obj) and the 14th to 17th reference markers (Mrk1d to Mrk4d). In addition, the access data processing unit (433a) can derive the positional relationship between the object marker (Mrk_obj) and the 14th to 17th corresponding markers (Mrk1d' to Mrk4d').

[0166] When the positional relationship between the object marker (Mrk_obj) and the 14th to 17th reference markers (Mrk1d to Mrk4d) is different from the positional relationship between the object marker (Mrk_obj) and the 14th to 17th corresponding markers (Mrk1d' to Mrk4d'), the skeleton generation unit (433) can use the object corresponding marker (Mrk_obj'). Specifically, the skeleton generation unit (433) can generate object motion data (OM) by arranging the object corresponding marker (Mrk_obj') so that the positional relationship between the object marker (Mrk_obj) and the 14th to 17th corresponding markers (Mrk1d' to Mrk4d') corresponds to the positional relationship between the object corresponding marker (Mrk_obj') and the 14th to 17th reference markers (Mrk1d to Mrk4d).

[0167] Object motion data (OM) may be data that tracks the movement of an actual object (Obj) and the positional relationship between the object marker (Mrk_obj) and the 14th to 17th reference markers (Mrk1d to Mrk4d) while approach detection data (Acc) is generated in object capture data (OC), and corrects the object capture data (OC) so that the movement and positional relationship correspond to the positional relationship between the object corresponding marker (Mrk_obj') and the 14th to 17th corresponding markers (Mrk1d' to Mrk4d'). In this case, when a plurality of object markers (Mrk_obj) are attached to the surface of the object (Obj), the object motion data (OM) may be skeletal data for a virtual object (VO).

[0168] For example, in the case of an object (Obj) that can bend around a hinge, an object marker (Mrk_obj) can be attached to the hinge, one side of the hinge, and the other side of the hinge. In this case, in the same way that the skeletal data (Skl) is generated by identifying the movement of the user's skeleton from the user capture data (UC), the skeleton generation unit (433) can identify the movement of the object (Obj) and generate object motion data (OM) as skeletal data for the object (Obj). In addition, the skeletal generation unit (433) can correct the object motion data (OM) generated as skeletal data for the object (Obj). Furthermore, the skeletal generation unit (433) can additionally use the approach detection data (Acc) and the object motion data (OM) to correct the skeletal data (Skl) so as to satisfy a predefined condition between the virtual object (VO) and the metaverse character (Char), thereby generating corrected skeletal data. For example, when a metaverse character (Char) sings while holding a microphone, a condition may be set so that the distance between a virtual object (VO) in the shape of a microphone and the mouth of the metaverse character (Char) is within a certain distance (e.g., 15 cm). In this case, the skeleton generation unit (433) may adjust the positions of the 16th and 17th corresponding markers (Mrk3d', Mrk4d') to satisfy the set condition.

[0169] As another example, if a user holds both ends of an object (Obj) with a hinge, the skeleton generation unit (433) can correct the angle of the hinge so that both ends of the virtual object (Obj) can be positioned on both hands of the metaverse character (Char). That is, the skeleton generation unit (433) can correct the object motion data (OM) generated as skeletal data for the object (Obj). In this case, the corrected object motion data (OM) may be referred to as corrected object skeletal data, but the present embodiment is not limited to this term.

[0170] Through this, as in FIG. <15C>, the present embodiment can implement a metaverse character (Char) that interacts with a virtual object (VO) without a scene transition. In particular, by adjusting the skeleton of the metaverse character (Char) according to setting detailed conditions, the metaverse character (Char) that interacts with a virtual object (VO) can be implemented more realistically.

[0171]

[0172] Hereinafter, with reference to FIGS. 16 to 18, a metaverse character implementation method performed in a metaverse object implementation device of the present invention will be described.

[0173] Fig. 16 is a flowchart for explaining a metaverse character implementation method performed in the metaverse object implementation device of Fig. 13, and Fig. 17 is a flowchart for explaining in detail the character rigging data and virtual object data generation steps of Fig. 16. Fig. 18 is a flowchart for explaining in detail the synthetic image data generation method of Fig. 16. Parts overlapping with the above contents are omitted or simplified.

[0174] Referring to FIG. 16, the metaverse object implementation device (400) receives user capture data (UC) and object capture data (OC) from the motion capture device (100) (S100').

[0175] The motion capture device (100) can generate user capture data (UC) that tracks the movement of a user and object capture data (OC) that tracks the movement of an actual object (Obj). The motion capture device (100) can track the movement of a user by tracking a plurality of markers attached to the body surface of the user. In the same way, the motion capture device (100) can track the movement of an actual object (Obj) by tracking markers attached to the surface of the actual object (Obj). The metaverse object implementation device (400) can receive the user capture data (UC) and object capture data (OC) in real time or non-real time.

[0176] Next, the metaverse object implementation device (400) receives access detection data (Acc) from the access detection device (200) (S200).

[0177] Next, the metaverse object implementation device (400) receives the shooting image data (RVd) from the shooting device (300) (S300).

[0178] Next, the metaverse object implementation device (400) uses at least one of character setting data (CSet), user capture data (UC), object capture data (OC), and access detection data (Acc) to generate at least one of character rigging data (CRig) for a metaverse character (Char) and virtual object data (VOD) for a virtual object (VO) (S400').

[0179] In detail, referring to FIG. 17, the metaverse object implementation device (400) generates skeletal data (Skl) using character characteristic data (CC) and user capture data (UC) (S410).

[0180] Next, the metaverse object implementation device (400) determines whether access detection data (Acc) and object capture data (OC) have been received (S430'). If the access detection data (Acc) and object capture data (OC) have been received, the metaverse object implementation device (400) generates object motion data (OM) for the object based on the object capture data (OC) (S440').

[0181] The skeleton generation unit (433) can generate object motion data (OM) for object capture data (OC) at the time when approach detection data (Acc) is generated. The object motion data (OM) may be a combination of an object corresponding marker (Mrk_obj') and object capture data (OC) at the time when approach detection data (Acc) is generated. In other words, the object motion data (OM) may be data about a virtual object (VO) moving from the position of an object corresponding marker (Mrk_obj') according to the movement of a tracked real object (Obj). The skeleton generation unit (433) may not perform analysis on object capture data (OC) at the time when approach detection data (Acc) is not generated. This is because correction of a virtual object (VO) or a metaverse character (Char) is unnecessary when the real object (Obj) does not interact with a user (User).

[0182] In addition, the metaverse object implementation device (400) generates corrected skeletal data by correcting skeletal data (Skl) based on approach detection data (Acc) and object motion data (OM) (S450').

[0183] First, the skeleton generation unit (433) can generate skeleton data (Skl) based on user capture data (UC). Next, the skeleton generation unit (433) can correct the skeleton data (Skl) based on approach detection data (Acc) and object motion data (OM). At this time, the skeleton generation unit (433) can correct the skeleton data (Skl) so that the metaverse character (Char) and the virtual object (VO) satisfy specific conditions to generate corrected skeleton data.

[0184] For example, when a metaverse character (Char) is holding a virtual object (VO) in its hand, the skeleton generation unit (433) can correct the position of the hand of the metaverse character (Char) so that the virtual object (VO) does not cover the face of the metaverse character (Char).

[0185] Next, the metaverse object implementation device (400) generates character rigging data (CRig) and virtual object data (VOD) based on at least one of character image data (CImg), skeleton data (Skl), and object motion data (OM) (S470').

[0186] The retarget unit (435) can generate character rigging data (CRig) by combining character image data (CImg) and correction skeleton data. In addition, the retarget unit (435) can generate virtual object data (VOD) by combining the approximate appearance of a virtual object with object motion data (OM).

[0187] On the other hand, if the approach detection data and object capture data are not received, the metaverse object implementation device (400) performs step S470'.

[0188]

[0189] *If proximity detection data (Acc) is not received or object capture data (OC) is not received, the skeleton generation unit (433) may not generate object motion data (OM) and may not correct the skeleton data (Skl). In this case, the retarget unit (435) may generate character rigging data (CRig) using the uncorrected skeleton data (Skl), but may not generate virtual object data (VOD).

[0190] Next, the metaverse object implementation device (400) generates synthetic image data (SVd) based on the shooting image data (RVd), character rigging data (CRig), and virtual object data (VOD) (S500').

[0191] In detail, referring to FIG. 18, the metaverse object implementation device (400) renders character rigging data (CRig) to generate character image data (CVd) (S510). Subsequently, the metaverse object implementation device (400) renders virtual object data (VOD) to generate object image data (OVd) (S520).

[0192] The metaverse character implementation module (470) can generate character image data (CVd) by rendering character rigging data (CRig). In addition, the metaverse character implementation module (470) can generate object image data (OVd) by rendering virtual object data (VOD). At this time, the character image data (CVd) and object image data (OVd) may be image data in which the specific appearance of the metaverse character (Char) and virtual object (VO) to be expressed in the synthetic image data (SVd) is implemented, such as the brightness and shade, the angle at which the metaverse character (Char) and virtual object (VO) are implemented, etc. are implemented through rendering.

[0193] Next, the metaverse object implementation device (400) generates synthetic image data (SVd) using at least one of the shooting image data (RVd), character image data (CVd), and object image data (OVd) (S530').

[0194] The image synthesis module (490) can generate composite image data (SVd) by synthesizing at least one of the captured image data (RVd) and the character image data (CVd) and the object image data (OVd). When the approach detection data (Acc) is not received, the image synthesis module (490) can generate composite image data (SVd) by synthesizing only the captured image data (RVd) and the character image data (CVd). When the approach detection data (Acc) is received, the image synthesis module (490) can generate composite image data (SVd) by synthesizing the captured image data (RVd), the character image data (CVd) and the object image data (OVd).

[0195] Meanwhile, although step S300 is depicted as being performed after step S200 in FIG. 16, step S300 may be performed before step S100', before step S200, or before step S500'.

[0196]

[0197] The metaverse character implementation system according to the present invention can detect a user's approach to an object and adjust the metaverse character accordingly. This allows for the same interaction between objects and metaverse characters in the metaverse virtual space.

[0198] Furthermore, the present invention can implement virtual objects corresponding to real-world objects that interact with users in the metaverse virtual space. This allows for more natural interactions between metaverse characters and objects in the metaverse virtual space.

[0199]

[0200] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

Claims

1. A motion capture device that generates user capture data by tracking the movements of a user interacting with an object; An access detection device that detects whether the user approaches the object and generates access detection data; A metaverse object implementation device that generates metaverse character image data based on the user capture data and the access detection data. Metaverse character implementation system.

2. In paragraph 1, The above metaverse object implementation device includes a skeleton generation unit that generates skeletal data representing the skeleton and movement of a metaverse character using the user capture data, The above skeleton generation unit generates corrected skeleton data by correcting the skeleton data based on the access detection data when the access detection data is input. Metaverse character implementation system.

3. In paragraph 2, The above skeleton generation unit generates the corrected skeleton data by correcting at least one of the size of the metaverse character, the position and angle of a part of the skeleton of the metaverse character. Metaverse character implementation system.

4. In paragraph 2, The above metaverse object implementation device, A retargeting unit that combines the appearance of the metaverse character with the skeletal data to generate character rigging data, Further comprising a metaverse character implementation module that renders the above character rigging data to generate the metaverse character image data. Metaverse character implementation system.

5. In paragraph 4, Further comprising a photographing device for photographing the user and the object to generate photographed image data, The above metaverse object implementation device further includes an image synthesis module that synthesizes the captured image data and the metaverse character image data to generate synthetic image data. Metaverse character implementation system.

6. In paragraph 2, The above metaverse object implementation device further includes a character characteristic processing unit that receives character setting data for the metaverse character and generates character characteristic data including numerical values ​​for the skeleton of the metaverse character. The above skeleton generation unit generates the skeleton data using the character characteristic data. Metaverse character implementation system.

7. In paragraph 1, The above approach detection device, comprising a first sensing device attached to the surface of the object; When the distance between the first detection device and the first part of the user is less than or equal to a reference distance, the approach detection data is generated. Metaverse character implementation system.

8. In paragraph 7, The above metaverse object implementation device, When the above approach detection data is generated, corrected skeletal data is generated by correcting the second part of the metaverse character corresponding to the first part of the user, and the positional relationship between the object and the second part is corrected to correspond to the positional relationship between the object and the first part. Metaverse character implementation system.

9. A skeleton generation unit that generates skeleton data representing the skeleton and movement of a metaverse object based on user capture data that tracks the user's movement and object capture data that tracks the movement of an actual object; A retargeting unit that combines the appearance of the metaverse object with the skeletal data to generate character rigging data; and Includes a metaverse character implementation module that renders the above character rigging data to generate metaverse object image data, The above skeleton generation unit generates corrected skeleton data by correcting the skeleton data based on approach detection data generated when the user and the actual object approach each other. Metaverse object implementation device.

10. In paragraph 9, The skeleton generation unit generates the corrected skeleton data by correcting at least one of the skeleton data of the metaverse character and the skeleton data of the virtual object so that the interaction between the user and the real object and the interaction between the metaverse character generated based on the user capture data and the virtual object generated based on the object capture data correspond to each other. Metaverse object implementation device.

Images