Method and system for displaying three-dimensional object in metaverse environment

Abstract

A method and system for displaying 3D objects in a metaverse environment are disclosed. The system includes a camera, a 2D image processing unit, a 2D-3D image conversion unit, and a display. The 2D image processing unit obtains spatial information, current object information, and next object information of a 2D object from background regions and object regions within the first images. The 2D-3D image conversion unit classifies the 2D object into either a person object or an object object based on the current object information, and sets the surface precision for 3D modeling for the person object and the object object differently as first and second reference values ​​according to the classification result. The first reference value is lower than the first reference value and has a difference of more than a certain level from the first reference value. The 2D-3D image conversion unit performs 3D modeling for each of the person object and the object object according to the reference values ​​and next object information set differently, and then displays the result.

Description

Technology Field

[0001] The present disclosure relates to an apparatus and system for displaying 3D objects in a metaverse environment. Background Technology

[0002] The metaverse refers to a virtual world designed to allow tasks and activities to take place just like in the real world. With recent technological advancements, various metaverse virtual spaces are emerging that serve as stages for remote work, business activities, shopping, and film production.

[0003] In the virtual space of the metaverse, offline activities that are physically limited can be integrated into the online environment, thereby improving convenience and work efficiency. For example, a user can efficiently perform a specific task, such as automotive design, by utilizing a virtual space specialized for that task. Consumers can easily try on various clothing items from online shopping malls using their avatars in a virtual fitting space. Video creators can gauge the overall composition and background of a video by preparing various tools, props, cast members, and filming locations offline and placing them in a virtual filming space in advance. The technology forming the background of this disclosure is disclosed in the following documents: 1) Cited Invention 1: Registered Patent Publication No. 10-2615263 (December 19, 2023) 2) Cited Invention 2: Registered Patent Publication No. 10-1717896 (March 21, 2017) The problem to be solved

[0004] In a metaverse environment, when converting 2D real-world images into high-quality 3D images and using the converted 3D images to create a virtual space suitable for a specific industry, field, specific device, or real-time situation, resource consumption increases as the quality of the 3D images used improves, leading to a decrease in work efficiency.

[0005] In real-time video production environments or multi-user simultaneous access environments, if high-quality 3D images are always used to realistically or naturally implement 3D objects within the metaverse virtual space, the level of resource consumption may increase.

[0006] In addition, if the quality of 3D images used to implement the metaverse virtual space is uniformly lowered to save resources, the sense of realism is reduced.

[0007] The present disclosure aims to provide embodiments that improve work efficiency while maintaining realism in providing 3D objects to be included in a metaverse virtual space. means of solving the problem

[0008] According to one aspect of the present disclosure, a method for displaying a 3D object in a metaverse environment comprises: capturing a plurality of first images of a real space through a camera; extracting object regions and background regions from the plurality of first images; obtaining spatial information from the background regions; obtaining current object information of a 2D object from the object regions, wherein the current object information includes information regarding a basic shape, texture, and current pose; generating next object information of the 2D object based on the current object information and the spatial information, wherein the next object information includes information regarding the basic shape, texture, and a next pose predicted to be continuous with the current pose within the real space; classifying the 2D object into one of a person object and an object object based on the current object information; if the 2D object is classified as the person object, setting a surface precision for 3D modeling as a first reference value and converting the person object into a 3D person object based on the first reference value and the next object information; if the 2D object is classified as the object object, setting the surface precision as the first It includes the operation of setting a second reference value that is lower than the reference value and has a difference of at least a certain level from the first reference value, converting the object object into a 3D object object based on the second reference value and the next object information, and the operation of displaying a second image containing the 3D person object or the 3D object object on a display.

[0009] According to one aspect of the present disclosure, a system for displaying a 3D object in a metaverse environment comprises: a camera that captures a plurality of first images of a real space; a 2D image preprocessing unit that extracts object regions and background regions from the plurality of first images, obtains spatial information from the background regions, and obtains current object information of a 2D object from the object regions, wherein the current object information includes information regarding a basic shape, a texture, and a current pose; a 2D image postprocessing unit that generates next object information of the 2D object based on the current object information and the spatial information, wherein the next object information includes information regarding the basic shape, the texture, and a next pose predicted to be continuous with the current pose within the real space; a system that classifies the 2D object into one of a person object and an object object based on the current object information, and if the 2D object is classified as the person object, sets a surface precision for 3D modeling as a first reference value, converts the person object into a 3D person object based on the first reference value and the next object information, and if the 2D object is classified as the object object, the It includes a 2D-3D image conversion unit that sets the surface precision to a second reference value that is lower than the first reference value and has a difference of more than a certain level from the first reference value, and converts the object object into a 3D object object based on the second reference value and the next object information, and a display that receives and displays a second image containing the 3D person object or the 3D object object. Effects of the invention

[0010] According to the embodiments of the present disclosure, by dynamically applying a method of providing 3D objects to be included in the metaverse virtual space, work efficiency can be improved while maintaining realism. Brief explanation of the drawing

[0011] FIG. 1 is a block diagram illustrating a system for displaying 3D objects in a metaverse environment according to one embodiment. FIG. 2 is a flowchart illustrating a method for displaying 3D objects in a metaverse environment according to one embodiment. Figure 3 is a detailed flowchart of the operation of converting a person object into a 3D person object among the methods of Figure 2. Specific details for implementing the invention

[0012] Hereinafter, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily practice them. The present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0014] FIG. 1 is a block diagram illustrating a system for displaying 3D objects in a metaverse environment according to one embodiment.

[0015] Referring to FIG. 1, a system (100) according to one embodiment is for displaying 3D objects in a metaverse environment and may include a camera (110), a 2D (two-dimensional) image preprocessing unit (120), a 2D image postprocessing unit (130), a 2D-3D (three-dimensional) image conversion unit (140), and a display (150).

[0016] The system (100) may include at least one electronic device, such as a client device and / or server, such as a smartphone, a personal computer, or an XR (extended reality) device. The illustrated components of the system (100) may operate in conjunction with each other. Each of the components of the system (100) may be composed of hardware, software, and / or a combination thereof.

[0017] The camera (110) captures a plurality of first images of the actual space (offline space). The plurality of first images captured by the camera (110) are transmitted to a 2D image preprocessing unit (120). Each of the plurality of first images may be a 2D image.

[0018] The 2D image preprocessing unit (120) extracts object regions and background regions from a plurality of first images captured by the camera (110). For example, the 2D image preprocessing unit (120) can separate each of the plurality of first images into an object region containing a 2D object and a background region excluding said object region.

[0019] The 2D image preprocessing unit (120) obtains spatial information (e.g., indoor, outdoor, house, office, shopping mall, location, address, street name, building name, etc.) from background regions within the first images.

[0020] The 2D image preprocessing unit (120) obtains current object information of a 2D object from object regions within the first images. The current object information may include information about the basic shape, texture, and current pose.

[0021] For example, the 2D image preprocessing unit (120) may extract an outline using image analysis of an object area and / or feature points of a 2D object within the object area, compare the extracted outline with pre-registered polygons, and select the polygon with the highest similarity to the basic shape of the 2D object according to the comparison result. The 2D image preprocessing unit (120) may extract a pattern of a 2D object within the object area, compare the extracted pattern with pre-registered patterns, and select the pattern with the highest similarity to the texture of the 2D object according to the comparison result. The 2D image preprocessing unit (120) may identify the pose of a 2D object within the object area, compare the identified pose with pre-registered poses (e.g., stationary pose, moving pose, lying pose, sitting pose, standing pose, exercise pose, etc.), and select the pose with the highest similarity to the current pose of the 2D object according to the comparison result.

[0022] The 2D image preprocessing unit (120) transmits current object information and spatial information of a 2D object obtained from a plurality of captured first images to the 2D image postprocessing unit (130) and / or the 2D-3D image conversion unit (140).

[0023] The 2D image post-processing unit (130) generates next object information of the 2D object based on the current object information and spatial information of the 2D object transmitted from the 2D image pre-processing unit (120). The next object information may include the basic shape and texture of the 2D object, and information about the next pose predicted to be continuous with the current pose of the 2D object in actual space. The basic shape and texture to be included in the next object information of the 2D object may be the same as the basic shape and texture included in the current object information of the 2D object.

[0024] In one embodiment, a posture information database for predicting the next posture may be stored in advance in memory (not shown). The posture information database may include data (e.g., 2D images, polygons) for a plurality of predicted postures mapped to a specific combination of space and posture. For example, if the spatial information is indoors and the current posture is a stationary posture, the 2D image post-processing unit (130) can predict the next posture as a stationary posture identical to the current posture. If the spatial information is a gym and the current posture is a first movement posture (e.g., a rolling posture moving from a first position to a second position), the 2D image post-processing unit (130) can predict the next posture as a second movement posture (e.g., a rolling posture moving to a third position). If the spatial information is an outdoor park and the current posture is a first exercise posture (e.g., a first posture during walking exercise), the 2D image post-processing unit (130) can predict the next posture as a second exercise posture (e.g., a second posture during walking exercise).

[0025] The 2D image post-processing unit (130) generates next object information of the 2D object and then transmits it to the 2D-3D image conversion unit (140).

[0026] The 2D-3D image conversion unit (140) classifies the 2D object included within the object regions of the first images into one of a person object and an object object based on the current object information of the 2D object transmitted from the 2D image preprocessing unit (120).

[0027] For example, the 2D-3D image conversion unit (140) can determine whether the 2D object (or type of the 2D object) corresponds to a person object or an object object by using image analysis of object regions containing the same 2D object and / or the shape or texture of the 2D object extracted from the object regions.

[0028] The 2D-3D image conversion unit (140) sets the surface precision for 3D modeling to a first reference value when the 2D object is classified as a person object. The first reference value may be a value that is pre-specified or adjusted. The 2D-3D image conversion unit (140) converts the person object into a 3D person object by performing 3D modeling (e.g., rendering) based on the first reference value and the next object information received from the 2D image post-processing unit (130).

[0029] When a 2D object is classified as a physical object, the 2D-3D image conversion unit (140) sets the surface precision for 3D modeling to a predetermined second reference value. The second reference value is lower than the first reference value and has a difference of at least a certain level from the first reference value. The 2D-3D image conversion unit (140) converts the physical object into a 3D physical object by performing 3D modeling (e.g., rendering) based on the second reference value and the next object information received from the 2D image post-processing unit (130).

[0030] For example, the reference values ​​for specifying surface precision for 3D modeling (e.g., rendering) may be values ​​for the number of meshes, polygons, or density. As the values ​​for the number of meshes, polygons, or density increase, the surface of the 3D object is represented more precisely, but the object creation speed may become relatively slower or the object creation time may increase relatively.

[0031] The 2D-3D image conversion unit (140) generates a second image containing a 3D person object or a 3D object object converted from a 2D object. The second image may be a 3D image.

[0032] For example, the 2D-3D image conversion unit (140) can generate a new 3D image by synthesizing a 3D person object or a 3D object object from a 2D object onto a spatial area of ​​a previous 3D image being displayed on a display (150).

[0033] As another example, the 2D-3D image conversion unit (140) can generate a new 3D image by 3D modeling background areas extracted from first images taken according to a third reference value that is lower than a first reference value and a second reference value, and by compositing a 3D person object or 3D object object converted from a 2D object onto the generated spatial area.

[0035] As described above, the system (100) sets the surface precision for 3D modeling differently depending on whether the 2D object (or type of 2D object) is classified as a person object or an object object.

[0036] For example, when a system (100) is classified as a person object, it can maintain a certain level of object quality (e.g., resolution, sharpness) by setting the surface precision to a first reference value of a relatively high level and creating a 3D object by 2D-3D conversion according to the first reference value. When a 2D object is classified as an object, the system (100) can speed up object creation or reduce object creation time by setting the surface precision to a second reference value of a relatively low level and creating a 3D object by 2D-3D conversion according to the second reference value.

[0037] In this way, the system (100) can dynamically apply a method of creating 3D objects by adjusting surface precision according to the object type or object classification result, thereby ensuring that both the object quality and the object creation speed (or object creation time) satisfy the given target values.

[0038] In some embodiments, the classification criteria for 2D objects included within the object regions of the first captured images may be further subdivided.

[0039] For example, the 2D-3D image conversion unit (140) classifies the 2D object into one of the object objects, human face objects, human body objects, or human limb objects, and sets different surface precision for each object according to the classification result. For example, for the object object, the surface precision can be set to the lowest fourth reference value among multiple levels of reference values ​​within a specified range so that the object creation speed (or object creation time) is shortened. For the body of the human object, excluding the face and limbs, the surface precision can be set to a third reference value at a level one step higher than the fourth reference value. For the limbs of the human object, which have relatively more movement, the surface precision can be set to a second reference value at a level one step higher than the third reference value. For the face of the human object, which requires relatively delicate expression depending on emotions or movements, the surface precision can be set to a first reference value at a level one step higher than the second reference value so that the object image quality is improved.

[0041] FIG. 2 is a flowchart illustrating a method for displaying a 3D object in a metaverse environment according to one embodiment. The method illustrated in FIG. 2 may be performed by the system (100) of FIG. 1 or one or more components within the system (100).

[0042] Referring to FIG. 2, a method for displaying a 3D object in a metaverse environment according to one embodiment may include operations 210 to 290.

[0043] In operation 210, the system (100) captures a plurality of first images of the actual space through a camera (110).

[0044] In operation 220, the system (100) extracts object regions and background regions from a plurality of first images captured in operation 210.

[0045] In operation 230, the system (100) obtains spatial information from background regions extracted through operation 220.

[0046] In operation 240, the system (100) obtains current object information of a 2D object within the object regions extracted through operation 220. The current object information may include information about the basic shape, texture, and current pose.

[0047] In operation 250, the system (100) generates next object information of the 2D object based on the current object information obtained through operation 240 and the spatial information obtained through operation 230. The next object information may include the basic shape, the texture, and information about the next pose predicted to be continuous with the current pose in the actual space.

[0048] In operation 260, the system (100) classifies the 2D object into one of a person object and an object object based on the current object information of the 2D object obtained in operation 240.

[0049] In operation 270, if the 2D object is classified as a person object as a result of the classification in operation 260, the system (100) sets the surface precision for performing 3D modeling for the 2D object as a first reference value. The system (100) converts the 2D object classified as a person object into a 3D person object by performing 3D modeling based on the first reference value and the next object information of the 2D object.

[0050] When performing 3D modeling according to the first reference value, the object quality may have a first quality value (e.g., in dpi units), and the object creation speed (or object creation time) may have a first speed value (first time).

[0051] In operation 280, if the 2D object is classified as an object rather than a person object as a result of the classification in operation 260, the system (100) sets the surface precision for performing 3D modeling of the 2D object as a second reference value. The second reference value used for 3D modeling of the object object is lower than the first reference value used for 3D modeling of the person object and may have a difference of at least a certain level from the first reference value. The system (100) converts the 2D object classified as an object object into a 3D object object by performing 3D modeling based on the set second reference value and the next object information of the 2D object.

[0052] When 3D modeling is performed according to a second reference value lower than the first reference value, the object quality may have a second quality value (e.g., in dpi units) lower than the first quality value, and the object creation speed (or object creation time) may have a second speed value (or second time) faster than the first speed value (first time).

[0053] In operation 290, the system (100) generates a second image containing a 3D person object converted through operation 270 or a 3D object object converted through operation 280, and displays the second image on a display (150).

[0055] Figure 3 is a detailed flowchart of the operation of converting a person object into a 3D person object among the methods of Figure 2.

[0056] Referring to FIG. 3, the operation 270 of converting a person object illustrated in FIG. 2 into a 3D person object may include operations 271 to 276. Operations 271 to 276 may be performed by the system (100) of FIG. 1 or one or more components within the system (100) (e.g., a 2D-3D image conversion unit (140)).

[0057] In operation 271, the system (100) selects the lowest first level (low level) among a plurality of preset levels for surface precision for 3D modeling so that the object creation speed is set to the first speed.

[0058] In operation 272, the system (100) generates a 3D modeling object having a first level (low level) of surface precision according to a first speed set in operation 271. The system (100) can generate a 3D modeling object according to 3D modeling parameter values ​​mapped to the first level (e.g., first values ​​for mesh, number of polygons, or density).

[0059] In operation 273, the system (100) evaluates whether the quality (e.g., resolution, sharpness) of the 3D modeling object created through operation 272 is greater than or equal to a preset value.

[0060] If the quality of the 3D modeling object is greater than or equal to a predetermined value as a result of the evaluation of operation 273, that is, if it satisfies the specified quality evaluation criteria, the system (100) proceeds to operation 274 and sets the 3D modeling object as a 3D person object to be included in the second image.

[0061] If, as a result of the evaluation of operation 273, the quality of the 3D modeling object is less than a predetermined value, that is, if it does not satisfy the predetermined quality evaluation criteria, the system (100) proceeds to operation 275. In operation 275, the system (100) selects (upgrades) the surface precision for 3D modeling to a second level (high level) that is one step higher than the lowest first level (low level) among a plurality of predetermined levels, thereby improving the quality of the object and setting the object creation speed to a second speed that is slower than the first speed.

[0062] In operation 276, the system (100) regenerates a 3D modeling object having a second level (high level) of surface precision according to a second speed set in operation 275. The system (100) can regenerate a 3D modeling object according to 3D modeling parameter values ​​mapped to the second level (e.g., mesh, number of polygons, or second values ​​for density).

[0063] Among the multiple levels specifying surface precision, selecting a higher level can improve object quality. However, while object quality is improved, object creation speed may slow down or object creation time may increase.

[0064] The system (100) can reduce the object creation speed (or object creation time) of a 3D person object by initially selecting the lowest first level (low level) among a plurality of levels that specify surface precision. Subsequently, if the generated 3D person object does not satisfy the specified quality evaluation criteria, the system (100) can selectively adjust the level of surface precision to increase by one step and then regenerate the 3D person object so that the object quality is gradually improved. The adjustment (upward adjustment) of the level of surface precision can be repeated until the specified quality evaluation criteria are satisfied.

[0065] Accordingly, in the process of creating 3D human objects that require relatively more precise representation compared to objects, surface precision can be controlled to an appropriate level, and both object quality and object creation speed (or object creation time) can satisfy given target values.

[0067] Although the present disclosure has been illustrated and described above in relation to embodiments to exemplify the principles of the present disclosure, the present disclosure is not limited to the configuration and operation as illustrated and described, and those skilled in the art to which the present disclosure pertains will understand that various changes and modifications to the present disclosure are possible without departing from the spirit and scope of the appended claims.

Claims

Claim 1 A method for displaying a 3D object in a metaverse environment comprises: capturing a plurality of first images of a real space through a camera; extracting object regions and background regions from the plurality of first images; obtaining spatial information from the background regions; obtaining current object information of a 2D object from the object regions, wherein the current object information includes information regarding a basic shape, texture, and current pose; generating next object information of the 2D object based on the current object information and the spatial information, wherein the next object information includes information regarding the basic shape, texture, and a next pose predicted to be continuous with the current pose within the real space; classifying the 2D object into one of a person object or an object based on the current object information; if the 2D object is classified as the person object, setting a surface precision for 3D modeling as a first reference value, and converting the person object into a 3D person object by rendering and compositing parts of the person object based on the first reference value and the next object information; if the 2D object is classified as the object object, the surface precision is the The method comprises: setting a second reference value that is lower than a first reference value and has a difference of at least a certain level from the first reference value, wherein the second reference value corresponds to the lowest fourth level value among a plurality of level values ​​within a specified range, and converting the object object into a 3D object object based on the second reference value and the next object information; and displaying a second image containing the 3D person object or the 3D object object on a display, wherein when the 2D object is classified as the person object, the method comprises the operation of setting the surface precision for 3D modeling to a first reference value, and performing a partial object classification that classifies the person object into one of a face part object, a body part object, and an limb part object.A method comprising, in response to the result of the partial object classification, setting the surface precision for 3D modeling to different level values ​​for each partial object, wherein the surface precision is set to a value excluding the lowest fourth level value among the plurality of level values ​​within the specified range, and wherein, in response to the result of the partial object classification, the action of setting the surface precision for 3D modeling to different level values ​​for each partial object comprises: in response to the person object being classified as a body part object, setting the surface precision to a third level value one step higher than the lowest fourth level value among the plurality of level values; in response to the person object being classified as an limb part object, setting the surface precision to a second level value one step higher than the third level value among the plurality of level values; and in response to the person object being classified as a face part object, setting the surface precision to a first level value one step higher than the second level value among the plurality of level values. Claim 2 A method according to claim 1, wherein the operation of converting the person object into the 3D person object by rendering and compositing the person object in parts comprises: an operation of setting the object creation speed to a first speed in response to setting the surface precision to the third level value among the plurality of level values; an operation of creating a 3D modeling part object having the surface precision of the third level value according to the first speed; an operation of evaluating whether the quality of the 3D modeling part object is greater than or equal to a predetermined value; an operation of setting the 3D modeling part object as the 3D person part object if the quality of the 3D modeling part object is greater than or equal to the predetermined value; an operation of changing the surface precision to the second level value, which is one step higher than the third level value among the plurality of level values, and setting the object creation speed to a second speed slower than the first speed if the quality of the 3D modeling part object is less than the predetermined value; and an operation of regenerating a 3D modeling part object having the surface precision of the second level value according to the second speed. Claim 3 A system for displaying 3D objects in a metaverse environment comprises: a camera that captures a plurality of first images of a real space; a 2D image preprocessing unit that extracts object regions and background regions from the plurality of first images, obtains spatial information from the background regions, and obtains current object information of a 2D object from the object regions, wherein the current object information includes information regarding a basic shape, texture, and current pose; a 2D image postprocessing unit that generates next object information of the 2D object based on the current object information and the spatial information, wherein the next object information includes information regarding the basic shape, texture, and a next pose predicted to be continuous with the current pose within the real space; a 2D image postprocessing unit that classifies the 2D object into one of a person object and an object object based on the current object information, and if the 2D object is classified as the person object, sets a surface precision for 3D modeling as a first reference value, and converts the person object into a 3D person object by rendering and compositing parts based on the first reference value and the next object information, and if the 2D object is classified as the object object, the A 2D-3D image conversion unit that sets surface precision to a second reference value lower than the first reference value and having a difference of more than a certain level from the first reference value, wherein the second reference value corresponds to the lowest fourth level value among a plurality of level values ​​within a specified range, and converts the object object into a 3D object object based on the second reference value and the next object information;A system configured to include a display that receives and displays a second image containing the 3D person object or the 3D object object, wherein the 2D-3D image conversion unit performs partial object classification by classifying the person object into one of a face part object, a body part object, and an limb part object, and in response to the result of the partial object classification, sets the surface precision for 3D modeling for each partial object to different level values, wherein the surface precision is set to a value excluding the lowest fourth level value among the plurality of level values ​​within the specified range, in response to the person object being classified as a body part object, the surface precision is set to a third level value one step higher than the lowest fourth level value among the plurality of level values, in response to the person object being classified as an limb part object, the surface precision is set to a second level value one step higher than the third level value among the plurality of level values, and in response to the person object being classified as a face part object, the surface precision is set to a first level value one step higher than the second level value among the plurality of level values.

Images