Method, apparatus, and computer readable medium for low-cost 2.5D pixel style avatar real-time creation with 3D interaction

Abstract

A method for generating a 2.5D pixel-style avatar in real-time capable of three-dimensional interaction at low cost according to one embodiment of the present invention is a method performed on a computing device comprising one or more processors, comprising: (A) a step of acquiring a two-dimensional (2D) high-resolution head image of a human, a two-dimensional low-resolution body image, and three-dimensional (3D) pose data from an input video stream; and (B) a step of generating a pixel human including a 2D high-resolution head, a 2D low-resolution body, and a 3D arm based on the 2D high-resolution head image, the 2D low-resolution body image, and the 3D pose data.

Description

Technology Field

[0001] The present invention relates to a method, apparatus, and computer-readable medium for real-time generation of a 2.5D pixel-style avatar capable of three-dimensional interaction at low cost. Background Technology

[0002] When conducting virtual meetings, the 2D video streaming method allows for video processing and transmission at a relatively low cost, but it has the disadvantage that mutual interaction is impossible. The 3D virtual environment method allows for interaction between avatars, but it has the disadvantage of requiring high-cost equipment (Depth cameras or VR equipment).

[0003] As mentioned above, the character representation method based on 2D streaming technology during online meetings has the disadvantage that it is impossible to interact with virtual objects. The character representation method based on 3D characters allows interaction with virtual objects, but it has the disadvantage of requiring a long character generation time and additional high-cost sensors. Current technology cannot solve the problems of both methods simultaneously.

[0004] Therefore, in order to improve communication efficiency during online meetings, there is a need for a technology that combines the advantages of 2D streaming and 3D character generation methods, and enables the interaction of arms required for 3D meetings at a low cost. Prior art literature

[0005] (Patent Document 0001) KR 10-2022-0082382 A The problem to be solved

[0006] The problem that the present invention aims to solve is to propose a new pixel-style avatar creation method by combining the advantages of a 2D streaming method and a 3D character generation method to improve communication efficiency during online meetings, and thereby provide a method, device, and computer-readable medium for real-time creation of a 2.5D pixel-style avatar that enables 3D interaction and allows for arm interaction necessary for 3D meetings at low cost. means of solving the problem

[0007] A method for real-time generation of a 2.5D pixel-style avatar capable of three-dimensional interaction at low cost according to an embodiment of the present invention for solving the above problem is,

[0008] A method performed on a computing device comprising one or more processors,

[0009] (A) a step of acquiring a 2D high-resolution head image of a human, a 2D low-resolution body image, and 3D pose data from an input video stream; and

[0010] (B) Based on the 2D high-resolution head image, the 2D low-resolution body image, and the 3D pose data, the method may include the step of generating a pixel human including a 2D high-resolution head, a 2D low-resolution body, and a 3D arm.

[0011] In a method for generating a 2.5D pixel-style avatar in real-time capable of three-dimensional interaction at low cost according to one embodiment of the present invention, step (A) comprises:

[0012] (A1) A step of obtaining a matting video stream by removing the background from the video stream and extracting the human image using matting; and

[0013] (A2) The method may include the step of estimating the pose of the human from the video stream and obtaining 3D pose coordinates of the pose landmark of the human.

[0014] In addition, in a method for generating a 2.5D pixel-style avatar in real time capable of three-dimensional interaction at low cost according to one embodiment of the present invention, step (A) is,

[0015] (A3) A step of separating the head and body of the human from the matting video stream based on the 3D coordinates of the pose landmark of the human, thereby obtaining the 2D high-resolution head image and the 2D high-resolution body-only image; and

[0016] (A4) The method may further include the step of downsampling the 2D high-resolution body-only image to obtain the 2D low-resolution body image.

[0017] In addition, in a method for generating a 2.5D pixel-style avatar in real time capable of three-dimensional interaction at low cost according to one embodiment of the present invention, step (B) is,

[0018] (B1) A step of generating a 2D avatar head including a 2D high-resolution face based on the above 2D high-resolution head image;

[0019] (B2) A step of generating a 2D point cloud avatar body based on the above 2D low-resolution body image;

[0020] (B3) A step of generating the 3D arm based on the 3D pose coordinates; and

[0021] (B4) Based on the 3D pose coordinates, the method may include the step of combining the 2D avatar head, the 2D point cloud avatar body, and the 3D arm to create the pixel human having a 3D arm.

[0022] In addition, in a method for generating a 2.5D pixel-style avatar in real-time that enables 3D interaction at low cost according to one embodiment of the present invention, step (A2) may include the step of obtaining 3D coordinates of 33 landmarks of the human using a MediaPipe pose.

[0023] In addition, in a method for generating a 2.5D pixel-style avatar in real-time that enables three-dimensional interaction at low cost according to one embodiment of the present invention, step (A4) can obtain a 2D low-resolution body image by downsampling the 2D high-resolution body-only image using a Gaussian Pyramid technique.

[0024] In addition, the method for real-time generation of a 2.5D pixel-style avatar capable of three-dimensional interaction at low cost according to one embodiment of the present invention, after step (B2),

[0025] (B2-1) A step of detecting gray edges of the 2D point cloud avatar body using a Laplacian edge detection technique;

[0026] (B2-2) A step of subtracting the gray edges of the detected avatar body from the alpha channel of the 2D point cloud avatar body; and

[0027] (B2-3) The method may further include the step of removing the gray edges of the 2D point cloud avatar body using an alpha-channel histogram equalization technique on the alpha channel of the avatar body from which the gray edges have been subtracted.

[0028] In addition, in a method for generating a 2.5D pixel-style avatar in real-time capable of three-dimensional interaction at low cost according to one embodiment of the present invention, step (B3) is,

[0029] (B3-1) Based on the above 3D pose coordinates, a step of generating a skeleton of one-pixel-width between the start point and the ending point of each arm region including the upper arm, lower arm and hand;

[0030] (B3-2) A step of calculating an adaptive arm width based on the alpha values ​​of the left and right pixels for each pixel of the above skeleton; and

[0031] (B3-3) The method may include the step of generating the 3D arm by setting the z-coordinate value of each arm region pixel to a predetermined value.

[0032] In addition, in a method for generating a 2.5D pixel-style avatar in real-time that enables three-dimensional interaction at low cost according to one embodiment of the present invention, the adaptive arm width may increase by 1 when the alpha value of the left pixel of the skeleton is greater than 0 or when the alpha value of the right pixel of the skeleton is greater than 0.

[0033] A device for real-time generation of a 2.5D pixel-style avatar capable of three-dimensional interaction at low cost according to an embodiment of the present invention for solving the above problem is a device comprising one or more processors that perform a method for real-time generation of a 2.5D pixel-style avatar capable of three-dimensional interaction at low cost,

[0034] The above-mentioned method for real-time generation of 2.5D pixel-style avatars capable of 3D interaction at low cost is,

[0035] (A) a step of acquiring a 2D high-resolution head image of a human, a 2D low-resolution body image, and 3D pose data from an input video stream; and

[0036] (B) Based on the 2D high-resolution head image, the 2D low-resolution body image, and the 3D pose data, the method may include the step of generating a pixel human including a 2D high-resolution head, a 2D low-resolution body, and a 3D arm.

[0037] In a 2.5D pixel-style avatar real-time generation device capable of three-dimensional interaction at low cost according to one embodiment of the present invention, step (A) is,

[0038] (A1) A step of obtaining a matting video stream by removing the background from the video stream and extracting the human image using matting; and

[0039] (A2) The method may include the step of estimating the pose of the human from the video stream and obtaining 3D pose coordinates of the pose landmark of the human.

[0040] In addition, in a 2.5D pixel-style avatar real-time generation device capable of three-dimensional interaction at low cost according to one embodiment of the present invention, step (A) is,

[0041] (A3) A step of separating the head and body of the human from the matting video stream based on the 3D coordinates of the pose landmark of the human, thereby obtaining the 2D high-resolution head image and the 2D high-resolution body-only image; and

[0042] (A4) The method may further include the step of downsampling the 2D high-resolution body-only image to obtain the 2D low-resolution body image.

[0043] In addition, in a 2.5D pixel-style avatar real-time generation device capable of three-dimensional interaction at low cost according to one embodiment of the present invention, step (B) is,

[0044] (B1) A step of generating a 2D avatar head including a 2D high-resolution face based on the above 2D high-resolution head image;

[0045] (B2) A step of generating a 2D point cloud avatar body based on the above 2D low-resolution body image;

[0046] (B3) A step of generating the 3D arm based on the 3D pose coordinates; and

[0047] (B4) Based on the 3D pose coordinates, the method may include the step of combining the 2D avatar head, the 2D point cloud avatar body, and the 3D arm to create the pixel human having a 3D arm.

[0048] In addition, in a 2.5D pixel style avatar real-time generation device capable of three-dimensional interaction at low cost according to one embodiment of the present invention, step (A2) may include the step of obtaining 3D coordinates of 33 landmarks of the human using a MediaPipe pose.

[0049] In addition, in a 2.5D pixel style avatar real-time generation device capable of three-dimensional interaction at low cost according to one embodiment of the present invention, step (A4) can obtain the 2D low-resolution body image by downsampling the 2D high-resolution body-only image using a Gaussian Pyramid technique.

[0050] In addition, in a device for real-time generation of a 2.5D pixel-style avatar capable of three-dimensional interaction at low cost according to one embodiment of the present invention, the method for real-time generation of a 2.5D pixel-style avatar capable of three-dimensional interaction at low cost, after step (B2),

[0051] (B2-1) A step of detecting gray edges of the 2D point cloud avatar body using a Laplacian edge detection technique;

[0052] (B2-2) A step of subtracting the gray edges of the detected avatar body from the alpha channel of the 2D point cloud avatar body; and

[0053] (B2-3) The method may further include the step of removing the gray edges of the 2D point cloud avatar body using an alpha-channel histogram equalization technique on the alpha channel of the avatar body from which the gray edges have been subtracted.

[0054] In addition, in a 2.5D pixel-style avatar real-time generation device capable of three-dimensional interaction at low cost according to one embodiment of the present invention, step (B3) is,

[0055] (B3-1) Based on the above 3D pose coordinates, a step of generating a skeleton of one-pixel-width between the start point and the ending point of each arm region including the upper arm, lower arm and hand;

[0056] (B3-2) A step of calculating an adaptive arm width based on the alpha values ​​of the left and right pixels for each pixel of the above skeleton; and

[0057] (B3-3) The method may include the step of generating the 3D arm by setting the z-coordinate value of each arm region pixel to a predetermined value.

[0058] In addition, in a 2.5D pixel-style avatar real-time generation device capable of low-cost 3D interaction according to one embodiment of the present invention, the adaptive arm width may increase by 1 when the alpha value of the left pixel of the skeleton is greater than 0 or when the alpha value of the right pixel of the skeleton is greater than 0.

[0059] According to one embodiment of the present invention, a computer-readable medium may be provided having a program recorded thereon for performing a method for generating a 2.5D pixel-style avatar in real time that enables three-dimensional interaction at low cost. Effects of the invention

[0060] According to a method and apparatus for real-time generation of a 2.5D pixel-style avatar capable of 3D interaction at low cost according to one embodiment of the present invention, by bridging the gap between a 2D streaming character and a 3D virtual character and incorporating the advantages of each, a 2.5D pixel-style avatar capable of 3D interaction in a virtual environment at low cost is generated, thereby enabling 3D interaction of the arm, which is essential in 3D space, at low cost while displaying rich facial expressions based on 2D streaming.

[0061] In addition, according to the method and apparatus for real-time generation of a 2.5D pixel-style avatar capable of 3D interaction at low cost according to one embodiment of the present invention, a standard for new character expression within the metaverse world is proposed, and when utilized, a richer and more realistic character expression is possible. Brief explanation of the drawing

[0062] FIG. 1 is a drawing illustrating a 2.5D pixel-style avatar real-time generation device capable of three-dimensional interaction at low cost according to an embodiment of the present invention. FIG. 2 is a flowchart of a method for generating a 2.5D pixel-style avatar in real-time that enables 3D interaction at low cost according to an embodiment of the present invention. FIG. 3 is a detailed flowchart of a method for generating a 2.5D pixel-style avatar in real-time that enables 3D interaction at low cost according to an embodiment of the present invention. FIG. 4 is a diagram illustrating a computing device capable of performing a method for generating a 2.5D pixel-style avatar in real-time that enables three-dimensional interaction at low cost according to an embodiment of the present invention. FIG. 5 is a diagram illustrating matting and pose estimation. Fig. 6 is a diagram illustrating high-resolution face video stream technology. FIG. 7 is a diagram illustrating low-resolution body-only video stream technology. FIG. 8 is a diagram illustrating 2D head visualization technology. Figures 9 and 10 are drawings for explaining point cloud edge processing technology. Fig. 11 is a diagram illustrating arm movement detection and 3D arm generation technology. FIG. 12 is a diagram illustrating the advantages of the technology proposed in the present invention. Specific details for implementing the invention

[0063] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" for components used in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. That is, the term "part" used in this invention refers to a hardware component such as software, FPGA, or ASIC, and the "part" performs certain roles. However, the meaning of "part" is not limited to software or hardware. The "part" may be configured to reside in an addressable storage medium or may be configured to run one or more processors. Accordingly, as an example, a 'part' includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or further separated into additional components and 'parts'.

[0064] In addition, when describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art may obscure the essence of the embodiments disclosed in this specification, such detailed description is omitted. Furthermore, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification, and the technical concept disclosed in this specification is not limited by the attached drawings; it should be understood that the drawings include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the invention.

[0065] Hereinafter, in describing the present invention, detailed descriptions of related known technologies that may unnecessarily obscure the essence of the invention are omitted.

[0066] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

[0067] FIG. 1 is a drawing illustrating a 2.5D pixel style avatar real-time generation device (100) capable of three-dimensional interaction at low cost according to an embodiment of the present invention, FIG. 2 and FIG. 3 are flowcharts of a method for real-time generation of a 2.5D pixel style avatar capable of three-dimensional interaction at low cost according to an embodiment of the present invention, and FIG. 4 is a block diagram of a computing device (400) in which the method for real-time generation of a 2.5D pixel style avatar capable of three-dimensional interaction at low cost according to an embodiment of the present invention can be performed.

[0068] A 2.5D pixel style avatar real-time generation device (100) capable of low-cost 3D interaction according to one embodiment of the present invention illustrated in FIG. 1 may include a human video stream processing unit (102) that obtains a 2D high-resolution head image, a 2D low-resolution body image, and 3D pose data of a human from a video stream input from a camera (106) such as a webcam, and a pixel human visualization unit (104) that generates a pixel human including a 2D high-resolution head, a 2D low-resolution body, and 3D arms based on the 2D high-resolution head image, the 2D low-resolution body image, and the 3D pose data.

[0069] The human video stream processing unit (102) may include a human matting unit (108) that obtains a matting video stream by removing the background from the video stream and extracting the appearance of the human using matting; a pose estimation unit (110) that obtains 3D pose coordinates of the human's pose landmark by estimating the human's pose from the video stream; a head and body separation unit (112) that obtains the 2D high-resolution head image and the 2D high-resolution body-only image by separating the human's head and body from the matting video stream based on the 3D coordinates of the human's pose landmark; and a downsampling unit (114) that obtains the 2D low-resolution body image by downsampling the 2D high-resolution body-only image.

[0070] The pixel human visualization unit (104) may include an avatar head generation unit (116) that generates a 2D avatar head including a 2D high-resolution face based on the 2D high-resolution head image, a 2D point cloud avatar body generation unit (118) that generates a 2D point cloud avatar body based on the 2D low-resolution body image, and a 2D point cloud avatar generation unit (120) that generates the 3D arm based on the 3D pose coordinates, and generates the pixel human having a 3D arm by combining the 2D avatar head, the 2D point cloud avatar body, and the 3D arm based on the 3D pose coordinates.

[0071] Referring to FIGS. 2 and 4, a method for generating a 2.5D pixel-style avatar in real time capable of three-dimensional interaction at low cost according to one embodiment of the present invention is a method performed by a computing device (400) including one or more processors (402), and may include the step (step S200) of obtaining a two-dimensional (2D) high-resolution head image of a human, a two-dimensional low-resolution body image, and three-dimensional (3D) pose data from an input video stream, and the step (step S202) of generating a pixel human including a two-dimensional high-resolution head, a two-dimensional low-resolution body, and a three-dimensional arm based on the two-dimensional high-resolution head image, the two-dimensional low-resolution body image, and the three-dimensional pose data.

[0072] Meanwhile, referring to FIG. 4, a computing device (400) capable of performing a method for generating a 2.5D pixel-style avatar in real-time that enables three-dimensional interaction at low cost according to one embodiment of the present invention may be made of hardware, software, or a combination thereof. For example, a computing device (400) capable of performing a method for generating a 2.5D pixel-style avatar in real-time that enables three-dimensional interaction at low cost according to one embodiment of the present invention may be implemented in the form of a computer or a server on the Internet that includes at least one processor for performing the above functions / steps / processes.

[0073] A computing device (400) capable of performing a method for generating a 2.5D pixel-style avatar in real-time that enables three-dimensional interaction at low cost according to one embodiment of the present invention may include at least one processor (402), memory (404), input / output interface (408), and network communication interface (406) connected via a bus (410). The processor (402) may be a semiconductor device that executes processing for instructions stored in the memory (404). The memory (404) may include various types of volatile or non-volatile storage media. For example, the memory (404) may include ROM (Read Only Memory) and RAM (Random Access Memory).

[0074] Additionally, the network communication interface (406) may include a communication module such as a modem that supports wired internet communication in a user terminal such as a smartphone, laptop PC, or desktop PC, wireless internet communication such as WiFi or WiBro, or mobile communication such as WCDMA, LTE, or 5G, or a communication module such as a modem that supports short-range wireless communication methods (e.g., Bluetooth, Zigbee, WiFi, etc.).

[0075] Accordingly, the steps of the method or algorithm described in connection with the embodiments disclosed herein may be directly implemented in hardware, software modules, or a combination of both, executed by the processor (402). The software module may reside in a storage / recording medium (i.e., memory (404)) readable by a device such as a computer, such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, or a CD-ROM. An exemplary memory (404) is coupled to the processor (402), and the processor (402) can read information (code) from the memory (404) and write information (code) to the memory (404). Alternatively, the memory (404) may be integral with the processor (402). The processor (402) and the memory (404) may reside within an application-specific integrated circuit (ASIC). The ASIC may reside within a user terminal. Alternatively, the processor (402) and memory (404) may reside as individual components within the user terminal.

[0076] The processor (402) can perform the step of acquiring a 2D high-resolution head image, a 2D low-resolution body image, and 3D pose data of a human from an input video stream (step S200), and the step of generating a pixel human including a 2D high-resolution head, a 2D low-resolution body, and 3D arms based on the 2D high-resolution head image, the 2D low-resolution body image, and the 3D pose data (step S202).

[0077] The memory (404) stores instructions that cause the processor (402) to perform the above steps, the input / output interface (408) can be connected to an input / output device that provides input and output functions, and the network communication interface (406) can provide the function to communicate with devices connected to a network.

[0078] A method and apparatus for real-time generation of a 2.5D pixel-style avatar capable of three-dimensional interaction at low cost, according to one embodiment of the present invention configured as above, will be described in detail below.

[0079] To overcome the limitations of the prior art, the present invention may perform a human video stream processing step (step S200) and a pixel human visualization step (step S202). The overall configuration is as follows.

[0080] In step S200, the human video stream processing unit (102) can obtain a two-dimensional (2D) high-resolution head image, a two-dimensional low-resolution body image, and three-dimensional (3D) pose data of a human from a video stream input from a camera (106), such as a webcam.

[0081] In step S202, the pixel human visualization unit (104) can generate a pixel human including a 2D high-resolution head, a 2D low-resolution body, and a 3D arm based on the 2D high-resolution head image, the 2D low-resolution body image, and the 3D pose data.

[0083] The human video stream processing process performed by the human video stream processing unit (102) is as follows.

[0085] Matting and Pose Estimation Technology

[0086] Referring to FIGS. 1 and FIGS. 3, when an image is input from a camera (106), such as a webcam, matting and pose estimation techniques can be applied. These two deep learning models can use a video stream received from a monocular RGB webcam as input.

[0087] In step S300, the human matting unit (108) can obtain a matted video stream by using matting to remove the background from the video stream and extract only the appearance of a human. For example, the Robust Video Matting (RVM) method may be used as the matting technique. As indicated by reference number 500 in FIG. 5, it can be confirmed that only the appearance of a human is extracted from the video stream by matting.

[0088] And in step S302, the pose estimation unit (110) can estimate the pose of the human from the video stream and obtain the 3D pose coordinates of the human's pose landmarks. For example, the pose estimation unit (110) can obtain the 3D coordinates of the human's 33 landmarks, i.e., 3D pose data, using MediaPipe poses. As indicated by reference numeral 502 in FIG. 5, the 3D coordinates of the human's pose landmarks can be extracted using MediaPipe poses. In one embodiment of the present invention, the terms 3D coordinates or 3D pose data have the same meaning and may be used interchangeably.

[0090] High-resolution face video stream technology

[0091] In step S304, the head and body separation unit (112) can separate the human's head and body from the matting video stream based on the 3D coordinates of the human's pose landmarks to obtain a 2D high-resolution head image and a 2D high-resolution body-only image.

[0092] For real-time facial expressions and communication, the head of the pixel avatar can use a 2D high-resolution head video stream, such as a 2D video conference. As shown in FIG. 6, the head image can be defined as a square area with a side length equal to twice the distance from the nose joint to the neck joint.

[0094] Low-resolution body-only video stream technology

[0095] In step S306, the downsampling unit (114) can downsample the 2D high-resolution body-only image to obtain a 2D low-resolution body image.

[0096] The body of the avatar can use a body-only video stream with the head removed. For real-time performance, the downsampling unit (114) can convert a high-resolution body-only image into a low-resolution image using a downsampling method.

[0097] As illustrated in FIG. 7, the downsampling unit (114) can obtain a 2D low-resolution body image by downsampling a 2D high-resolution body-only image using a Gaussian Pyramid technique during downsampling. Accordingly, the video stream input to the downsampling unit (114) can be downsampled into images of various resolutions, such as 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the input.

[0098] The high-resolution result produced by high-resolution face video stream technology preserves the details of the human face, while the low-resolution result produced by low-resolution body-only video stream technology enables faster generation of body images due to the reduced computational load. Through the above process, head and body images and pose data required for avatar generation can be obtained. In one embodiment of the present invention, "face" or "face" has the same meaning and may be used interchangeably.

[0100] The pixel human visualization process performed by the pixel human visualization unit (104) generates a pixel avatar using images and data previously extracted by the following process.

[0102] 2D Head Visualization Technology

[0103] In step S308, the avatar head generation unit (116) can generate a 2D avatar head including a 2D high-resolution face based on the 2D high-resolution head image.

[0104] If the avatar head is in a low-resolution format like the body, the face or facial expressions may not be visible in detail, which may result in a lack of realism. To solve this problem, the avatar head generation unit (116) can display the avatar head on a 2D plane using a 2D high-resolution head image. FIG. 8 illustrates a 2D head visualization technique.

[0106] Point Cloud Edge Processing technology

[0107] In step S310, the 2D point cloud avatar body generation unit (118) can generate a 2D point cloud avatar body based on a 2D low-resolution body image.

[0108] To explain this in detail, in step S310, the 2D point cloud avatar body generation unit (118) generates a 2D point cloud avatar body, detects gray edges of the 2D point cloud avatar body using a Laplacian edge detection technique, subtracts the detected gray edges of the avatar body from the alpha channel of the 2D point cloud avatar body, and removes the gray edges of the 2D cloud avatar body using an alpha-channel histogram equalization technique on the alpha channel of the avatar body from which the gray edges were subtracted.

[0109] In the case of a pixel avatar body, if the low-resolution image is directly converted to a point cloud, unnatural gray edges will exist as shown in Fig. 10a.

[0110] To remove these edges, the 2D point cloud avatar body generation unit (118) may use a Laplacian edge detection technique and an alpha-channel histogram equalization technique.

[0111] The edge detection result using the Laplacian edge detection technique is as shown in the right image of Fig. 9. By removing this result from the original image, most edges can be removed. Since the distribution of alpha values ​​becomes more uniform after applying histogram equalization in the alpha-channel, it becomes possible to remove edges as a result, as shown in Fig. 10c.

[0112] FIG. 10a shows a downsampled body-only point cloud image, FIG. 10b shows the result of subtracting the edge detection result from the alpha channel of the image in FIG. 10a, and FIG. 10c shows the result of cleanly removing edges by applying a histogram equalization technique to the alpha channel of the image in FIG. 10b.

[0114] Arm movement detection and 3D arm generation technology

[0115] In step S312, the 2D point cloud avatar generation unit (120) having a 3D arm can generate a 3D arm based on the 3D pose coordinates and combine the 2D avatar head, the 2D point cloud avatar body, and the 3D arm based on the 3D pose coordinates to generate a 2.5D pixel human having a 3D arm that enables low-cost 3D interaction of the arm, which is essential in 3D space, while also displaying rich facial expressions based on 2D streaming.

[0116] To explain this in detail, in step S312, the 2D point cloud avatar generation unit (120) having a 3D arm may use an adaptive arm region method when specifically determining the arm range in generating the 3D arm.

[0117] As illustrated in FIG. 11, a 2D point cloud avatar generator (120) having 3D arms can first generate a skeleton (1106) of one-pixel-width between the start point and the ending point of each arm region, including the upper arm (1100), the lower arm (1102), and the hand (1104), based on 3D pose coordinates.

[0118] Next, the 2D point cloud avatar generation unit (120) having a 3D arm, as described in Equation 1, for each pixel of the skeleton (1106), the alpha value of the left pixel ( ) and the alpha value of the right pixel( adaptive arm width based on ) ) can be calculated.

[0119]

[0120] In the above, represents the adaptive arm width, and represents the adaptive parameter, and represents the minimum arm width threshold, and represents the maximum arm width threshold, and represents the alpha value of the left pixel of the skeleton (1106), and can represent the alpha value of the right pixel of the skeleton (1106).

[0121] As described in mathematical formula 1, the adaptive arm width ( ) can be increased by 1 if the alpha value of the left pixel of the skeleton (1106) is greater than 0 or the alpha value of the right pixel of the skeleton (1106) is greater than 0.

[0122] Next, the 2D point cloud avatar generation unit (120) having 3D arms can convert the avatar arms into 3D by setting the z-coordinate value of each arm area pixel to a predetermined value. Since the units of the 3D coordinates estimated by pose estimation and the coordinate system of virtual space are different, a parameter for converting the estimated z-coordinates to virtual space coordinates can be calculated. Then, by setting the 3D coordinates of the avatar arm part to a predetermined value, a 3D arm can be generated as a result. As shown in FIG. 11, the 2D point cloud avatar generation unit (120) having 3D arms can generate 3D arms in the order of upper arm (1100), lower arm (1102), and hand (1104).

[0123] FIG. 12 is a diagram illustrating the advantages of a method and apparatus for real-time generation of a 2.5D pixel-style avatar capable of three-dimensional interaction at low cost according to an embodiment of the present invention.

[0124] Referring to FIG. 12, the method and apparatus for real-time generation of a 2.5D pixel-style avatar capable of three-dimensional interaction at low cost according to one embodiment of the present invention incorporates the advantages of a 2D video streaming method 2D conference and a 3D virtual environment method 3D virtual avatar, respectively, and can be seen to enable the implementation of three-dimensional interaction of the arm, which is essential in three-dimensional space, at low cost while displaying rich facial expressions based on 2D streaming.

[0125] The present invention described above can be implemented as computer-readable code on a medium on which a program is recorded. The computer-readable medium may be one that continuously stores a program executable by a computer, or temporarily stores it for execution or download. Furthermore, the medium may be various recording or storage means in the form of a single or multiple hardware components, and is not limited to a medium directly connected to a computer system but may also exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software. Accordingly, the above detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the present invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included within the scope of the present invention.

[0126] As described above, the present invention has been explained by specific details such as specific components, limited embodiments, and drawings; however, this is provided merely to aid in a more comprehensive understanding of the invention, and the invention is not limited to the above embodiments. A person skilled in the art to which the invention pertains will be able to make various modifications and variations within the scope of the essential characteristics of the invention. Accordingly, the concept of the present invention should not be limited to the described embodiments, and all technical concepts that are equivalent to or have equivalent variations to the claims set forth below, as well as the claims themselves, should be interpreted as being included within the scope of the rights of the present invention. Explanation of the symbols

[0127] 100: Low-cost 2.5D pixel-style avatar real-time generator capable of 3D interaction 102: Human Video Stream Processing Unit 104: Pixel Human Visualization Unit 106 : Camera 108 : Human Matting Department 110: Pose Estimation Section 112: Head and Body Separation Section 114: Downsampling Section 116: Avatar Head Generation Section 118 : 2D Point Cloud Avatar Body Generation Section 120 : 2D point cloud avatar generator with 3D arms 400: Computing device 402: Processor 404: Memory 406: Network communication interface 408: I / O interface 410: Bus

Claims

Claim 1 A method performed on a computing device comprising one or more processors, comprising: (A1) a step of obtaining a matting video stream by removing a background from a video stream and extracting the appearance of a human using matting; (A2) a step of obtaining 3D pose coordinates of pose landmarks of the human by estimating the pose of the human from the video stream; (A3) a step of obtaining a 2D high-resolution head image and a 2D high-resolution body-only image by separating the head and body of the human from the matting video stream based on the 3D coordinates of the pose landmarks of the human; and (A4) a step of obtaining a 2D low-resolution body image by downsampling the 2D high-resolution body-only image (A); A method for generating a 2.5D pixel-style avatar in real-time capable of 3D interaction at low cost, comprising the following steps: (B) generating a 2D avatar head including a 2D high-resolution face based on the 2D high-resolution head image; (B2) generating a 2D point cloud avatar body based on the 2D low-resolution body image; (B3) generating a 3D arm based on the 3D pose coordinates; and (B4) generating a pixel human having a 3D arm by combining the 2D avatar head, the 2D point cloud avatar body, and the 3D arm based on the 3D pose coordinates. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 A method for generating a 2.5D pixel-style avatar in real-time capable of 3D interaction at low cost, wherein, in claim 1, step (A2) includes the step of obtaining 3D coordinates of 33 landmarks of the human using a MediaPipe pose. Claim 6 In claim 1, the step (A4) is a method for generating a 2.5D pixel-style avatar in real-time that enables 3D interaction at low cost, wherein the 2D low-resolution body image is obtained by downsampling the 2D high-resolution body-only image using a Gaussian Pyramid technique. Claim 7 A method for generating a 2.5D pixel-style avatar in real-time capable of 3D interaction at low cost, wherein, after step (B2), (B2-1) detecting gray edges of the 2D point cloud avatar body using a Laplacian edge detection technique; (B2-2) subtracting the detected gray edges of the avatar body from the alpha channel of the 2D point cloud avatar body; and (B2-3) removing the gray edges of the 2D point cloud avatar body using an alpha-channel histogram equalization technique on the alpha channel of the avatar body from which the gray edges were subtracted. Claim 8 A method for generating a 2.5D pixel-style avatar in real-time that enables 3D interaction at low cost, wherein the step (B3) comprises: (B3-1) generating a skeleton of one-pixel-width between the start point and the ending point of each arm region, including the upper arm, lower arm, and hand, based on the 3D pose coordinates; (B3-2) calculating an adaptive arm width based on the alpha values ​​of the left and right pixels for each pixel of the skeleton; and (B3-3) generating the 3D arm by setting the z-coordinate value of each arm region pixel to a predetermined value. Claim 9 A method for generating a 2.5D pixel-style avatar in real-time with low-cost 3D interaction, wherein the adaptive arm width increases by 1 when the alpha value of the left pixel of the skeleton is greater than 0 or when the alpha value of the right pixel of the skeleton is greater than 0. Claim 10 A device comprising one or more processors for performing a method for generating a 2.5D pixel-style avatar in real-time capable of 3D interaction at low cost, wherein the method for generating a 2.5D pixel-style avatar in real-time capable of 3D interaction at low cost comprises: (A1) a step of obtaining a matting video stream by removing a background from a video stream and extracting the appearance of a human using matting; (A2) a step of estimating the pose of the human from the video stream to obtain 3D pose coordinates of the human's pose landmarks; (A3) a step of obtaining a 2D high-resolution head image and a 2D high-resolution body-only image by separating the head and body of the human from the matting video stream based on the 3D coordinates of the human's pose landmarks; and (A4) a step of obtaining a 2D low-resolution body image by downsampling the 2D high-resolution body-only image. A 2.5D pixel-style avatar real-time generation device capable of 3D interaction at low cost, comprising the step (B) of: (B1) generating a 2D avatar head including a 2D high-resolution face based on the 2D high-resolution head image; (B2) generating a 2D point cloud avatar body based on the 2D low-resolution body image; (B3) generating a 3D arm based on the 3D pose coordinates; and (B4) generating a pixel human having a 3D arm by combining the 2D avatar head, the 2D point cloud avatar body, and the 3D arm based on the 3D pose coordinates. Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 A 2.5D pixel-style avatar real-time generation device capable of 3D interaction at low cost, wherein the step (A2) includes the step of obtaining 3D coordinates of 33 landmarks of the human using a MediaPipe pose. Claim 15 In claim 10, the step (A4) is a 2D low-resolution body image obtained by downsampling the 2D high-resolution body-only image using a Gaussian Pyramid technique, a 2.5D pixel-style avatar real-time generation device capable of 3D interaction at low cost. Claim 16 A 2.5D pixel-style avatar real-time generation device capable of 3D interaction at low cost, further comprising, after step (B2), (B2-1) a step of detecting gray edges of the 2D point cloud avatar body using a Laplacian edge detection technique; (B2-2) a step of subtracting the detected gray edges of the avatar body from the alpha channel of the 2D point cloud avatar body; and (B2-3) a step of removing the gray edges of the 2D point cloud avatar body using an alpha-channel histogram equalization technique on the alpha channel of the avatar body from which the gray edges were subtracted. Claim 17 A 2.5D pixel-style avatar real-time generation device capable of 3D interaction at low cost, wherein step (B3) comprises: (B3-1) generating a skeleton of one-pixel-width between the start point and the ending point of each arm region including the upper arm, lower arm and hand, based on the 3D pose coordinates; (B3-2) calculating an adaptive arm width based on the alpha values ​​of the left and right pixels for each pixel of the skeleton; and (B3-3) generating the 3D arm by setting the z-coordinate value of each arm region pixel to a predetermined value. Claim 18 A 2.5D pixel-style avatar real-time generation device capable of 3D interaction at low cost, wherein the adaptive arm width increases by 1 when the alpha value of the left pixel of the skeleton is greater than 0 or when the alpha value of the right pixel of the skeleton is greater than 0. Claim 19 A computer-readable medium having a program recorded thereon for performing the method of any one of claims 1, 5 to 9.

Images