Electronic device for generating virtual avatar and operation method thereof

Abstract

The present disclosure relates to an electronic device for generating a virtual avatar and an operation method thereof, and the operation method of the electronic device for generating a virtual avatar, according to an embodiment of the present disclosure, may comprise the steps of: acquiring image information regarding a streamer; on the basis of the image information, extracting a landmark for at least one of the motion and expression of the streamer; checking expression information regarding the streamer on the basis of the landmark; and generating a virtual avatar corresponding to the streamer on the basis of the expression information.

Description

Electronic device for generating a virtual avatar and method for operating the same

[0001] The present disclosure relates to a content production technology, and more specifically, to an electronic device for generating a virtual avatar based on video information about a streamer and an operating method thereof.

[0002] With the widespread adoption of the Internet, various media platforms are being built to support personal broadcasting and content sharing. Content creators (or streamers) can then deliver their content to viewers through these platforms. Unlike traditional broadcasting, content delivery via media platforms is rapidly gaining popularity because it doesn't impose any specific restrictions on channel creation, content topics, or viewer access to content. This has led to a surge in the number of streamers providing content through these platforms.

[0003] The surge in streamers has led to a wider variety of content, and this has led to the emergence of content utilizing virtual avatars. Content utilizing virtual avatars is gaining traction as a new form of content, offering opportunities for limitless experiences that transcend the constraints of reality while simultaneously providing interactions similar to those in the real world. While extensive research into virtual avatars has driven this trend, current virtual avatar implementation technology remains limited in terms of realism and expressiveness.

[0004] In this regard, reference may be made to prior literature such as Korean Patent Publication No. 10-2023-0050570A and Korean Patent Publication No. 10-2022-0160558A.

[0005] The present disclosure aims to provide an electronic device and an operating method thereof for generating a virtual avatar based on video information about a streamer.

[0006] The present disclosure aims to provide an electronic device and an operating method thereof for generating a virtual avatar that reflects expressions related to the movement and facial expressions of a streamer.

[0007] The problems to be solved by the present disclosure are not limited to the problems described above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0008] In an operating method of an electronic device for generating a virtual avatar according to an embodiment of the present disclosure, the operating method may include a step of obtaining image information about a streamer, a step of extracting a landmark for at least one of a movement and an expression of the streamer based on the image information, a step of confirming expression information about the streamer based on the landmark, and a step of generating a virtual avatar corresponding to the streamer based on the expression information.

[0009] For example, the step of extracting the landmark may include a step of performing a preprocessing process on the image information, wherein the preprocessing process may include at least some of resizing, normalization, and binary translation processes on the image information.

[0010] For example, when extracting a landmark for the movement of the streamer based on the image information, the landmark may be composed of a plurality of reference points corresponding to joint positions recognized from an image of a body part of the streamer included in the image information.

[0011] For example, when extracting a landmark for the streamer's facial expression based on the above video information, the landmark may be composed of a plurality of reference points derived in correspondence to the streamer's facial image included in the above video information through an artificial intelligence model.

[0012] As an example, the step of extracting the landmark may include a step of sampling a plurality of images corresponding to a preset period based on the image information, a step of arranging the plurality of images in chronological order to form an image information set, and a step of extracting a landmark for at least one of the streamer's movements and expressions based on the image information set.

[0013] For example, the step of checking expression information for the streamer may include the step of assigning a weight to each of a plurality of expression processing items based on the landmark, and the virtual avatar may be generated by implementing at least one of the expressions and movements derived based on the weights for each of the plurality of expression processing items.

[0014] As an example, the method may further include a step of generating content based on the virtual avatar.

[0015] For example, image information for the above streamer can be derived through fusion of image data acquired from multiple shooting devices.

[0016] An electronic device for generating a virtual avatar according to one embodiment of the present disclosure comprises a transceiver, a memory storing instructions, and a processor, wherein the processor connected to the transceiver and the memory is configured to obtain image information about a streamer, extract a landmark for at least one of a movement and an expression of the streamer based on the image information, confirm expression information about the streamer based on the landmark, and generate a virtual avatar corresponding to the streamer based on the expression information.

[0017] In one embodiment of the present disclosure, a computer program stored in a computer-readable storage medium for executing a method for generating a virtual avatar, combined with hardware, may include the steps of acquiring image information about a streamer, extracting a landmark for at least one of a movement and an expression of the streamer based on the image information, confirming expression information about the streamer based on the landmark, and generating a virtual avatar corresponding to the streamer based on the expression information.

[0018] According to the present disclosure, by providing an electronic device and an operating method thereof for generating a virtual avatar based on video information about a streamer, the convenience of content production by the streamer can be improved.

[0019] According to the present disclosure, an electronic device and an operating method thereof for generating a virtual avatar that reflects expressions related to the movement and facial expressions of a streamer are provided, thereby enabling the implementation of a more vivid virtual avatar.

[0020] The effects according to the present disclosure are not limited to the effects described above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0021] FIG. 1 is a diagram illustrating a content providing system according to an embodiment of the present disclosure.

[0022] FIG. 2 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0023] FIG. 3 is a block diagram illustrating an expression identification unit according to an embodiment of the present disclosure.

[0024] FIG. 4 is a flowchart for explaining an operation method of an electronic device according to an embodiment of the present disclosure.

[0025] FIG. 5A is a diagram illustrating a process of extracting landmarks related to a streamer's facial expression according to an embodiment of the present disclosure.

[0026] FIG. 5b is a diagram illustrating a process of extracting landmarks related to the movement of a streamer according to an embodiment of the present disclosure.

[0027] FIG. 6 is a diagram for explaining a process for determining an expression implemented in a virtual avatar according to an embodiment of the present disclosure.

[0028] FIG. 7a is a drawing for explaining an implementation aspect of a virtual avatar according to a streamer's facial expression according to an embodiment of the present disclosure.

[0029] FIG. 7b is a drawing for explaining an implementation aspect of a virtual avatar according to the movement of a streamer according to an embodiment of the present disclosure.

[0030] FIG. 8 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0031] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by the exemplary embodiments. Unless otherwise defined, all terms (including technical and scientific terms) used in this specification shall be used with meanings that can be commonly understood by those of ordinary skill in the technical field to which this disclosure pertains. However, this may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc.

[0032] Additionally, terms defined in commonly used dictionaries should not be interpreted ideally or excessively unless explicitly and specifically defined otherwise. In certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in this disclosure should be defined based on their meaning and the overall content of this disclosure, rather than simply their names.

[0033] Throughout this specification, when a part is said to "include" a certain component, this does not mean that other components may be included, but rather that other components may be excluded, unless specifically stated otherwise. Furthermore, the singular forms used herein also include plural forms unless specifically stated otherwise. Furthermore, the expression "at least one of a, b, and / or c" used throughout this specification can encompass "a alone," "b alone," "c alone," "a and b," "a and c," "b and c," or "all of a, b, and c."

[0034] Meanwhile, terms such as "first and / or second" used in this specification may be used to describe various components, but are only used to distinguish one component from another and are not intended to be limited to the components referred to by those terms. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and the second component may also be referred to as the first component.

[0035] In addition, terms such as “unit”, “module”, etc. described in this specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software, or a combination of hardware and software. In addition, embodiments of the present disclosure in this specification may be represented by functional block configurations and various processing steps. These functional blocks may be implemented by various numbers of hardware or / and software configurations that execute specific functions. For example, embodiments of the present disclosure may employ direct circuit configurations such as memory, processing, logic, look-up tables, etc. that may execute various functions under the control of one or more microprocessors or other control devices.

[0036] Similar to the components disclosed herein that can be implemented as software programs or software elements, embodiments of the present disclosure may be implemented in a programming or scripting language such as C, C++, Java, assembler, etc., including various algorithms implemented as a combination of data structures, processes, routines, or other programming constructs. Functional aspects may be implemented as algorithms that run on one or more processors. Furthermore, the present embodiments may employ conventional techniques for at least one of electronic configuration, signal processing, and data processing. Terms such as “mechanism,” “element,” “means,” and “configuration” may be used broadly and are not limited to mechanical and physical components. The terms may include the meaning of a series of software processes (routines) in connection with a processor, etc.

[0037] Each block of the processing flow diagrams attached to this specification and combinations of the flow diagrams can be performed by computer program instructions. These computer program instructions can be installed in a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, so that the instructions, when executed by the processor of the computer or other programmable data processing equipment, create a means for performing the functions described in the flow diagram block(s).

[0038] These computer program instructions may be stored in a computer-available or computer-readable memory that can direct a computer or other programmable data processing device to implement a function in a particular manner, and the instructions stored in the computer-available or computer-readable memory may also produce an article of manufacture that includes instruction means for performing the function described in the flowchart block(s).

[0039] Since the computer program instructions may be installed on a computer or other programmable data processing device, a series of operational steps may be performed on the computer or other programmable data processing device to create a computer-executable process, and the instructions that cause the computer or other programmable data processing device to perform the steps for performing the functions described in the flowchart block(s) may also provide steps for performing the functions described in the flowchart block(s).

[0040] Additionally, each block may represent a module, segment, or portion of code that contains one or more executable instructions for performing a specific logical function(s). Furthermore, in some alternative implementation examples, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may actually be executed substantially concurrently, or the blocks may sometimes be executed in reverse order, depending on their respective functions.

[0041] The “electronic device” or “terminal” referred to herein may be implemented as a computer or portable terminal capable of connecting to a server or other terminal via a network. Here, the computer includes, for example, a notebook, desktop, or laptop equipped with a web browser, and the portable terminal may include, for example, a wireless communication device that ensures portability and mobility, such as a communication-based terminal such as IMT (International Mobile Telecommunication), CDMA (Code Division Multiple Access), W-CDMA (W-Code Division Multiple Access), LTE (Long Term Evolution), and all types of handheld-based wireless communication devices such as smartphones and tablet PCs. In addition, the “electronic device” or “terminal” referred to herein may also include a processor, a memory that stores and executes program data, permanent storage such as a disk drive, a communication port that communicates with an external device, a user interface device such as a touch panel, a key, a button, etc.

[0042] Meanwhile, in embodiments according to the present disclosure, functions related to artificial intelligence may be implemented through a processor and memory. In this case, the processor may be any one of a general-purpose processor such as a CPU (Center Processing Unit), an AP (Application Processor), a DSP (Digital Signal Processor), a graphics-only processor such as a GPU (Graphics Processing Unit), a VPU (Vision Processing Unit), and an AI-only processor such as an NPU (Neural Network Processing Unit). The processor may process input data according to predefined operation rules or AI models stored in the memory. Alternatively, if the processor is an AI-only processor, the AI-only processor may be designed with a hardware structure specialized for processing a specific AI model. In some embodiments according to the present disclosure, functions related to artificial intelligence may be implemented through a plurality of processors.

[0043] In embodiments of the present disclosure, predefined operating rules or artificial intelligence models may be configured to perform machine learning. Here, "configured to perform machine learning" means that the predefined operating rules or artificial intelligence models are trained using a learning algorithm and a plurality of learning data sets to perform a desired characteristic (or purpose). This learning may be performed within the device itself implementing the artificial intelligence according to the present disclosure, or may be performed through a separate server and / or system.

[0044] Artificial intelligence models can be implemented as neural networks (or artificial neural networks) and operate based on statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network can refer to a general model in which artificial neurons (nodes) form a network by combining synapses, and through learning, the strength of the synaptic connections changes, thereby achieving problem-solving capabilities. A neural network can be composed of multiple neural network layers. For example, a neural network can include an input layer, a hidden layer, and an output layer. Each of the multiple neural network layers can include at least one node and at least one weight, and can perform neural network operations through operations between the weights and the computational results of previous layers. At least one weight of the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, at least one weight can be updated during the learning process to reduce or minimize the loss or cost values ​​obtained from the artificial intelligence model. Neural networks can infer the desired outcome from arbitrary input.

[0045] The learning methods of artificial intelligence models can be categorized into supervised learning, where input and output data are provided as training data, and the correct answer (output data) corresponding to the problem (input data) is determined, unsupervised learning, where only input data is provided without output data, and the correct answer (output data) corresponding to the problem (input data) is not determined, and reinforcement learning, where a reward is given whenever an action is taken in the current state, and learning progresses in the direction of maximizing this reward. Alternatively, they can be categorized according to the architecture, which is the structure of the learning model.

[0046] In an embodiment of the present disclosure, the artificial intelligence model is a CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, etc., R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), RNN (Recurrent Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT for natural language processing, SP-BERT, MRC / QA, Text Analysis, Dialog System, GPT-3, GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, ResNet for data intelligence, Anomaly Detection, Prediction, Time-Series Forecasting, Optimization, At least one of various artificial intelligence structures and algorithms, such as Recommendation, Data Creation, etc., may be used, and the above-described examples are merely listing examples of artificial intelligence structures and algorithms used according to embodiments of the present disclosure, and do not limit the artificial intelligence structures and algorithms used according to embodiments of the present disclosure.

[0047] Meanwhile, in the present disclosure, methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable codes or program instructions executable on a processor. Here, the computer-readable recording medium may include a magnetic storage medium (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and an optical reading medium (e.g., CD-ROM, DVD: Digital Versatile Disc)). The computer-readable recording medium may be distributed and executed on network-connected computer systems.

[0048] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In describing the embodiments, descriptions of technical details that are well known in the technical field to which the present invention pertains and are not directly related to the present invention will be omitted. This is to convey the gist of the present invention more clearly without obscuring unnecessary explanation. For the same reason, some components in the accompanying drawings are exaggerated, omitted, or schematically depicted. Furthermore, the size of each component does not entirely reflect the actual size. Throughout this specification, the same reference numerals may refer to the same or corresponding components.

[0049] FIG. 1 is a drawing illustrating a content provision system (10) according to an embodiment of the present disclosure.

[0050] Referring to FIG. 1, a content providing system (10) according to an embodiment of the present disclosure may include a first electronic device (110), a second electronic device (120), a streamer terminal (130), and a viewer terminal (140). In the content providing system (10) according to an embodiment of the present disclosure, the first electronic device (110) and the second electronic device (120), the first electronic device (110) and the streamer terminal (130), and the second electronic device (120) and the viewer terminal (140) may communicate and exchange information through a network or a physically implemented electrical connection structure.

[0051] According to an embodiment of the present disclosure, the network may include at least one of a Personal Area Network (PAN), a Local Area Network (LAN), a Campus Area Network (CAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a Broad Band Network (BBN), and the Internet. In addition, the network may include at least one of network topologies including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network. In an embodiment of the present disclosure, the communication method performed is not limited to the types of the above-described networks, and may include not only a communication method utilizing a communication network that the network may include, but also short-range wireless communication between devices.

[0052] In an embodiment of the present disclosure, each of the first electronic device (110) and the second electronic device (120) may include a transceiver, a memory, and a processor. In addition, each of the first electronic device (110) and the second electronic device (120) refers to a unit that processes at least one function or operation, which may be implemented by hardware, software, or a combination of hardware and software. In an embodiment, each of the first electronic device (110) and the second electronic device (120) may include a plurality of computer systems or computer software implemented as network servers. For example, each of the first electronic device (110) and the second electronic device (120) may refer to a computer system and computer software that are connected to a lower-level device that can communicate with another network server through a computer network such as an intranet or the Internet, receive a task execution request, perform the task accordingly, and provide the execution result.

[0053] Meanwhile, although the first electronic device (110) and the second electronic device (120) are mentioned in this specification as being physically separate components, this is only some embodiments according to the present disclosure, and according to another embodiment of the present disclosure, the first electronic device (110) and the second electronic device (120) may have logically separate structures, in which case they may be implemented by separate functions in one server. In addition, each of the first electronic device (110) and the second electronic device (120) may be understood as a broad concept including a series of application programs that can operate on a network server and various databases built therein. For example, each of the first electronic device (110) and the second electronic device (120) may be implemented using a network server program that is provided in various ways according to an operating system such as DOS, Windows, Linux, Unix, or MacOS.

[0054] In an embodiment of the present disclosure, the first electronic device (110) may provide a program for producing and generating content to the streamer terminal (130). For example, the program provided from the first electronic device (110) to the streamer terminal (130) may be a program for producing and generating metaverse-based content. In an embodiment, the streamer may produce and generate content to be provided to viewers using the program provided from the first electronic device (110). Specifically, the streamer may provide the first electronic device (110) with any information for producing and generating content through the streamer terminal (130), and the first electronic device (110) may produce content based on the information obtained from the streamer terminal (130).

[0055] In an embodiment, any information for content production and creation may include video (or image) information about the streamer. The video information about the streamer may refer to information obtained by filming the streamer using a filming device (not shown) such as a camera, camcorder, or webcam, and may be provided to the first electronic device (110) through a streamer terminal (130) that communicates with the filming device via a network or a physically implemented electrical connection. In an embodiment, the filming device may include a sensor device for detecting video information, and the sensor device may detect a general visible light range to acquire an RGB image or a binary image. Alternatively, in some embodiments, video information about the streamer obtained from a plurality of filming devices may be provided to the first electronic device (110) through the streamer terminal (130). In this case, each piece of image information obtained from the plurality of filming devices may be fused through data fusion. Alternatively, in some embodiments, the first electronic device (110) may receive pre-generated content from a streamer terminal (130) and extract video information about the streamer from the pre-generated content.

[0056] In an embodiment of the present disclosure, a first electronic device (110) may generate a virtual avatar based on video information about a streamer. In an embodiment, the first electronic device (110) may generate a virtual avatar corresponding to the video information about the streamer using a preset algorithm or artificial intelligence model. Specifically, the first electronic device (110) may generate a virtual avatar by reflecting expressions related to the streamer's facial expressions and movements. The configuration and operation of the first electronic device (110) for generating a virtual avatar will be described in detail with reference to FIGS. 2 to 8, which will be described later. The first electronic device (110) may provide content produced and generated based on the generated virtual avatar to a second electronic device (120).

[0057] In an embodiment of the present disclosure, the second electronic device (120) may store content information acquired from the first electronic device (110) and provide the content to the viewer terminal (140). The second electronic device (120) may transmit information about the content to the viewer terminal (140) of the viewer viewing the content, or may provide a media service that provides the content to the viewer in a real-time streaming manner. For example, the second electronic device (120) may provide the content to the viewer through a media platform such as YouTube, Africa TV, or Twitch.

[0058] A content provision system (10) according to an embodiment of the present disclosure can improve the convenience of content creation for streamers by generating a virtual avatar based on video information about the streamer and providing content produced and generated based on the virtual avatar to viewers. In addition, the content provision system (10) according to an embodiment of the present disclosure can implement a more vivid virtual avatar by generating a virtual avatar that reflects expressions related to the streamer's movements and facial expressions and providing content produced and generated based on the virtual avatar to viewers, thereby enabling viewers to enjoy more realistic virtual content.

[0059] FIG. 2 is a block diagram illustrating an electronic device (200) according to an embodiment of the present disclosure.

[0060] The electronic device (200) illustrated in FIG. 2 may correspond to the first electronic device (110, see FIG. 1) of FIG. 1 described above, and referring to FIG. 2, the electronic device (200) may include an expression identification unit (210) and a content generation unit (220).

[0061] According to an embodiment of the present disclosure, the expression identification unit (210) can identify expressions related to the expression and movement of the streamer based on image information about the streamer acquired from the streamer terminal (130, see FIG. 1). Specifically, the expression identification unit (210) analyzes image information based on a preset algorithm for the electronic device (200), and identifies expressions related to the movement and expression of the streamer based on the analysis result, or identifies expressions related to the movement and expression of the streamer based on the image information about the streamer through an artificial intelligence model built in the electronic device (200). The electronic device (200) can provide expression information including information related to the identified expression to the content generation unit (220). The specific configuration and operation of the expression identification unit (210) according to an embodiment of the present disclosure will be described in detail with reference to FIGS. 3, 5A, and 5B, which will be described later.

[0062] According to an embodiment of the present disclosure, the content generation unit (220) may generate a virtual avatar corresponding to a streamer based on expression information acquired from the expression identification unit (210). In the embodiment, the virtual avatar may be generated to embody the movement and expression of the streamer, and the content generation unit (220) may generate content based on the generated virtual avatar. Meanwhile, content information including information related to content generated from the content generation unit (220) may be provided to a second electronic device (120, see FIG. 1) for providing a media service, and may provide the content to a viewer terminal (140, see FIG. 1) requesting viewing of the corresponding content.

[0063] Meanwhile, although not shown, in some embodiments, when image information is provided to the electronic device (200) from the streamer terminal (130), a sampling processing unit (not shown) for extracting image information from the image information may sample the image to extract a plurality of images. The sampling processing unit may provide an image information set in which information about each of the plurality of images is arranged in chronological order to the expression identification unit (210), and the expression identification unit (210) may derive expression information about changes in the streamer's facial expression and movement based on each of the images included in the image information set. In an embodiment, a sampling period for extracting image information by the sampling processing unit may be preset for the electronic device (200).

[0064] FIG. 3 is a block diagram illustrating an expression identification unit (300) according to an embodiment of the present disclosure.

[0065] The expression identification unit (300) illustrated in FIG. 3 may correspond to the expression identification unit (200, see FIG. 2) of FIG. 2 described above, and referring to FIG. 3, the expression identification unit (300) according to an embodiment of the present disclosure may include a preprocessing unit (310), a landmark extraction unit (320), and an expression determination unit (330).

[0066] The preprocessing unit (310) according to an embodiment of the present disclosure may perform preprocessing on image information (or an image information set) acquired from a streamer terminal (130, see FIG. 1), and the preprocessing process may include at least some processes among resizing, normalization, and binary translation of the image information. In some embodiments, if the preprocessing process for the image information is not required, the preprocessing process performed by the preprocessing unit (310) may be excluded. The preprocessing unit (310) may provide the image information (or an image information set) on which the preprocessing process has been performed to the landmark extraction unit (320).

[0067] The landmark extraction unit (320) according to an embodiment of the present disclosure may extract landmarks from the image information (or image information set) acquired from the preprocessing unit (310) to identify the expression and movement of the streamer. In an embodiment, a landmark for identifying the expression of the streamer may be acquired from the streamer's face in the image information, and may mean a set of two-dimensional or three-dimensional reference points identified from the streamer's face. Meanwhile, in an embodiment, a landmark for identifying the movement of the streamer may be acquired from a body part (e.g., a hand, an arm, a head, a torso, etc.) of the streamer in the image information, and may mean a set of two-dimensional or three-dimensional reference points identified from a body part of the streamer. In an embodiment, the landmark extraction unit (320) may extract landmarks through an artificial intelligence model, and information on the extracted landmarks may be provided to the expression determination unit (330). The learning method of the artificial intelligence model for landmark extraction and the landmarks extracted in relation to the streamer's facial expressions and movements will be described in detail through FIGS. 5a and 5b described below.

[0068] The expression determination unit (330) according to an embodiment of the present disclosure may determine an expression related to the streamer's facial expression and movement based on the landmark extracted from the landmark extraction unit (320). Specifically, the expression determination unit (330) may derive a weight for each of a plurality of preset expression processing items based on landmark information, and may determine expression information based on the derived weight. In some embodiments, the expression determination unit (330) may derive a weight for each of a plurality of expression processing items using an artificial intelligence model. The expression information may be provided to the content generation unit (220, see FIG. 2) and applied to a virtual avatar of content generated by the content generation unit (220). A specific operation for determining the expression information by the expression determination unit (330) will be described in detail with reference to FIG. 6, which will be described later.

[0069] FIG. 4 is a flowchart for explaining an operation method of an electronic device (200, see FIG. 2) according to an embodiment of the present disclosure.

[0070] In step S410, the electronic device (200) according to an embodiment of the present disclosure can obtain video (or image) information about the streamer from the streamer terminal (130, see FIG. 1). The video information about the streamer can be obtained by a photographing device electrically connected to the streamer terminal (130), and in some embodiments, the video information can be sampled in response to a preset sampling period, and each of the plurality of sampled images can be sequentially arranged in time order to form an image information set.

[0071] In step S420, the electronic device (200) according to an embodiment of the present disclosure may perform preprocessing on image information (or a set of image information). In the embodiment, the preprocessing on the image information may be performed in accordance with specifications set for the electronic device (200) and may include at least some of the processes of resizing, normalization, and binary conversion.

[0072] At step S430, the electronic device (200) according to an embodiment of the present disclosure may extract landmarks for identifying the streamer's facial expressions and movements based on image information. In the embodiment, the landmarks may refer to a set of two-dimensional or three-dimensional reference points extracted to identify the streamer's facial expressions and movements, and may be derived based on image information input to an artificial intelligence model. Specific details regarding the landmarks derived by the electronic device (200) will be described in detail with reference to FIGS. 5A and 5B , which will be described later.

[0073] In step S440, the electronic device (200) according to the embodiment of the present disclosure can determine an expression to be reflected on a virtual avatar corresponding to the streamer based on the landmark extracted in step S430. Specifically, the electronic device (200) can determine an expression to be applied to the virtual avatar based on first landmark information derived to identify the streamer's expression, and can determine a movement to be applied to the virtual avatar based on second landmark information derived to identify the streamer's movement. The method for determining an expression by the electronic device (200) will be described in detail with reference to FIG. 6 below.

[0074] In step S450, the electronic device (200) according to an embodiment of the present disclosure can generate content based on the expression information identified in step S440. In the embodiment, the electronic device (200) can generate a virtual avatar to which the streamer's expressions and movements are applied, and can generate content to be provided to the viewer based on the generated virtual avatar. According to the embodiment, since the electronic device (200) derives expression information based on video information (or a set of image information), the virtual avatar generated based on the derived expression information can reflect changes in the streamer's expression over time. Accordingly, the electronic device (200) can implement a vivid virtual avatar by reflecting the expressions and movements of the actual streamer.

[0075] In step S460, the electronic device (200) according to an embodiment of the present disclosure may provide the content information generated in step S450 to an external server (e.g., a second electronic device (120, see FIG. 1)) for providing a media service. The external server may store the content information obtained from the electronic device (200) and provide the content to a viewer terminal (140, see FIG. 1).

[0076] FIG. 5A is a diagram illustrating a process of extracting landmarks related to a streamer's facial expression according to an embodiment of the present disclosure.

[0077] FIG. 5A (A) illustrates image information for a streamer provided to an electronic device (200, see FIG. 2), and the electronic device (200) can extract a landmark (LM) related to the streamer's expression corresponding to the streamer's face shown in the image information for the streamer as illustrated in FIG. 5A (B). In an embodiment, the electronic device (200) can extract a landmark (LM) related to the streamer's expression using a preset algorithm or an artificial intelligence model. When the streamer's expression is identified through the artificial intelligence model, the artificial intelligence model can be trained to extract a landmark (LM) composed of a plurality of reference points for identifying the facial shape and expression from a facial image, and a plurality of facial images and landmark information corresponding to each of the plurality of facial images can be used as learning data for training the artificial intelligence model. In one embodiment, a landmark (LM) for identifying a streamer's facial expression may be composed of 476 reference points, and the streamer's facial expression may be identified based on the landmark derived from the electronic device (200). A method for identifying facial expressions by the electronic device (200) will be described in detail with reference to FIG. 6 below.

[0078] FIG. 5b is a diagram illustrating a process of extracting landmarks related to the movement of a streamer according to an embodiment of the present disclosure.

[0079] FIG. 5B is a diagram for explaining a process of identifying hand movement by an electronic device (200, see FIG. 2), wherein the electronic device (200) recognizes the hand of the streamer shown in image information about the streamer, and extracts a landmark for identifying the movement of the streamer's hand through tracking the recognized hand. In an embodiment, the electronic device (200) may extract a landmark (LM) related to the movement of the streamer's hand using a preset algorithm or an artificial intelligence model, and in some embodiments, the landmark (LM) related to the movement of the hand may be set to be derived corresponding to a joint position located in the hand recognized from the image information about the streamer. In one embodiment, the landmark (LM) for identifying the movement of the streamer's hand may be composed of 21 reference points for one hand (42 reference points may be derived for both hands), and the movement of the streamer's hand may be identified based on the landmark (LM) derived from the electronic device (200). According to an embodiment of the present disclosure, the electronic device (200) can identify the head movement, body movement, etc. of the streamer in addition to the hand movement by the same principle, and a description related thereto is omitted.

[0080] FIG. 6 is a diagram for explaining a process for determining an expression implemented in a virtual avatar according to an embodiment of the present disclosure.

[0081] FIG. 6 illustrates a table for determining an expression based on a landmark derived from a face of a streamer derived from an electronic device (200, see FIG. 2). The table illustrated in FIG. 6 may be predefined and stored for the electronic device (200). Specifically, a table in which a plurality of expression processing items are defined may be stored for the electronic device (200), and the table illustrated in FIG. 6 includes a plurality of expression processing items related to the expression of the streamer. The electronic device (200) may derive a weight for each of the plurality of expression processing items based on the extracted landmarks, and may determine expression information expressed in a virtual avatar based on the weight derived for each item. The weight assigned to each item may be a value in the range of 0 to 1, and a weight of 0 may mean a neutral state in which there is no movement corresponding to the item, and a weight of 1 may mean a state in which the movement corresponding to the item is maximum. Meanwhile, the plurality of expression processing items illustrated in FIG. 6 are according to one embodiment of the present disclosure, and the number and properties of the plurality of expression processing items stored in advance for the electronic device (200) according to the present disclosure are not limited to the items illustrated in FIG. 6.

[0082] FIG. 7a is a drawing for explaining an implementation aspect of a virtual avatar according to a streamer's facial expression according to an embodiment of the present disclosure.

[0083] Referring to FIG. 7A, it is possible to confirm the expression implementation aspect (DATA_OUT) of a virtual avatar generated in response to the video information (or image information) (DATA_IN) of a streamer input to an electronic device (200, see FIG. 2). In the first case (CASE 1), when video information of a streamer with a smiling expression is input, the electronic device (200) can derive a landmark corresponding to the streamer's face through the above-described process, and can generate a virtual avatar based on the facial expression information confirmed based on the derived landmark. Meanwhile, it can be confirmed that a virtual avatar reflecting the streamer's expression is generated in the second case (CASE 2) and the third case (CASE 3).

[0084] FIG. 7b is a drawing for explaining an implementation aspect of a virtual avatar according to the movement of a streamer according to an embodiment of the present disclosure.

[0085] Referring to FIG. 7b, it is possible to confirm the movement implementation aspect (DATA_OUT) of the virtual avatar generated in response to the video information (or image information (DATA_IN)) of the streamer input to the electronic device (200, see FIG. 2). In the first case (CASE 1), when video information of a streamer with two fingers spread out is input, the electronic device (200) can derive a landmark corresponding to the movement of the streamer through the above-described process, and can generate a virtual avatar based on the movement information confirmed based on the derived landmark. Meanwhile, it can be confirmed that a virtual avatar reflecting the movement of the streamer is generated in the second case (CASE2) and the third case (CASE 3).

[0086] FIG. 8 is a block diagram illustrating an electronic device (800) according to an embodiment of the present disclosure.

[0087] The electronic device (800) illustrated in FIG. 8 may correspond to the first electronic device (110, see FIG. 1) of FIG. 1 described above. Referring to FIG. 8, the electronic device (800) according to an embodiment of the present disclosure may include a transceiver (810), a processor (820), and a memory (830).

[0088] The electronic device (800) can be connected to at least one of an external terminal and an external device via a transceiver (810) and exchange data. For example, the electronic device (800) can be connected to a streamer terminal (130, see FIG. 1) via the transceiver (810).

[0089] The processor (820) can perform at least one operation by the device and at least one method described through the above-described FIGS. 1 to 7B. In addition, the processor (820) can execute a program for performing at least one operation by the device and at least one method described through the above-described FIGS. 1 to 7B, and can process information and control the electronic device (800) to perform at least one operation by the device and at least one method described through the above-described FIGS. 1 to 7B.

[0090] The memory (830) can store information for performing operations and at least one method by at least one device described through FIGS. 1 to 7B. Furthermore, the memory (830) can store the code of a program executed by the processor (820). In an embodiment, the memory (830) can be a volatile memory or a non-volatile memory.

[0091] The above-described embodiments are specific examples for implementing the present disclosure. The present disclosure will encompass not only the above-described embodiments, but also embodiments that can be simply designed or easily modified. Furthermore, the present disclosure will encompass techniques that can be easily modified and implemented using the above-described embodiments. Therefore, the scope of the present disclosure should not be limited to the above-described embodiments, but should be defined not only by the claims set forth below, but also by equivalents of the claims of the present disclosure.

[0092] Meanwhile, this disclosure is derived from research conducted as part of the Korea SMEs and Startups Agency's MAC L2E Model-based virtual YouTuber real-time facial expression and motion recognition rendering solution (Project No.: RS-2024-00437589, Project Management (Specialist) Agency Name: Korea SMEs and Startups Agency, Research Project Name: Startup Growth Technology Development (R&D), Project Performing Agency Name: Over the Hand Co., Ltd., Research Period: 2024. 07.01~2026.06.30). There is no financial interest in the Korean government or the project provider in any aspect of this disclosure.

Claims

1. In a method of operating an electronic device for generating a virtual avatar, A step of obtaining video information about a streamer; A step of extracting a landmark for at least one of the streamer's movements and expressions based on the above video information; A step of checking expression information about the streamer based on the landmark; and An operating method comprising a step of generating a virtual avatar corresponding to the streamer based on the above expression information.

2. In paragraph 1, The step of extracting the above landmark includes a step of performing a preprocessing process on the image information, An operating method in which the above preprocessing process includes at least some of resizing, normalization, and binary translation processes for the image information.

3. In paragraph 1, A method of extracting a landmark for the movement of the streamer based on the above image information, wherein the landmark is composed of a plurality of reference points corresponding to joint positions recognized from an image of a body part of the streamer included in the above image information.

4. In paragraph 1, A method of extracting a landmark for the streamer's facial expression based on the above video information, wherein the landmark is composed of a plurality of reference points derived in correspondence to the streamer's facial image included in the above video information through an artificial intelligence model.

5. In paragraph 1, The steps to extract the above landmarks are: A step of sampling multiple images in response to a preset cycle based on the above image information; A step of arranging the above multiple images in chronological order to form an image information set; and An operating method comprising a step of extracting a landmark for at least one of the streamer's movements and expressions based on the set of image information.

6. In paragraph 1, The step of checking expression information for the above streamer includes the step of assigning a weight to each of a plurality of expression processing items based on the landmark, The above virtual avatar is a motion method in which at least one of facial expressions and movements derived based on weights for each of the plurality of expression processing items is implemented.

7. In paragraph 1, A method of operation further comprising the step of generating content based on the virtual avatar.

8. In paragraph 1, The video information for the above streamer is an operation method derived through fusion of each video data acquired from multiple shooting devices.

9. An electronic device for creating a virtual avatar, It includes a transceiver, a memory for storing instructions, and a processor, The processor connected to the transceiver and the memory, Obtain video information about the streamer, Based on the above video information, a landmark for at least one of the streamer's movements and expressions is extracted, Check the expression information about the streamer based on the above landmark, An electronic device that generates a virtual avatar corresponding to the streamer based on the above expression information.

10. A computer program stored in a computer-readable storage medium for executing a method of creating a virtual avatar, in combination with hardware, How to create the above virtual avatar: A step of obtaining video information about a streamer; A step of extracting a landmark for at least one of the streamer's movements and expressions based on the above video information; A step of checking expression information about the streamer based on the landmark; and A computer program comprising a step of generating a virtual avatar corresponding to the streamer based on the above expression information.

Images