Electronic device and method for displaying avatar in virtual environment

By leveraging the processing speeds of multiple circuits within electronic devices to process voice information and generate mouth shapes, the method addresses synchronization delays and resource inefficiencies, improving user experience and immersion in virtual environments.

WO2024228472A1PCT designated stage expired Publication Date: 2024-11-07SAMSUNG ELECTRONICS CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/003888
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-13
Filing Date
2024-03-27
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

Current electronic devices face challenges in accurately and efficiently generating an avatar's mouth shape in real-time within a virtual environment, leading to synchronization delays and resource inefficiencies, which affect user experience and immersion.

Method used

The electronic device employs a method to identify and utilize the processing speeds of multiple circuits, such as CPUs and NPUs, to efficiently process voice information, enhance speech features, and generate mouth shapes, allowing for real-time avatar animation and reduced resource usage by optimizing computation on-device.

Benefits of technology

This approach enhances the accuracy and speed of mouth shape generation, reducing synchronization delays and improving user experience by efficiently utilizing device resources, while minimizing reliance on external resources for real-time avatar rendering.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024003888_07112024_PF_FP_ABST
    Figure KR2024003888_07112024_PF_FP_ABST
Patent Text Reader

Abstract

This electronic device may comprise a display, a memory for storing instructions, and at least one processor. The instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to: with regard to feature value identification of voice data, identify a first processing speed of each of a plurality of processing circuits for processing the voice data; with regard to mouth shape identification of the voice data, identify a second processing speed of each of the plurality of processing circuits; obtain voice information from the outside of the electronic device while displaying an avatar; obtain a plurality of feature values of the voice information by using a first processing circuit identified on the basis of the first processing speed; obtain information for generating mouth shapes on the basis of the plurality of feature values, by using a second processing circuit identified on the basis of the second processing speed; and display, through the display, the avatar including the mouth shapes generated on the basis of the information.
Need to check novelty before this filing date? Find Prior Art

Description

Electronic device and method for displaying an avatar in a virtual environment

[0001] The descriptions below relate to electronic devices and methods for displaying avatars within a virtual environment.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services that display computer-generated information in conjunction with external objects in the real world. The AR may be referred to as a virtual environment. The electronic devices may include wearable devices that can be worn by a user. For example, the electronic devices may include user equipment, AR glasses, and / or a head-mounted device (HMD).

[0003] An electronic device may include a display. The electronic device may include at least one processor. The at least one processor may be configured to identify a first processing speed of each of a plurality of processing circuits for processing voice data in relation to feature value identification of the voice data. The at least one processor may be configured to identify a second processing speed of each of the plurality of processing circuits in relation to mouth shape identification of the voice data, which is linked to the feature value identification. The at least one processor may be configured to acquire voice information from an external source of the electronic device while displaying an avatar. The at least one processor may be configured to acquire a plurality of feature values ​​of the voice information using a first processing circuit among the plurality of processing circuits, the first processing circuit being identified based on the first processing speed. The at least one processor may be configured to obtain information for generating a mouth shape based on the plurality of feature values ​​using a second processing circuit identified based on the second processing speed among the plurality of processing circuits. The at least one processor may be configured to display the avatar including the mouth shape generated based on the information through the display. One or more programs including instructions causing the avatar to be displayed through the display within the space may be stored.

[0004] As described above, a method performed by an electronic device may include an operation of identifying a first processing speed of each of a plurality of processing circuits for processing voice data in relation to identifying a feature value of the voice data. The method may include an operation of identifying a second processing speed of each of the plurality of processing circuits in relation to identifying a mouth shape of the voice data, which is linked to identifying the feature value. The method may include an operation of acquiring voice information from an external source of the electronic device while displaying an avatar. The method may include an operation of acquiring a plurality of feature values ​​of the voice information using a first processing circuit among the plurality of processing circuits, which is identified based on the first processing speed. The method may include an operation of acquiring information for generating a mouth shape based on the plurality of feature values, using a second processing circuit among the plurality of processing circuits, which is identified based on the second processing speed. The method may include an operation of displaying the avatar, which includes the mouth shape generated based on the information, through a display of the electronic device.

[0005] The non-transitory computer-readable storage medium as described above may store one or more programs including instructions that, when executed by at least one processor of an electronic device (101) including a display, cause the electronic device to identify a first processing speed of each of a plurality of processing circuits for processing voice data with respect to feature value identification of the voice data. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the at least one processor, cause the electronic device to identify a second processing speed of each of the plurality of processing circuits with respect to mouth shape identification of the voice data, which is linked to the feature value identification. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the at least one processor, cause the electronic device to acquire voice information from an external source while displaying an avatar. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the at least one processor, cause a first processing circuit among the plurality of processing circuits to be identified based on the first processing speed to obtain a plurality of feature values ​​of the speech information. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the at least one processor, cause a second processing circuit among the plurality of processing circuits to be identified based on the second processing speed to obtain information for generating a mouth shape based on the plurality of feature values.The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the at least one processor, cause the avatar, including the mouth shape generated based on the information, to be displayed through the display.

[0006] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0007] FIG. 2A illustrates an example of a perspective view of a wearable device according to various embodiments.

[0008] FIG. 2b illustrates an example of one or more hardware elements arranged within a wearable device according to various embodiments.

[0009] FIGS. 3A and 3B illustrate examples of the appearance of a wearable device according to various embodiments.

[0010] FIG. 4 illustrates an example of a method for identifying a mouth shape of an avatar corresponding to a user according to various embodiments.

[0011] FIG. 5 illustrates an exemplary block diagram of an electronic device according to various embodiments.

[0012] FIGS. 6A and 6B illustrate examples of flowcharts for a method for identifying a mouth shape of an avatar in a virtual environment according to various embodiments.

[0013] FIG. 7 illustrates an example of a delay time between the timing for acquiring voice information and the timing for playing the voice information according to various embodiments.

[0014] FIG. 8a illustrates an example of a band pass filter (BPF) for removing noise from voice information according to various embodiments.

[0015] FIGS. 8B and 8C illustrate examples of methods for enhancing voice from speech information according to various embodiments.

[0016] FIG. 8d illustrates an example of normalizing the volume of voice information according to various embodiments.

[0017] FIG. 9 illustrates an example of a block diagram for obtaining feature values ​​of voice information according to various embodiments.

[0018] FIG. 10 illustrates an example of a method for obtaining information for generating a mouth shape based on voice information according to various embodiments.

[0019] FIG. 11 illustrates examples of a method for generating animation for an avatar including a mouth shape according to various embodiments.

[0020] FIGS. 12A through 12C illustrate examples of how to play an animation for an avatar including a mouth shape according to various embodiments.

[0021] FIG. 13 illustrates an example flowchart of a method for applying a specified mouth shape to an avatar including a closed mouth shape according to various embodiments.

[0022] FIG. 14 illustrates an example flowchart of a method for identifying an avatar's mouth shape based on the performance of multiple processing circuits according to various embodiments.

[0023] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0024] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0025] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled. However, this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." Conditions described as "more than" may be replaced with "more than," conditions described as "less than," and conditions described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of the elements from A (including A) to B (including B).

[0026] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0027] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0028] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0029] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0030] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0031] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0032] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0033] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0034] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0035] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0036] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0037] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0038] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0039] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0040] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0041] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0042] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0043] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0044] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0045] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0046] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0047] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0048] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0049] FIG. 2A illustrates an example of a perspective view of a wearable device according to various embodiments. FIG. 2B illustrates an example of one or more hardware components arranged within a wearable device according to various embodiments.

[0050] According to one embodiment, a wearable device (101-1) may have a form of glasses that can be worn on a body part of a user (e.g., head). The wearable device (101-1) of FIGS. 2A and 2B may be an example of the electronic device (101) of FIG. 1. The wearable device (101-1) may include a head mounted display (HMD). For example, the housing of the wearable device (101-1) may include a flexible material, such as rubber and / or silicone, that is configured to adhere closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (101-1) may include one or more straps that can be twined around the user's head, and / or one or more temples that can be detachably attached to the ears of the head.

[0051] Referring to FIG. 2A, according to one embodiment, a wearable device (101-1) may include at least one display (250) and a frame (200) supporting at least one display (250).

[0052] According to one embodiment, a wearable device (101-1) may be worn on a part of a user's body. The wearable device (101-1) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (101-1). For example, the wearable device (101-1) may display a virtual reality image provided from at least one optical device (282, 284) of FIG. 2B on at least one display (250) in response to a user's designated gesture acquired through the motion recognition cameras (260-2, 264) of FIG. 2B. For example, the display (250) may include at least a part of the display module (160) of FIG. 1.

[0053] According to one embodiment, at least one display (250) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0054] Referring to FIG. 2B, at least one display (250) can provide the user with visual information transmitted from external light and other visual information distinct from the visual information through a lens included in the at least one display (250). The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (250) can include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area can be formed on the second surface (232) of the at least one display (250). When a user wears the wearable device (101-1), external light can be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (282, 284) on a real screen transmitted through external light, in a display area formed on the second surface (232).

[0055] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits the diffracted light to a user. The at least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (233, 234). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (233, 234) may be propagated to the other end of the at least one waveguide (233, 234) by the nano-pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be arranged within the wearable device (101-1) to guide a screen displayed by at least one display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (233, 234).

[0056] The wearable device (101-1) can analyze an object included in a real image collected through a shooting camera (245), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (101-1) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (101-1) can execute SLAM (simultaneous localization and mapping) using a multi-camera, an IMU (inertial measurement units) (or an IMU sensor), and / or a ToF (time-of-flight). A user wearing the wearable device (101-1) can view an image displayed on at least one display (250).

[0057] According to one embodiment, the frame (200) may be configured as a physical structure that allows the wearable device (101-1) to be worn on the user's body. According to one embodiment, the frame (200) may be configured so that, when the user wears the wearable device (101-1), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0058] Referring to FIG. 2A, the frame (200) may include a region (220) that is in contact with at least a portion of a user's body when the user wears the wearable device (101-1). For example, the region (220) of the frame (200) that is in contact with a portion of the user's body may include a region that is in contact with a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (101-1) makes contact with. According to one embodiment, the frame (200) may include a nose pad (210) that is in contact with a portion of the user's body. When the wearable device (101-1) is worn by the user, the nose pad (210) may be in contact with a portion of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that are in contact with another portion of the user's body that is distinct from the portion of the user's body.

[0059] For example, the frame (200) may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of the wearer's ear, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of the ear opposite the ear. The first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to one embodiment, the wearable device (101-1) can identify an external object (e.g., a user's fingertip) touching the frame (200) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (200).

[0060] According to one embodiment, the wearable device (101-1) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 5). For example, the hardwares may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers 255-1, 255-2), a microphone (e.g., microphones 265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (290) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (200).

[0061] According to one embodiment, the microphones (e.g., microphones 265-1, 265-2, 265-3) of the wearable device (101-1) may be disposed on at least a portion of the frame (200) to acquire sound signals. A first microphone (265-1) disposed on the bridge (203), a second microphone (265-2) disposed on the second rim (202), and a third microphone (265-3) disposed on the first rim (201) are illustrated in FIG. 2B, but the number and arrangement of the microphones (265) are not limited to the embodiment of FIG. 2B. When the number of microphones (265) included in the wearable device (101-1) is two or more, the wearable device (101-1) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame (200).

[0062] According to one embodiment, at least one optical device (282, 284) can project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) can be a projector. At least one optical device (282, 284) can be disposed adjacent to at least one display (250) or can be included within at least one display (250) as a part of at least one display (250). According to one embodiment, the wearable device (101-1) can include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) disposed at an edge of a first display (250-1) and a second optical device (284) disposed at an edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) disposed on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) disposed on the second display (250-2).

[0063] In one embodiment, the camera (260) may include a recording camera (245), an eye tracking camera (ET camera) (260-1), and / or a motion recognition camera (260-2). The recording camera (245), the eye tracking camera (260-1), and the motion recognition cameras (260-2, 264) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data representing the gaze of a user wearing the wearable device (101-1). For example, the wearable device (101-1) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (260-1). An example in which the gaze tracking camera (260-1) is positioned toward the user's right eye is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (260-1) may be positioned solely toward the user's left eye, or may be positioned toward both eyes.

[0064] In one embodiment, the capturing camera (245) can capture an actual image or background to be aligned with a virtual image in order to implement augmented reality or mixed reality content. The capturing camera (245) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information about the actual image or background including the image of the specific object acquired using the capturing camera (245) is superimposed with a virtual image provided through at least one optical device (282, 284). In one embodiment, the capturing camera (245) can be placed on a bridge (203) disposed between the first rim (201) and the second rim (202).

[0065] The gaze tracking camera (260-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (101-1) and thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (101-1) looks straight ahead, the wearable device (101-1) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (260-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (260-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (260-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (101-1) is positioned.

[0066] The motion recognition camera (260-2, 264) can provide a specific event on a screen provided on at least one display (250) by recognizing the movement of the user's entire body, such as the user's torso, hands, or face, or a part of the body. The motion recognition camera (260-2, 264) can recognize the user's motion (gesture recognition), obtain a signal corresponding to the motion, and provide a display corresponding to the signal on at least one display (250). The processor can identify the signal corresponding to the motion and perform a designated function based on the identification. In one embodiment, the motion recognition camera (260-2, 264) can be disposed on the first limb (201) and / or the second limb (202).

[0067] The camera (260) included in the wearable device (101-1) is not limited to the above-described gaze tracking camera (260-1) and motion recognition cameras (260-2, 264). For example, the wearable device (101-1) can identify an external object included in the FoV using the camera (260) positioned toward the user's FoV. The wearable device (101-1) can identify an external object based on a sensor for identifying the distance between the wearable device (101-1) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (101-1) may include a camera (260) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (101-1).

[0068] Although not shown, in one embodiment, the wearable device (101-1) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (260). The light source may include an LED of an infrared wavelength. The light source may be disposed on at least one of the frame (200) and the hinge units (206, 207).

[0069] According to one embodiment, the battery module (270) may supply power to electronic components of the wearable device (101-1). In one embodiment, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be disposed at an end of the first temple (204) and / or the second temple (205).

[0070] The antenna module (275) can transmit signals or power to the outside of the wearable device (101-1), or receive signals or power from the outside. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0071] The speaker (255) can output an audio signal to the outside of the wearable device (101-1). The audio output module may be referred to as a speaker. In one embodiment, the speaker (255) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ear of a user wearing the wearable device (101-1). For example, the speaker (255) may include a second speaker (255-2) positioned within the first temple (204) and thus adjacent to the user's left ear, and a first speaker (255-1) positioned within the second temple (205) and thus adjacent to the user's right ear.

[0072] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (101-1) to the user. For example, when the wearable device (101-1) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0073] Referring to FIG. 2B, according to one embodiment, a wearable device (101-1) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) or the second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (101-1) (e.g., hardwares illustrated by different blocks in FIG. 5) may be disposed on the PCB (290). The wearable device (101-1) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0074] According to one embodiment, a wearable device (101-1) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (101-1) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (101-1). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (101-1) may identify a user's motion and / or gesture performed to execute or stop a specific function of the wearable device (101-1) based on the IMU.

[0075] FIGS. 3A and 3B illustrate examples of the appearance of a wearable device according to various embodiments.

[0076] The wearable device (101-1) of FIGS. 3A and 3B may be an example of the electronic device (101) of FIG. 1. According to one embodiment, an example of the appearance of a first side (310) of a housing of the wearable device (101-1) may be illustrated in FIG. 3A, and an example of the appearance of a second side (320) opposite to the first side (310) may be illustrated in FIG. 3B.

[0077] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (101-1) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (101-1) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the two eyes, may be disposed on the first surface (310). The wearable device (101-1) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0078] According to one embodiment, the wearable device (101-1) may include cameras (260-3, 260-4) for photographing and / or tracking the user's two eyes adjacent to the first display (250-1) and the second display (250-2), respectively. For example, the cameras (260-3, 260-4) may be referred to as ET cameras. According to one embodiment, the wearable device (101-1) may include cameras (260-5, 260-6) for photographing and / or recognizing the user's face. The cameras (260-5, 260-6) may be referred to as FT cameras.

[0079] Referring to FIG. 3B, a camera (e.g., cameras 260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., a depth sensor 330) for obtaining information related to the external environment of the wearable device (101-1) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3A. For example, the cameras (260-7, 260-8, 260-9, 260-10) may be disposed on the second surface (320) to recognize external objects. For example, using the cameras (260-11, 260-12), the wearable device (101-1) may obtain images and / or videos to be transmitted to each of the user's two eyes. The camera (260-11) may be placed on the second surface (320) of the wearable device (101-1) to obtain an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (260-12) may be placed on the second surface (320) of the wearable device (101-1) to obtain an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes.

[0080] According to one embodiment, the wearable device (101-1) may include a depth sensor (330) disposed on the second surface (320) to identify a distance between the wearable device (101-1) and an external object. Using the depth sensor (330), the wearable device (101-1) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (101-1).

[0081] Although not shown, a microphone may be placed on the second side (320) of the wearable device (101-1) to acquire sound output from an external object. The number of microphones may be one or more depending on the embodiment.

[0082] As described above, according to one embodiment, the wearable device (101-1) may include hardware (e.g., cameras (260-7, 206-8, 260-9, 260-10) and / or a depth sensor (330)) for identifying a body part including a user's hand. The wearable device (101-1) may identify a gesture indicated by a motion of the body part. The wearable device (101-1) may provide a user interface (UI) based on the identified gesture to a user wearing the wearable device (101-1). The UI may support a function for editing images and / or videos stored in the wearable device (101-1). The wearable device (101-1) may communicate with an external electronic device different from the wearable device (101-1) in order to more accurately identify the gesture.

[0083] FIG. 4 illustrates an example of a method for identifying a mouth shape of an avatar corresponding to a user according to various embodiments.

[0084] The avatar may represent an avatar corresponding to the user within a virtual environment provided by the electronic device (101) of FIG. 1. For example, the user may be a user of the electronic device (101). The virtual environment may represent an example of extended reality (XR) provided through the electronic device (101). For example, XR may include augmented reality (AR), virtual reality (VR), and mixed reality (MR). For example, the electronic device (101) for AR may provide augmented information based on real objects. For example, the electronic device (101) may include AR glasses or VR glasses for providing information to the user based on real objects. For example, the electronic device (101) may include a video see-through (VST) device. For example, the electronic device (101) may include a terminal (user equipment). For example, the electronic device (101) may include a personal computer (PC). Hereinafter, the electronic device (101) may also be referred to as a wearable device (e.g., the wearable device (101-1) of FIGS. 2A to 3B).

[0085] The mouth shape may represent a visual object of a portion of the face of the avatar. For example, the portion may represent an area of ​​the face of the avatar where the mouth (mouse) is located. For example, the mouth shape may be identified by the position and shape of at least one of the mouth, lips, or teeth of the mouth. However, the embodiments of the present disclosure are not limited thereto. For example, the mouth shape may include a shape of a portion of the face of the avatar that may change according to the movement of muscles or joints of the face of the avatar based on at least one syllable uttered by the user.

[0086] Referring to FIG. 4, an example of a user (400) making a video call with another user through an electronic device (101) is illustrated. For example, the user (400) may use a video call service by using an avatar (410) corresponding to the user (400) within the virtual environment to converse with another avatar (450) corresponding to the other user. For example, the electronic device (101) may display an avatar (410) corresponding to the user (400) within the virtual environment and another avatar (450) corresponding to the other user through a display (e.g., a display module (160)). For example, the virtual environment may be connected to the electronic device (101) and an external electronic device (not shown) of the other user.

[0087] Referring to FIG. 4, the electronic device (101) can generate an avatar (410) by acquiring information about the user (400). For example, the electronic device (101) can generate the appearance and movement of the avatar (410) by acquiring information about the appearance and movement of the user (400). For example, the electronic device (101) can acquire information about the appearance and movement of the user (400) through a camera of the electronic device (101) (e.g., a camera module (180)). In addition, the electronic device (101) can generate the mouth shape of the avatar (410) and the voice to be spoken by the avatar (410) by acquiring voice information acquired when the user (400) speaks. For example, the electronic device (101) can acquire the voice information through a microphone (e.g., an input module (150)). However, the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may obtain information (or voice information) about the mouth shape of the avatar (410) and the voice to be spoken by the avatar (410) as input in the form of text. In addition, for example, the electronic device (101) may obtain information (or voice information) about the mouth shape of the avatar (410) and the voice to be spoken by the avatar (410) through a server (or system) providing the virtual environment. In the example of FIG. 5, the electronic device (101) may obtain information about the mouth shape and the voice to be spoken by another avatar (450) of the other user, through information input to the external electronic device of the other user.

[0088] As described above, the electronic device (101) may acquire voice information from the user (400) or the other user. At this time, the electronic device (101) may acquire and process the voice information from the user (400) or the other user, thereby generating an avatar (410) or another avatar (450). At this time, a difference may occur between the time at which the voice information is acquired (or the time at which the user (400) or the other user utters the voice included in the voice information) and the time at which the avatar (410) or the other avatar (450) is generated. This difference may be referred to as a delay time. For example, even though the user (400) is uttering a voice with his / her mouth open, the avatar (410) in the virtual environment may have a mouth shape that does not yet utter a voice. In other words, synchronization between the user (400) and the avatar (410) may not be correct. For example, the above synchronization may be referred to as lip sync, which is synchronization for changing mouth shapes in real time. For example, the lip sync may be caused by a delay time for the electronic device (101) to process information about the voice spoken by the user (400) and, based on this, generate an avatar (410) including the mouth shape (or an animation including an avatar (410) having the mouth shape).

[0089] As described above, methods for controlling lip sync for voices have been improved by increasing the accuracy of voice identification or enhancing the image quality of animations including avatars. However, these methods for increasing the accuracy of voice identification or image quality may be indirect solutions to lip sync problems, rather than directly reducing them.

[0090] Hereinafter, an electronic device and method optimized for each electronic device for generating an avatar based on real-time voice information according to an embodiment of the present disclosure are described. The electronic device and method according to an embodiment of the present disclosure can quickly and flexibly reduce lip sync even in an internal environment (or on-device environment) of the electronic device. In other words, the electronic device and method according to an embodiment of the present disclosure can quickly generate an avatar (or an avatar's mouth shape, or an animation including an avatar having a mouth shape) with higher accuracy by monitoring and efficiently utilizing resources within the electronic device. Accordingly, the electronic device and method according to an embodiment of the present disclosure can provide a more immersive user experience to the user. Furthermore, the electronic device and method according to an embodiment of the present disclosure can secure real-time performance even in a multi-tasking environment through optimized operations for generating an avatar having a mouth shape based on voice during the runtime of the electronic device. In addition, the electronic device and method according to the embodiment of the present disclosure can reduce the overall resource usage by utilizing only the resources of the electronic device itself (on-device) and not using the resources of the server providing the virtual environment and additional resources (e.g., data) for connection with the server.

[0091] FIG. 5 illustrates an exemplary block diagram of an electronic device according to various embodiments. The electronic device (101) of FIG. 5 may be an example of the electronic device (101) of FIG. 1 and the wearable device (101-1) of FIGS. 2A to 3B.

[0092] Referring to FIG. 5, an exemplary situation is illustrated in which an electronic device (101) and an external electronic device (580) are connected to each other based on a wired network and / or a wireless network. For example, the wired network may include a network such as the Internet, a local area network (LAN), a wide area network (WAN), or a combination thereof. For example, the wireless network may include a network such as long term evolution (LTE), 5g new radio (NR), wireless fidelity (WiFi), Zigbee, near field communication (NFC), Bluetooth, Bluetooth low-energy (BLE), or a combination thereof. Although the electronic device (101) and the external electronic device (580) are illustrated as being directly connected, the electronic device (101) and the external electronic device (580) may be indirectly connected via one or more routers and / or access points (APs). In other words, the electronic device (101) is exemplified as being directly connected to the communication circuit (590) of the external electronic device (580) through the communication circuit (520), but the embodiments of the present disclosure are not limited thereto.

[0093] Referring to FIG. 5, according to one embodiment, an electronic device (101) may include at least one of a processor (120), a memory (130), a display (510), and a communication circuit (520). The processor (120), the memory (130), the display (510), and the communication circuit (520) may be electrically and / or operably coupled with each other by a communication bus. Hereinafter, operably coupled hardware components may mean that a direct connection or an indirect connection is established between the hardware components, either wired or wireless, such that a second hardware component is controlled by a first hardware component among the hardware components. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the hardware components illustrated in FIG. 5 (e.g., at least a portion of the processor (120), the memory (130), and the communication circuit (520)) may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of hardware components included in the electronic device (101) is not limited to those illustrated in FIG. 5. For example, the electronic device (101) may include only some of the hardware components illustrated in FIG. 5.

[0094] According to one embodiment, the processor (120) of the electronic device (101) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), and a field programmable gate array (FPGA). As an example, the hardware component for processing data may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP), and / or a neural processing unit (NPU). The number of processors (120) may be one or more. For example, the processor (120) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core. The processor (120) of FIG. 5 may include at least a part of the processor (120) of FIG. 1.

[0095] For example, the processor (120) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits including at least one processor, one or more of which may be configured to perform the various functions described below individually or collectively in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms encompass, for example, and without limitation, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, and also situations where one processor may perform all of the recited functions. Additionally, the at least one processor may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. At least one processor may execute program instructions to achieve or perform the various functions.

[0096] According to one embodiment, the memory (130) of the electronic device (101) may include a hardware component for storing data and / or instructions input to and / or output from the processor (120). The memory (130) may include, for example, a volatile memory such as a random-access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multimedia card (eMMC). The memory (130) of FIG. 5 may include at least a portion of the memory (130) of FIG. 1.

[0097] According to one embodiment, a display (510) of an electronic device (101) may output visualized information (e.g., a screen of FIG. 4 or FIG. 10) to a user. The number of displays (510) included in the electronic device (101) may be one or more. For example, the display (510) may be controlled by a processor (120) and / or a graphic processing unit (GPU) (not shown) to output visualized information to a user. The display (510) may include a flat panel display (FPD) and / or electronic paper. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), a digital mirror device (DMD), one or more light emitting diodes (LEDs), and / or a micro LED. The LED may include an organic LED (OLED). The display (510) of FIG. 5 may include at least a portion of the display module (160) of FIG. 1.

[0098] In one embodiment, light transmission may occur in at least a portion of the display (510). The electronic device (101) may provide a user experience related to augmented reality by providing a combination of light output through the display (510) and light transmitting through the display (510) to the user. As described above with reference to FIGS. 2A and 2B and / or 3A and 3B , the display (510) of the electronic device (101) according to one embodiment may have a structure for covering the entire field-of-view (FoV) of the user, or radiating light toward the FoV, while being worn on a body part of the user, such as the head. Although not illustrated, the electronic device (101) may include other output means for outputting information in a form other than a visual or auditory form. For example, the electronic device (101) may include at least one speaker for outputting an audio signal, and / or a motor (or actuator) for providing haptic feedback based on vibration.

[0099] According to one embodiment, the communication circuit (520) of the electronic device (101) may include hardware for supporting transmission and / or reception of electrical signals between the electronic device (101) and an external electronic device (580). The communication circuit (520) may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit (520) may support transmission and / or reception of electrical signals based on various types of communication means, such as Ethernet, Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), and 5G new radio (NR). The communication circuit (520) of FIG. 5 may include at least a portion of the communication module (190) and / or the antenna module (197) of FIG. 1.

[0100] Although not illustrated, in one embodiment, the electronic device (101) may include output means for outputting information in a form other than a visualized form. For example, the electronic device (101) may include a speaker for outputting an acoustic signal. For example, the electronic device (101) may include a motor for providing haptic feedback based on vibration.

[0101] Referring to FIG. 5, according to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (120) of the electronic device (101) may be stored in the memory (130) of the electronic device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or an application. Hereinafter, when an application is installed in the electronic device (e.g., the electronic device (101)), it may mean that one or more instructions provided in the form of an application are stored in the memory (130), and that the one or more applications are stored in a format executable by the processor of the electronic device (e.g., a file having an extension designated by the operating system of the electronic device (101)). According to one embodiment, the electronic device (101) can perform the operations of FIGS. 6A and 6B, FIG. 13, and FIG. 14 by executing one or more instructions stored in the memory (130).

[0102] Referring to FIG. 5, one or more instructions included in the memory (130) may be divided into a processing circuit performance identifying portion (530), a voice information obtaining portion (540), a voice feature identifying portion (550), a mouth shape identifying portion (560), and / or an avatar generating portion (570). For example, each of the processing circuit performance identifying portion (530), the voice information obtaining portion (540), the voice feature identifying portion (550), the mouth shape identifying portion (560), and the avatar generating portion (570) may be implemented as a program or software.

[0103] For example, the electronic device (101) can obtain information about a plurality of processing circuits using the processing circuit performance identification unit (530). For example, the plurality of processing circuits may include a central processing unit (CPU), a graphic processing unit (GPU), or a neural processing unit (NPU). The plurality of processing circuits may represent circuits for processing voice information obtained through the voice information acquisition unit (540). For example, the information about the plurality of processing circuits may include at least one of information indicating whether the NPU or the GPU is included in the electronic device (101), or information indicating the manufacturer of the CPU. For example, the electronic device (101) can obtain information about the plurality of processing circuits based on the framework of the artificial intelligence model during the runtime of the artificial intelligence model. For example, the electronic device (101) can obtain information about the plurality of processing circuits that the framework of the artificial intelligence model can support. In other words, the electronic device (101) can obtain information about some of the processing circuits actually included that the framework can support. These some may be referred to as the plurality of processing circuits. However, the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) can obtain information about the plurality of processing circuits included in the electronic device (101) through a separate user interface of a software application for providing the virtual environment. For example, the information may be input by a user.

[0104] For example, the electronic device (101) can identify the performance of each of the plurality of processing circuits using the processing circuit performance identification unit (530). For example, the electronic device (101) can identify the processing speed of each of the plurality of processing circuits. At this time, the electronic device (101) can identify the processing speed for each processing algorithm for voice information processed by the plurality of processing circuits. For example, the electronic device (101) can identify the first processing speed of each of the plurality of processing circuits for the feature value identification described below. For example, the electronic device (101) can identify the second processing speed of each of the plurality of processing circuits for the lip shape identification described below. For example, the electronic device (101) can identify the third processing speed of each of the plurality of processing circuits for the voice portion enhancement described below. For example, the first processing speed can be identified by performing the feature value identification based on reference data in each of the plurality of processing circuits. For example, the second processing speed can be identified by performing the lip shape identification based on reference data in each of the plurality of processing circuits. For example, the third processing speed can be identified by performing the voice portion enhancement based on reference data in each of the plurality of processing circuits. The reference data can represent dummy data for identifying the performance of each of the plurality of processing circuits. For example, the first processing speed, the second processing speed, and the third processing speed can each be defined as a ratio of the processing time to the time length of input data (e.g., the length of the reference data). For example, the ratio can be referred to as a real time ratio (RT).For example, the first processing speed may include the processing speed of a CPU that performs the feature value identification using an artificial model, the processing speed of an NPU that performs the feature value identification using an artificial model, the processing speed of a GPU that performs the feature value identification using an artificial model, or the processing speed of a CPU that performs the feature value identification using an MFCC (mel frequency cepstral coefficient) algorithm.

[0105] For example, the electronic device (101) can update the performance of each of the plurality of processing circuits using the processing circuit performance identification unit (530). For example, the electronic device (101) can generate an avatar based on a processing circuit identified among the plurality of processing circuits based on a processing speed (e.g., the first processing speed, the second processing speed) for voice information acquired using the voice information acquisition unit (540) described below. At this time, the electronic device (101) can store the actual processing speed at which the voice information is processed based on the processing circuit. The actual processing speed and the processing speed (or expected processing speed) identified based on the reference data may be different from each other. This may be because the expected processing speed is the speed at which the reference data is processed, and the actual processing speed is the speed at which the voice information is processed, and thus different data is processed. In addition, this may be because the first time point (timing) at which the expected processing speed is measured and the second time point at which the actual processing speed for processing the voice information is identified are different. For example, at the first time point, the plurality of processing circuits may not be used. However, at the second time point, some of the plurality of processing circuits may be used for processing other than processing the voice information. Accordingly, the electronic device (101) may update the actual processing speed to the performance of the processing circuit for which the actual processing speed is measured. Based on the actual processing speed, the electronic device (101) may identify a processing circuit for voice information to be acquired in the future among the plurality of processing circuits. For example, the expected processing speed may be referred to as a processing speed predicted based on the reference data.

[0106] For example, the electronic device (101) can obtain voice information using the voice information acquisition unit (540). The voice information can be referred to as voice data. For example, the voice information can include voice, noise, or background sound. For example, the voice information can be obtained from outside the electronic device (101). For example, the voice information can be transmitted from an external electronic device (580) through a server or system for providing the virtual environment. For example, the voice information can be obtained through a microphone of the electronic device (101) as the user of the electronic device (101) speaks. For example, the voice information can include text input to the electronic device (101) or the external electronic device (580). For example, the text input can include machine-synthesized voice, such as text to speech (TTS). For example, the voice information may be composed of an entire utterance, a sentence, a word, or a specified length of a user of the electronic device (101) or another user of the external electronic device (580). For example, the specified length may be defined as a specified size (e.g., n bytes) or a specified length of time. For example, the voice information may be composed of a plurality of input signals. Each of the plurality of input signals may be composed of the specified length. Specific details related to the plurality of input signals constituting the voice information are described below in FIG. 7.

[0107] For example, the electronic device (101) may enhance voice features using the voice feature identification unit (550). For example, enhancing the voice features may include removing noise from the voice information, enhancing a voice portion of the voice information, and normalizing the volume of the voice portion. Enhancing the voice features may be referred to as voice enhancement. For example, the electronic device (101) may remove signals in the frequency domain identified as noise from the voice information using a band pass filter (BPF). Specific details related thereto are described below in FIG. 8A.

[0108] For example, the electronic device (101) can enhance the voice portion of the voice information from which the noise has been removed. For example, the electronic device (101) can enhance the voice portion using an artificial intelligence model (AI model). For example, the enhancement based on the AI ​​model can be performed based on a central processing unit (CPU), a graphic processing unit (GPU), or a neural processing unit (NPU) among a plurality of processing circuits of the electronic device (101). Alternatively, the electronic device (101) can enhance the voice portion using a computational algorithm. The computational algorithm can represent a non-AI model-based process. For example, the computational algorithm can include an improved minima controlled recursive averaging (IMCRA) algorithm or a log minimum mean square error (logMMSE) algorithm. For example, the enhancement based on the above-described operation algorithm may be performed based on a central processing unit (CPU) among the plurality of processing circuits of the electronic device (101). However, the embodiments of the present disclosure are not limited thereto, and the operation algorithm may include an algorithm capable of enhancing the voice portion from the voice information. In other words, the electronic device (101) may utilize the artificial intelligence model based on a CPU, GPU, or NPU among the plurality of processing circuits, or may utilize the operation algorithm based on the CPU. Specific details related thereto are described below in FIGS. 8B and 8C.

[0109] For example, the electronic device (101) may normalize the voice information in which the voice portion is enhanced. For example, the normalization of the voice information (or voice portion) may refer to adjusting the volume of the voice information. For example, the electronic device (101) may change the volume of the voice information to have a value within a specified range. The specified range may be set for a normalized input for an artificial intelligence model for processing the voice information. In other words, the artificial intelligence model may generate more accurate output based on the normalized input. In the above example, the specified range is exemplified as being set for the artificial intelligence model, but the embodiments of the present disclosure are not limited thereto. For example, when the voice information has a value within the specified range, a quantitative comparison between different voice information is possible, thereby simplifying the calculation. Specific details related thereto are described in FIG. 8D below.

[0110] For example, the electronic device (101) can obtain feature values ​​using the voice feature identification unit (550). For example, the electronic device (101) can extract feature values ​​from the voice information, which is an analog signal. For example, the electronic device (101) can obtain the feature values ​​based on the mel frequency cepstral coefficient (MFCC) algorithm. For example, the electronic device (101) can obtain a spectrum by applying a fast-Fourier transform (FFT) to each frame of the voice information. For example, the electronic device (101) can obtain the spectrum for the frequency domain by applying the FFT to the voice information. The electronic device (101) can obtain a mel spectrum by applying a mel filter bank to the spectrum. For example, the electronic device (101) can obtain the mel spectrum based on a mel scale that represents the relationship between the frequency domain and the low frequency band recognized by an actual person. The electronic device (101) can obtain MFCCs by applying cepstral analysis to the mel spectrum. The MFCCs can be referred to as the feature values. For example, the electronic device (101) can obtain the feature values ​​that are a part of the entire feature values, which are peaks obtained based on the cepstral analysis. The peaks can be referred to as formants. For example, the feature values ​​can be 40. However, the embodiments of the present disclosure are not limited thereto. For example, the number of feature values ​​can be less than 40 or greater than 40.

[0111] For example, the electronic device (101) can learn an artificial intelligence model based on the acquired feature values. In other words, the electronic device (101) can learn the artificial intelligence model by inputting the feature values. Accordingly, the electronic device (101) can obtain refined feature values. In the above-described example, the electronic device (101) obtains the feature values ​​based on the MFCC algorithm, and exemplifies a method of using the feature values ​​without additional processing or refining the feature values ​​using the artificial intelligence model, but the embodiments of the present disclosure are not limited thereto.

[0112] For example, the electronic device (101) may obtain the feature values ​​without the MFCC algorithm based on the voice information by using the artificial intelligence model. For example, if a processing circuit (e.g., NPU, GPU) having a relatively fast processing speed is available among the plurality of processing circuits of the electronic device (101), the electronic device (101) may obtain the feature values ​​by using the artificial intelligence model. Alternatively, if a processing circuit (e.g., CPU) having a relatively slow processing speed is available among the plurality of processing circuits of the electronic device (101), the electronic device (101) may obtain the feature values ​​by using the MFCC algorithm. Specific details related thereto are described below in FIG. 9.

[0113] As described above, the electronic device (101) may utilize the artificial intelligence model based on a plurality of processing circuits. For example, the plurality of processing circuits may include a CPU, a GPU, or an NPU. Additionally, the electronic device (101) may utilize the MFCC algorithm based on the CPU.

[0114] For example, the electronic device (101) may obtain information for generating a mouth shape for the voice information using the mouth shape identification unit (560). For example, the information for generating the mouth shape may include at least one of a visual phoneme (viseme), a face landmark, a blend weight, or a face mesh for the voice information. For example, the visual phoneme may represent a mouth shape symbol of an avatar indicating that the voice of the voice information is uttered. For example, the face landmark may represent coordinates for the face of an avatar indicating that the voice of the voice information is uttered. For example, the face landmark may include three-dimensional coordinates or two-dimensional coordinates. The blend weight may represent an emotion parameter for changing the facial expression of the avatar. For example, the blend weights can be obtained based on a retargeting model. For example, the blend weights can be obtained from the facial feature points or the voice information. For example, the face mesh can represent a mesh formed by the facial feature points. Specific details related to this are described below in FIG. 10.

[0115] For example, the electronic device (101) may obtain information for generating the mouth shape using an artificial intelligence model. For example, the electronic device (101) may utilize the artificial intelligence model based on a plurality of processing circuits. For example, the plurality of processing circuits may include a CPU, a GPU, or an NPU.

[0116] For example, the electronic device (101) can generate an avatar having the mouth shape using the avatar generation unit (570). For example, the electronic device (101) can obtain an avatar having the mouth shape based on information for generating the mouth shape obtained through the mouth shape identification unit (560). For example, the electronic device (101) can generate an animation including an avatar having the mouth shape. For example, the animation can display visual information including the virtual environment and the avatar for a time corresponding to a plurality of frames. For example, the plurality of frames can be referenced as playback frames set for the animation. The animation can include an avatar having the mouth shape of each of the plurality of frames. The plurality of frames can include key frames of a specified cycle. For example, the electronic device (101) can generate an avatar having the mouth shape for each of the plurality of frames, or generate an avatar having the mouth shape for each of the key frames. Specific details related to this are described in Figure 11 below.

[0117] For example, the electronic device (101) can display the generated avatar through the display (510) using the avatar generation unit (570). For example, the electronic device (101) can display the animation including the avatar through the display (510). In other words, the electronic device (101) can display the avatar or the animation including the avatar through the display (510). The electronic device (101) can play the avatar or the animation through the display (510). At this time, the electronic device (101) can change the playback speed, delete some of the playlist (content), or use a parallel processing method in order to minimize the delay time felt by the user (e.g., the user (400)). Specific details related thereto are described below in FIGS. 12A to 12C.

[0118] For example, the electronic device (101) may, using the avatar generation unit (570), identify whether the mouth of the currently displayed avatar is closed before displaying the avatar generated for the voice information. For example, the currently displayed avatar may be displayed as the electronic device (101) executes a software application that provides the virtual environment. The time before displaying the avatar may include the time before the electronic device (101) acquires the voice information and then processes the acquired voice information. For example, when the mouth is closed, the electronic device (101) may display the avatar having a designated mouth shape based on the volume of the voice information through the display (510). In other words, when the currently displayed avatar does not open its mouth and acquires voice information spoken by a user (e.g., user (400)), the avatar having a designated mouth shape based on the volume of the voice information may be displayed in order to reduce a delay that the user may experience. Specific details related to this are described in Figure 13 below.

[0119] FIGS. 6A and 6B illustrate examples of flowcharts for a method for identifying a mouth shape of an avatar in a virtual environment according to various embodiments.

[0120] At least some of the methods of FIGS. 6A and 6B may be performed by the electronic device (101) of FIG. 5. For example, at least some of the methods may be controlled by the processor (120) of the electronic device (101).

[0121] Referring to FIGS. 6A and 6B , in operation (610), the processor (120) may obtain information about a plurality of processing circuits. For example, the processor (120) may obtain information about the plurality of processing circuits related to generation of the mouth shape. For example, the plurality of processing circuits may include a central processing unit (CPU), a graphic processing unit (GPU), or a neural processing unit (NPU). The plurality of processing circuits may represent circuits for processing voice information. For example, the information about the plurality of processing circuits may include at least one of information indicating whether the NPU or the GPU is included in the electronic device (101), or information indicating a manufacturer of the CPU. The information indicating the manufacturer of the CPU may be referenced as information indicating a manufacturer of an application processor (AP). For example, the information indicating the manufacturer of the AP may include a software development kit (SDK). This is because when using the GPU or NPU within the AP, the SDK to be used may change depending on the manufacturer of the AP.

[0122] For example, the processor (120) may obtain information about the plurality of processing circuits based on the framework of the artificial intelligence model during the runtime of the artificial intelligence model. For example, the processor (120) may obtain information about the plurality of processing circuits that the framework of the artificial intelligence model can support. In other words, even if the electronic device (101) actually includes various processing circuits, it may obtain information about some of the various processing circuits that the framework can support. The some may be referred to as the plurality of processing circuits. However, the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may obtain information about the plurality of processing circuits that the electronic device (101) includes through a user interface of a software application for providing the virtual environment. For example, the information may be input by a user.

[0123] In operation (605), the processor (120) may identify the processing speed of each of the plurality of processing circuits. For example, the processor (120) may identify the processing speed of each of the plurality of processing circuits according to a processing algorithm for voice information. For example, the processing algorithm may include at least one of voice enhancement, feature value identification, lip shape identification, and avatar (or animation) generation and display for the voice information acquired from an external source.

[0124] For example, the processor (120) may identify the processing speed of each of the plurality of processing circuits using reference data for the processing algorithm. For example, the processor (120) may identify a first processing speed of each of the plurality of processing circuits for the feature value identification. For example, the processor (120) may identify a second processing speed of each of the plurality of processing circuits for the mouth shape identification. For example, the processor (120) may identify a third processing speed of each of the plurality of processing circuits for the voice enhancement. For example, the first processing speed may be identified by performing the feature value identification based on reference data in each of the plurality of processing circuits. For example, the second processing speed may be identified by performing the mouth shape identification based on reference data in each of the plurality of processing circuits. For example, the third processing speed may be identified by performing the voice enhancement based on reference data in each of the plurality of processing circuits. The above reference data may represent dummy data for identifying the performance of each of the plurality of processing circuits. For example, the first processing speed may include the processing speed of a CPU that performs the feature value identification using an artificial model, the processing speed of an NPU that performs the feature value identification using an artificial model, the processing speed of a GPU that performs the feature value identification using an artificial model, or the processing speed of a CPU that performs the feature value identification using an MFCC (mel frequency cepstral coefficient) algorithm (or a non-AI model-based processing algorithm).For example, each of the first processing speed, the second processing speed, and the third processing speed may be defined as a ratio of the processing time to the time length of the input data (e.g., the length of the reference data). For example, the ratio may be referred to as the real time ratio (RT). Specific examples related to this are provided in the table below.

[0125] Processing Algorithm Processing Circuit Real-time noise removal (non-AI based) CPU0.10RT Noise removal (non-AI based) GPU0.02RT Voice part enhancement (AI based) CPU0.13RT Voice part enhancement (AI based) GPU0.01RT Feature value identification (AI based) CPU0.03RT Feature value identification (AI based) GPU0.02RT Feature value identification (AI based) NPU0.01RT Feature value identification (non-AI based) CPU0.02RT Mouth shape identification (AI based) CPU0.13RT Mouth shape identification (AI based) GPU0.03RT Mouth shape identification (AI based) NPU0.01RT

[0126] Referring to the above table, the processor (120) can identify the processing speed (or real-time ratio) of each of the plurality of processing circuits for each processing algorithm. The processing algorithm may include AI-based algorithms and non-AI-based (non-AI model-based) algorithms. The non-AI-based algorithms may also be referred to as computational algorithms.

[0127] For example, if the length of the input information is 120 ms and the processing time for a specific processing algorithm through the CPU is 60 ms, the real-time ratio of the CPU for the specific processing algorithm may be 0.5 RT. Also, for example, if the length of the input information is 120 ms and the processing time for a specific processing algorithm through the GPU is 36 ms, the real-time ratio of the GPU for the specific processing algorithm may be 0.3 RT. Also, for example, if the length of the input information is 120 ms and the processing time for a specific processing algorithm through the NPU is 15 ms, the real-time ratio of the NPU for the specific processing algorithm may be 0.125 RT. For example, the processor (120) may identify a processing circuit for the specific processing algorithm based on a processing speed of each of the plurality of processing circuits. In the above example, the processor (120) can identify the NPU as a processing circuit for the specific algorithm, since the NPU has the smallest real-time ratio for the specific algorithm. Specific examples of a method for identifying the processing circuit are shown in the table below.

[0128] Processing AlgorithmProcessing CircuitReal-time Non-selective Noise Removal (Non-AI based)CPU0.10RTNoise Removal (Non-AI based)GPU0.02RTTOVoice Partial Enhancement (AI based)CPU0.13RTVoice Partial Enhancement (AI based)GPU0.01RTFeature Value Identification (AI based)CPU0.03RTFeature Value Identification (AI based)GPU0.02RTFeature Value Identification (AI based)NPU0.01RTOFeature Value Identification (Non-AI based)CPU0.02RTMouth Shape Identification (AI based)CPU0.13RTMouth Shape Identification (AI based)GPU0.03RTMouth Shape Identification (AI based)NPU0.01RTO

[0129] Referring to the above table, the processor (120) can select (or identify) a processing circuit having the shortest processing speed among the plurality of processing circuits for each processing algorithm.

[0130] For example, the processor (120) may change the length of the speech information, which is an input of the processing algorithm, in relation to the real-time ratio. For example, the processor (120) may change the length of the speech information when the real-time ratio has a value greater than or equal to 1.0RT. For example, when the length of the speech information is 120ms and the real-time ratio of the CPU for the specific algorithm is 1.0RT, the processor (120) may reduce (e.g., less than 120ms) or increase (e.g., more than 120ms) the length of the speech information to be processed by the CPU. For example, the length of the speech information may be reduced from 120ms to 60ms. In the above example, the speech information may be composed of one input signal. However, the embodiments of the present disclosure are not limited thereto, and the speech information may include a plurality of input signals having the length.

[0131] In operation (610), the processor (120) may obtain voice information from an external source. For example, the voice information may be referred to as voice data. For example, the voice information may include voice, noise, or background sound. For example, the voice information may be obtained from an external source of the electronic device (101). For example, the voice information may be transmitted from an external electronic device (580) through a server or system for providing the virtual environment. For example, the voice information may be obtained through a microphone of the electronic device (101) as the user of the electronic device (101) speaks. For example, the voice information may include text input to the electronic device (101) or the external electronic device (580). For example, the text input may include machine-synthesized voice, such as text to speech (TTS).

[0132] In operation (615), the processor (120) may generate a plurality of input signals from the voice information. For example, the voice information may be composed of an entire utterance, a sentence, a word, or a specified length of a user of the electronic device (101) or another user of the external electronic device (580). For example, the specified length may be defined as a specified size (e.g., n bytes) or a specified time length. For example, the voice information may be composed of a plurality of input signals. Each of the plurality of input signals may be composed of the specified length. However, the embodiments of the present disclosure are not limited thereto, and when the voice information is set to the specified time length, the voice information may be composed of a single input signal. Each input signal among the plurality of input signals may be a unit in which a processing algorithm for the input signal is performed. Specific details related to the plurality of input signals constituting the voice information are described below in FIG. 7.

[0133] Referring to FIG. 6B, in operation (620), the processor (120) may identify whether an input signal includes speech. For example, the processor (120) may identify one input signal among the plurality of input signals. For example, the one input signal may be determined over time. For example, the one input signal may represent an initial input signal among the plurality of input signals. Hereinafter, for convenience of explanation, the one input signal (or the initial input signal) may be referred to as a first input signal.

[0134] For example, the processor (120) may identify whether the first input signal includes speech. In operation (620), if the first input signal includes speech, the processor (120) may perform operation (625). For example, if the first input signal includes speech, the processor (120) may apply a processing algorithm to the first input signal. Conversely, in operation (620), if the first input signal does not include speech, the processor (120) may perform operation (650). For example, if the first input signal does not include speech, the processor (120) may not apply the processing algorithm to the first input signal.

[0135] Although not illustrated in FIGS. 6A and 6B , in operation (620), the processor (120) may, in response to identifying that the first input signal includes voice, identify whether the mouth of the currently displayed avatar is closed. For example, the processor (120) may, in response to the execution of a software application providing the virtual environment, display the avatar corresponding to the user of the electronic device (101). In a state where the avatar is displayed, the processor (120) may perform at least one of operations (600) to (615). For example, the processor (120) may, within the state, obtain voice information from an external source. For example, the processor (120) may, before processing the first input signal, identify whether another input signal exists. In other words, the processor (120) can identify whether an avatar for other voice information prior to the voice information including the first input signal is displayed, or whether the first input signal is the first input signal within the voice information. If the first input signal is the first input signal or an avatar for the other voice information is not displayed, the processor (120) can identify a designated mouth shape based on the volume of the first input signal. The processor (120) can display an avatar including the identified designated mouth shape through the display (510). Specific details related thereto are described below in FIG. 13.

[0136] In operation (625), the processor (120) may perform voice enhancement. For example, the processor (120) may perform the voice enhancement on the first input signal. For example, the voice enhancement may include removing noise from the voice information (or the first input signal), enhancing the voice portion of the voice information (or the first input signal) against the background noise, and normalizing the magnitude of the voice information (or the first input signal). For example, the processor (120) may identify processing circuits for processing noise removal, voice portion enhancement, and normalization, respectively. For example, the processor (120) may identify a processing circuit to process noise removal among the plurality of processing circuits based on a processing speed for noise removal. For example, the processor (120) may identify a processing circuit to process voice portion enhancement among the plurality of processing circuits based on a processing speed for voice portion enhancement. For example, the processor (120) can identify a processing circuit to process the normalization among the plurality of processing circuits based on the processing speed for normalization.

[0137] For example, the processor (120) may, based on a processing circuit identified based on processing speed, use a band pass filter (BPF) to remove signals in the frequency range identified as noise from among the first input signal. Specific details related thereto are described below in FIG. 8A.

[0138] For example, the processor (120) may enhance the speech portion of the speech information from which the noise has been removed using a processing circuit identified based on the processing speed. For example, the processor (120) may enhance the speech portion using an artificial intelligence (AI) model. For example, the enhancement based on the AI ​​model may be performed based on a central processing unit (CPU), a graphic processing unit (GPU), or a neural processing unit (NPU) among a plurality of processing circuits of the electronic device (101). Alternatively, the processor (120) may enhance the speech portion using a computational algorithm. The computational algorithm may represent a non-AI model-based algorithm. For example, the computational algorithm may include an improved minima controlled recursive averaging (IMCRA) algorithm or a log minimum mean square error (logMMSE) algorithm. For example, the enhancement based on the above-described computational algorithm may be performed based on a central processing unit (CPU) among the multiple processing circuits of the electronic device (101). In other words, the processor (120) may utilize the artificial intelligence model based on a CPU, GPU, or NPU among the multiple processing circuits, and may utilize the computational algorithm based on the CPU. Specific details related thereto are described below in FIGS. 8B and 8C.

[0139] For example, the processor (120) may normalize the voice information in which the voice portion is enhanced using a processing circuit identified based on the processing speed. For example, the normalization of the voice information may refer to adjusting the volume of the voice information. For example, the processor (120) may change the volume of the voice information to fall within a specified range. The specified range may be set to input normalized input information for an artificial intelligence model for processing the voice information. In other words, the artificial intelligence model may generate more accurate output when the normalized input information is input. In the above example, the specified range is exemplified as being set for the artificial intelligence model, but the embodiments of the present disclosure are not limited thereto. For example, when the voice information has a value within the specified range, a quantitative comparison between different voice information is possible, thereby simplifying the calculation. Specific details related thereto are described in FIG. 8D below.

[0140] In operation (630), the processor (120) may obtain a plurality of feature values. For example, the processor (120) may obtain the plurality of feature values ​​based on a processing circuit identified based on a processing speed. For example, the processor (120) may extract the plurality of feature values ​​from the first input signal (or the voice information), which is an analog signal. For example, the processor (120) may obtain the feature values ​​based on a mel frequency cepstral coefficient (MFCC) algorithm.

[0141] For example, the processor (120) can obtain a spectrum by applying a fast-Fourier transform (FFT) to the first input signal. For example, the processor (120) can obtain the spectrum for the frequency domain by applying the FFT to the first input signal. The processor (120) can obtain a mel spectrum by applying a mel filter bank to the spectrum. For example, the processor (120) can obtain the mel spectrum based on a mel scale that represents the relationship between the frequency domain and a low frequency band recognized by an actual human. The processor (120) can obtain MFCCs by applying a cepstral analysis to the mel spectrum. The MFCCs can be referred to as the feature values. For example, the processor (120) may acquire the feature values, which are a portion of the total feature values, which are peaks acquired based on the cepstral analysis. The peaks may be referred to as formants. For example, the number of feature values ​​may be 40. However, the embodiments of the present disclosure are not limited thereto. For example, the processor (120) may learn an artificial intelligence model based on the acquired feature values. In other words, the processor (120) may learn the artificial intelligence model by using the feature values ​​as input. Accordingly, the processor (120) may acquire refined feature values.

[0142] In the above-described example, the electronic device (101) obtains the feature values ​​based on the MFCC algorithm, and exemplifies a method of using the feature values ​​without additional processing or refining the feature values ​​using the artificial intelligence model. However, the embodiments of the present disclosure are not limited thereto. For example, the processor (120) may obtain the feature values ​​without the MFCC algorithm based on the voice information using the artificial intelligence model. For example, if a processing circuit (e.g., NPU, GPU) with a relatively fast processing speed is available among the plurality of processing circuits of the electronic device (101), the processor (120) may obtain the feature values ​​using the artificial intelligence model. Alternatively, if a processing circuit (e.g., CPU) with a relatively slow processing speed is available among the plurality of processing circuits of the electronic device (101), the processor (120) may obtain the feature values ​​using the MFCC algorithm. Specific details related thereto are described below in FIG. 9.

[0143] As described above, the processor (120) may utilize the artificial intelligence model based on a plurality of processing circuits. For example, the plurality of processing circuits may include a CPU, a GPU, or an NPU. Furthermore, the electronic device (101) may utilize the MFCC algorithm based on the CPU.

[0144] In operation (635), the processor (120) may obtain information for generating a mouth shape. For example, the processor (120) may obtain information for generating the mouth shape using an artificial intelligence model. For example, the electronic device (101) may utilize the artificial intelligence model based on a plurality of processing circuits. For example, the plurality of processing circuits may include a CPU, a GPU, or an NPU.

[0145] For example, the information for generating the mouth shape may include at least one of a visual phoneme (viseme), a face landmark, a blend weight, or a face mesh for the speech information. For example, the visual phoneme may represent a mouth shape symbol of an avatar indicating that the speech of the speech information is being uttered. For example, the face landmark may represent coordinates for the face of the avatar indicating that the speech of the speech information is being uttered. For example, the face landmark may include three-dimensional coordinates or two-dimensional coordinates. The blend weight may represent an emotion parameter for changing the facial expression of the avatar. For example, the blend weight may be acquired based on a retargeting model. For example, the blend weight may be acquired from the face landmark or the speech information. For example, the face mesh may represent a mesh formed by the facial feature points. Details related thereto are described in FIG. 10 below.

[0146] In operation (640), the processor (120) may generate an avatar including the mouth shape. For example, the processor (120) may obtain an avatar having the mouth shape based on information for generating the mouth shape. For example, the processor (120) may generate an animation including an avatar having the mouth shape. For example, the animation may display visual information including the virtual environment and the avatar for a time corresponding to a plurality of frames. For example, the plurality of frames may be referenced as playback frames set for the animation. The animation may include an avatar having the mouth shape of each of the plurality of frames. The plurality of frames may include key frames of a specified period. For example, the processor (120) may generate an avatar having the mouth shape for each of the plurality of frames, or generate an avatar having the mouth shape for each of the key frames. Specific details related thereto are described below in FIG. 11.

[0147] In operation (645), the processor (120) may display the avatar. For example, the processor (120) may display the generated avatar through the display (510). For example, the processor (120) may display the animation including the avatar through the display (510). In other words, the processor (120) may display the avatar or the animation including the avatar through the display (510). Displaying the avatar or the animation may be understood as being substantially the same as playing the avatar or the animation through the display (510). The processor (120) may change the playback speed, delete some of the playlist (content), or use a parallel processing method in order to minimize the delay time felt by the user (e.g., the user (400)). Specific details related thereto are described in FIGS. 12A to 12C below.

[0148] In operation (650), the processor (120) may identify whether the input signal is the last input signal. For example, the processor (120) may identify whether the first input signal is the last input signal among the plurality of input signals (or the voice information). If it is identified in operation (650) that the first input signal is the last input signal, the processor (120) may perform operation (660). Alternatively, if it is identified in operation (650) that another input signal (e.g., a second input signal) is further included among the plurality of input signals in addition to the first input signal, the processor (120) may perform operation (655). For example, the second input signal may represent an input signal following the first input signal among the plurality of input signals.

[0149] In operation (655), the processor (120) may identify a processing speed of at least one processing circuit. For example, the at least one processing circuit may include a processing circuit used to apply a processing algorithm to the first input signal. In the example of Table 2, the at least one processing circuit may include a GPU as a processing circuit for noise removal, a GPU as a processing circuit for voice portion enhancement, an NPU as a processing circuit for feature value identification, and an NPU as a processing circuit for lip shape identification. The processor (120) may identify an actual processing speed of each of the GPU and the NPU for each processing algorithm for the first input signal. The actual processing speed may be different from the expected processing speed identified in Table 2. The actual processing speed and the expected processing speed identified based on the reference data may be different from each other. This may be because the expected processing speed is a speed at which the reference data is processed, and the actual processing speed is a speed at which the voice information is processed, and thus different data is processed. Additionally, this may be because the first time point (timing) at which the expected processing speed is measured and the second time point at which the actual processing speed for processing the voice information is identified are different. For example, at the first time point, the plurality of processing circuits may not be used. However, at the second time point, some of the plurality of processing circuits may be used for processing other than processing the voice information. For example, the expected processing speed may be referenced as a processing speed predicted based on reference data.

[0150] For example, the processor (120) may update the performance of the at least one processing circuit. For example, the processor (120) may store the actual processing speed at which the first input signal is processed based on the at least one processing circuit. The processor (120) may update the actual processing speed with the performance of the processing circuit for which the actual processing speed is measured. Thereafter, the processor (120) may select (or identify) a processing circuit to process voice information (e.g., the second input signal) to be acquired in the future among the plurality of processing circuits based on the actual processing speed.

[0151] In operation (660), the processor (120) may display an avatar including a closed mouth shape. For example, if the processor (120) identifies that the processed input signal is the last input signal among the plurality of input signals (or the voice information), the processor (120) may display an avatar including a closed mouth shape. In other words, the processor (120) may generate an avatar including a closed mouth shape to be displayed until voice information other than the voice information is acquired. For example, acquiring the other voice information may include acquiring an input to change the appearance or operation of the avatar in addition to acquiring the other voice information from an external source.

[0152] Referring to the above, an electronic device and method optimized for each electronic device for generating an avatar based on real-time voice information according to an embodiment of the present disclosure are described. The electronic device and method according to an embodiment of the present disclosure can quickly and flexibly reduce lip sync even in an internal environment (or on-device environment) of the electronic device. In other words, the electronic device and method according to an embodiment of the present disclosure can quickly generate an avatar (or an avatar's mouth shape, or an animation including an avatar having a mouth shape) with higher accuracy by monitoring and efficiently utilizing resources within the electronic device. Accordingly, the electronic device and method according to an embodiment of the present disclosure can provide a more immersive user experience to the user. Furthermore, the electronic device and method according to an embodiment of the present disclosure can secure real-time performance even in a multi-tasking environment through optimized operations for generating an avatar having a mouth shape based on voice during the runtime of the electronic device. In addition, the electronic device and method according to the embodiment of the present disclosure can reduce the overall resource usage by utilizing only the resources of the electronic device itself (on-device) and not using the resources of the server providing the virtual environment and additional resources (e.g., data) for connection with the server.

[0153] FIG. 7 illustrates an example of a delay time between the timing for acquiring voice information and the timing for playing the voice information according to various embodiments.

[0154] Figure 7 illustrates examples (700, 750) of acquiring, processing, and reproducing voice information having different lengths. Example (700) may represent a case where the specified length of voice information (or input signal) acquired from an external source is set to 120 ms. Example (750) may represent a case where the specified length of voice information (or input signal) acquired from an external source is set to 80 ms. For example, the specified length may be defined as a specified size (e.g., n bytes) or a specified time length. For example, the voice information may be composed of a plurality of input signals. Each of the plurality of input signals may be composed of the specified length. In examples (700) and (750), for convenience of explanation, it is assumed that the voice information includes one input signal.

[0155] Referring to example (700), the processor (120) can obtain the voice information having a length of 120 ms. For example, the processor (120) can record the voice information having a length of 120 ms for 120 ms. The processor (120) can process the voice information. For example, the processor (120) can process the voice information having a length of 120 ms for 60 ms. For example, the processor (120) can generate an avatar including a mouth shape for the voice information having a length of 120 ms. Thereafter, the processor (120) can display (or play) the avatar including the mouth shape for the 120 ms. A user of an electronic device (101) can identify the time length (730) between the first timing (710) at which the voice information is input and the second timing (720) at which an avatar including the mouth shape for the voice information is played back as a delay time. In other words, the time length (730) between the first timing (710) at which the voice information is input and the second timing (720) at which the avatar begins to be played back can be a delay time.

[0156] In contrast, referring to example (750), the processor (120) can obtain the voice information having a length of 80 ms. For example, the processor (120) can record the voice information having a length of 80 ms for 80 ms. The processor (120) can process the voice information. For example, the processor (120) can process the voice information having a length of 80 ms for 40 ms. For example, the processor (120) can generate an avatar including a mouth shape for the voice information having a length of 80 ms. Thereafter, the processor (120) can display (or play) the avatar including the mouth shape for the 80 ms. A user of an electronic device (101) can identify the time length (780) between the first timing (760) at which the voice information is input and the second timing (770) at which an avatar including the mouth shape for the voice information is played back as a delay time. In other words, the time length (780) between the first timing (760) at which the voice information is input and the second timing (770) at which the avatar begins to be played back can be a delay time.

[0157] As described above, if the time for inputting voice information is long or the processing time for generating an avatar including a mouth shape for the input voice information is long, the user may feel that the delay time is long. For example, the processor (120) may set the specified time length based on the delay time between the first timing and the second timing. For example, the specified time length may be identified based on the performance (e.g., processing speed) of the processing circuit that processes the voice information (or input signal) and the accuracy of the artificial intelligence model. For example, the performance of the processing circuit may be referenced as the performance of the artificial intelligence model that processes the voice information (or input signal). In order to reduce the delay time, the processor (120) may set the specified time length for processing a single voice information (or input signal) to a minimum length. However, if the specified time length is short, a break may occur when playing the animation for the entire user's utterance. Additionally, as the specified time length decreases, overhead may occur in the processing circuit to process a large amount of audio information (or input signals). Therefore, the processor (120) can set an optimal specified time length to generate seamless animation without processing circuit overhead while minimizing the delay time. For example, the processor (120) can distinguish the audio information into a plurality of input signals, each of which has the specified time length.

[0158] FIG. 8a illustrates an example of a band pass filter (BPF) for removing noise from voice information according to various embodiments.

[0159] FIG. 8A illustrates an example of a graph (800) representing the gain of the speech information according to frequency, for explaining the operation of the BPF for noise removal performed in operation (625) of FIG. 6B. The horizontal axis of the graph (800) may represent frequency (unit: Hz (Herts)), and the vertical axis of the graph (800) may represent the gain of the speech information (unit: dB (decibel)). The graph (800) includes a line (805) representing the gain of the speech information according to frequency.

[0160] Referring to line (805), the gain of the voice information according to frequency may have a symmetrical value based on the center frequency (f0). For example, at the center frequency (f0), the gain may be a maximum value of 0 dB. For example, at the first frequency (f H ), the gain may be about -3 dB. Or, at the second frequency (f L ), the gain can be about -3 dB. At the first frequency (f H ) and the second frequency (f L ) can be referred to as the cutoff frequency. The first frequency (f H ) and the second frequency (f L ) can be referred to as bandwidth (B).

[0161] Referring to the above, the processor (120) can identify (or select) a signal in a frequency domain within a length (810) from the voice information using a BPF. For example, the signal in the frequency domain within the length (810) may include a voice portion included in the voice information. In other words, the frequency domain within the length (810) may represent a frequency band for a typical human voice. The processor (120) can identify the remaining frequency domains excluding the frequency domain as noise and cancel or filter the remaining domains excluding the frequency domain. Thereafter, the processor (120) can enhance the voice portion from the voice information from which the noise has been removed. Specific details related thereto are described below with reference to FIGS. 8B and 8C.

[0162] FIGS. 8B and 8C illustrate examples of methods for enhancing voice from speech information according to various embodiments.

[0163] FIGS. 8b and 8c illustrate examples (820, 840) of the voice information over time to illustrate the enhancement of the voice portion performed in operation (625) of FIG. 6b.

[0164] Example (820) of FIG. 8B illustrates background sound (822) and voice (824) included in the speech information over time. For example, the processor (120) can separate the background sound (822) and voice (824) included in the speech information. For example, the processor (120) can identify the background sound (822) and voice (824) included in the speech information, respectively, using an artificial intelligence model. Alternatively, the processor (120) can identify the background sound (822) and voice (824) included in the speech information, respectively, using a computational algorithm (or based on a non-AI model).

[0165] Example (840) of FIG. 8C illustrates an example of a graph representing the voice information and the amplitude of the voice (824) of the voice information over time. Example (840) may include a first line (842) representing the amplitude of the voice information acquired by the electronic device (101) (or the processor (120)) over time, and a second line (844) representing the amplitude of the voice (824) of the voice information. Referring to example (840), the first line (842) and the second line (844) may be formed to have similar amplitudes. The difference between the second line (844) and the first line (842) may include a portion that is not the voice (824), such as background sound (822). For example, the processor (120) can separate background sound (822) and voice (824) based on the artificial intelligence model or the computational algorithm, and perform additional processing to improve the quality of the separated voice (824).

[0166] Referring to FIGS. 8B and 8C, the processor (120) may separate background sound (822) and voice (824) based on the artificial intelligence model or the computational algorithm, and enhance the voice (824) to have an amplitude similar to the voice information. For example, the processor (120) may enhance the voice (824) using the artificial intelligence model. For example, the enhancement based on the artificial intelligence model may be performed based on a central processing unit (CPU), a graphic processing unit (GPU), or a neural processing unit (NPU) among a plurality of processing circuits of the electronic device (101). Alternatively, the processor (120) may enhance the voice (824) using the computational algorithm. For example, the computational algorithm may include an improved minima controlled recursive averaging (IMCRA) algorithm or a log minimum mean square error (logMMSE) algorithm. For example, the enhancement based on the above-described operation algorithm may be performed based on a central processing unit (CPU) among the plurality of processing circuits of the electronic device (101). However, the embodiments of the present disclosure are not limited thereto, and the operation algorithm may include an algorithm capable of enhancing the voice portion from the voice information. In other words, the processor (120) may utilize the artificial intelligence model based on a CPU, GPU, or NPU among the plurality of processing circuits, and may utilize the operation algorithm based on the CPU.

[0167] For example, identifying background sound (822) and voice (824) using the artificial intelligence model may take a lot of time compared to using the computational algorithm. Therefore, when a processing circuit having a processing speed faster than the reference processing speed is used, the processor (120) can identify background sound (822) and voice (824) using the artificial intelligence model. Conversely, when a processing circuit having a processing speed slower than the reference processing speed is used, the processor (120) can identify background sound (822) and voice (824) using the computational algorithm.

[0168] FIG. 8d illustrates an example of normalizing the volume of voice information according to various embodiments.

[0169] FIG. 8d illustrates examples (860, 880) for explaining the normalization of speech information performed in operation (625) of FIG. 6b. The normalization may refer to adjusting (adjusting, or changing) the volume of a portion of speech information (or speech (e.g., speech (824) of FIG. 8b)).

[0170] FIG. 8d illustrates an example (860) of plotting the magnitude of the acquired voice information over time and an example (880) of plotting the magnitude of the normalized voice information over time. The magnitude may also be referred to as amplitude. Comparing examples (860) and (880), the amplitude (870) before normalization may have a smaller value than the amplitude (890) after normalization. For example, the processor (120) may increase the amplitude (870) so that the amplitude (870) has a value within a specified range. However, the embodiments of the present disclosure are not limited thereto. For example, if the amplitude (870) has a larger value than the amplitude (890), the processor (120) may decrease the amplitude (870) so that the amplitude (870) has a value within the specified range.

[0171] As described above, the processor (120) can change the size of the speech information to have a value within a specified range. The specified range can be set to input normalized input information for an artificial intelligence model for processing the speech information. The specified range can be a representative value of the size of the input information. For example, the representative value can include an average value or a median value. When the processor (120) inputs the normalized input information, it can generate more accurate output using the artificial intelligence model. In the above example, the specified range is exemplified as being set for the artificial intelligence model, but the embodiments of the present disclosure are not limited thereto. For example, when the speech information has a value within the specified range, quantitative comparison between different speech information can be made, thereby simplifying the calculation.

[0172] As described above, the electronic device (101) can normalize the magnitude of voice information (or input signal) to transform voice information that may be input in various environments into a form that appears to have been acquired under identical conditions. Accordingly, the electronic device (101) can more accurately perform feature value identification and lip shape identification based on the voice information. Specific details regarding the feature value identification are described below in FIG. 9 .

[0173] FIG. 9 illustrates an example of a block diagram for obtaining feature values ​​of voice information according to various embodiments.

[0174] FIG. 9 illustrates an example of a block diagram for obtaining multiple feature values ​​performed in operation (630) of FIG. 6b.

[0175] Referring to FIG. 9, the processor (120) can acquire a plurality of feature values ​​based on voice information (910). In the example of FIG. 9, the voice information (910) can be referenced as a single input signal. For example, the processor (120) can identify the plurality of feature values ​​using MFCC (920) (or MFCC algorithm) based on the voice information (910).

[0176] For example, the processor (120) can obtain a spectrum by applying a fast-Fourier transform (FFT) to the voice information (910). For example, the processor (120) can obtain the spectrum for the frequency domain by applying the FFT to the voice information (910). The processor (120) can obtain a mel spectrum by applying a mel filter bank to the spectrum. For example, the processor (120) can obtain the mel spectrum based on a mel scale that represents the relationship between the frequency domain and a low frequency band recognized by an actual human. The processor (120) can obtain MFCCs by applying a cepstral analysis to the mel spectrum. The MFCCs can be referred to as the feature values. For example, the processor (120) may obtain the feature values, which are a portion of the total feature values, which are peaks obtained based on the cepstral analysis. The peaks may be referred to as formants. For example, the feature values ​​may be 40. For example, the number of feature values ​​may be less than 40 or greater than 40. For example, the processor (120) may obtain visual phonemes (930) based on the feature values ​​obtained using MFCC (920).

[0177] For example, the processor (120) can learn the artificial intelligence model (940) based on the acquired feature values. In other words, the processor (120) can learn the artificial intelligence model (940) by inputting the feature values. For example, the artificial intelligence model (940) can include a convolution neural network encoder (CNN). Accordingly, the processor (120) can obtain refined feature values. For example, the processor (120) can obtain the visual phoneme (930) based on the refined feature values. In the above example, the electronic device (101) obtains the feature values ​​based on the MFCC algorithm, and exemplifies a method of using the feature values ​​without additional processing or refining the feature values ​​using the artificial intelligence model (940), but the embodiments of the present disclosure are not limited thereto.

[0178] For example, the processor (120) may obtain the feature values ​​without the MFCC algorithm based on the speech information (910) by using the artificial intelligence model (940). For example, the processor (120) may obtain the feature values ​​by inputting the speech information (910) into the artificial intelligence model (940). For example, the processor (120) may obtain the visual phoneme (930) based on the feature values ​​obtained using the artificial intelligence model (940).

[0179] For example, the visual phoneme (930) of FIG. 9 may be referenced as an example of information for generating a mouth shape. For example, the information for generating the mouth shape may include at least one of a visual phoneme (viseme), a face landmark, a blend weight, or a face mesh for the speech information. For example, the visual phoneme may represent a mouth shape symbol of an avatar indicating that the speech of the speech information is being uttered. For example, the face landmark may represent coordinates for the face of an avatar indicating that the speech of the speech information is being uttered. For example, the face landmark may include three-dimensional coordinates or two-dimensional coordinates. The blend weight may represent an emotion parameter for changing the facial expression of the avatar. For example, the blend weight may be acquired based on a retargeting model. For example, the blend weights may be obtained from the facial feature points or the voice information. For example, the face mesh may represent a mesh formed by the facial feature points.

[0180] For example, if a processing circuit (e.g., NPU, GPU) having a relatively fast processing speed is available among the plurality of processing circuits of the electronic device (101), the processor (120) can obtain the feature values ​​using the artificial intelligence model (940). Alternatively, if a processing circuit (e.g., CPU) having a relatively slow processing speed is available among the plurality of processing circuits of the electronic device (101), the processor (120) can obtain the feature values ​​using the MFCC algorithm. For example, if a processing circuit having a processing speed faster than a reference processing speed is used among the plurality of processing circuits, the processor (120) can obtain the feature values ​​using the artificial intelligence model (940), or learn the artificial intelligence model (940) based on the feature values ​​obtained using the MFCC (920). In contrast, when a processing circuit having a processing speed slower than the reference processing speed is used among the plurality of processing circuits, the processor (120) may obtain the feature values ​​using MFCC (920). For example, the processor (120) may use an artificial intelligence model (940) based on the plurality of processing circuits. For example, the plurality of processing circuits may include a CPU, a GPU, or an NPU. In addition, the processor (120) may use MFCC (920) based on the CPU.

[0181] As described above, the electronic device and method according to the embodiment of the present disclosure may use MFCC (920) and / or an artificial intelligence model (940) to identify feature values. When the artificial intelligence model (940) is used to identify the feature values, the feature values ​​can be obtained through an artificial intelligence model network without using an additional module (e.g., a module for MFCC (920)), and the additionally required preprocessing process can be reduced. However, as described above, when the processing speed of the processing circuit currently available to the processor (120) is slower than the reference processing speed, the processor (120) may use MFCC (920) for rapid processing of voice information without using the artificial intelligence model (940).

[0182] FIG. 10 illustrates an example of a method for obtaining information for generating a mouth shape based on voice information according to various embodiments.

[0183] Referring to FIG. 10, examples (1000, 1050) of information for generating the mouth shape obtained in the operation (635) of FIG. 6b are illustrated.

[0184] Referring to example (1000), the processor (120) can identify facial feature points (1010) based on feature values. For example, the feature values ​​can be obtained from the speech information based on the MFCC (920) or the artificial intelligence model (940) of FIG. 9. For example, the processor (120) can identify facial feature points (1010) from the feature values ​​using the artificial intelligence model based on a processing circuit identified based on a processing speed among a plurality of processing circuits. For example, the facial feature points (1010) can represent coordinates for the face of an avatar to indicate that the speech of the speech information is spoken. In example (1000) of FIG. 10, the facial feature points (1010) are 2D coordinates, but the embodiments of the present disclosure are not limited thereto. For example, the facial feature points (1010) can be configured as 3D coordinates.

[0185] Referring to example (1050), the processor (120) can identify a face mesh (1060) based on the feature values. For example, the processor (120) can identify the face mesh (1060) based on the facial feature points (1010) identified from the feature values ​​in example (1000). For example, the face mesh (1060) can represent a mesh formed by the points of the facial feature points (1010). For example, the processor (120) can generate a visual object representing a mouth shape for the voice information based on the face mesh (1060). The visual object can represent a visual object for representing a mouth part of an avatar corresponding to a user of the electronic device (101) (or an external electronic device (580)). For example, the processor (120) can synthesize the mouth portion of the avatar with the visual object to create the avatar having (including) the mouth shape.

[0186] Referring to the above, the processor (120) may generate facial feature points (1010) based on the feature values ​​using the artificial intelligence model, and may generate a face mesh (1060) based on the facial feature points (1010). The processor (120) may generate a visual object representing the mouth shape based on the face mesh (1060). However, the embodiments of the present disclosure are not limited thereto.

[0187] For example, the processor (120) may obtain a visual phoneme based on the feature values ​​using the artificial intelligence model. For example, the processor (120) may obtain the visual phoneme, which is a mouth shape symbol representing the voice of the voice information, based on the feature values ​​obtained from the speech information using the artificial intelligence model. For example, the visual phoneme may be mapped to a designated value (hereinafter, a first value). The processor (120) may identify a blend weight based on the visual phoneme. For example, the blend weight may be mapped to a designated value (hereinafter, a second value). The processor (120) may obtain the blend weight from the visual phoneme using a mapping table between the first value and the second value. For example, the processor (120) may obtain a face mesh (1060) by applying the blend weight.

[0188] Alternatively, the processor (120) may obtain facial feature points (1010) based on the feature values ​​using the artificial intelligence model. For example, the processor (120) may obtain facial feature points (1010) representing the voice of the voice information based on the feature values ​​obtained from the voice information using the artificial intelligence model. For example, the facial feature points (1010) may include three-dimensional coordinates or two-dimensional coordinates. For example, the processor (120) may obtain a face mesh (1060) from the facial feature points (1010). For example, the processor (120) may obtain the face mesh (1060) from the facial feature points (1010) using a retargeting model. The retargeting model may represent a model for adjusting the face mesh (1060) using the facial feature points (1010).

[0189] Alternatively, the processor (120) may obtain a blend weight based on the feature values ​​using the artificial intelligence model. For example, the processor (120) may obtain a blend weight, which is a value for generating a mouth shape for expressing the voice of the voice information, based on the feature values ​​obtained from the voice information using the artificial intelligence model. For example, the blend weight may represent a value mapped according to the shape of the avatar's mouth. For example, the blend weight may represent a value mapped to factors for forming the shape of the mouth, such as the corners of the mouth, the middle part of the mouth, lip wrinkles, or lip curvature. For example, the shape of the corners of the mouth may be changed by changing the mapped value. For example, the processor (120) may obtain a face mesh (1060) based on the blend weight.

[0190] Alternatively, the processor (120) may obtain a face mesh (1060) based on the feature values ​​obtained from the voice information using the artificial intelligence model. For example, the face mesh (1060) may be an output of the artificial intelligence model that uses the feature values ​​as input. Alternatively, the processor (120) may obtain the face mesh (1060) directly from the voice information using the artificial intelligence model. In other words, the processor (120) may obtain the face mesh (1060) from the obtained voice information without the process of identifying the feature values.

[0191] Utilizing the above artificial intelligence model may indicate that the processor (120) inputs information into the artificial intelligence model and obtains output by utilizing a processing circuit identified based on processing speed among the plurality of processing circuits. For example, the plurality of processing circuits may include an NPU, a GPU, or a CPU. The artificial intelligence model may represent an artificial intelligence model learned by voice information processed through operation (625) or operation (630).

[0192] For example, the processor (120) may generate a visual object representing the shape of the mouth based on the acquired face mesh (1060). For example, the processor (120) may generate an avatar (or animation) synthesized with the visual object. Details regarding the method for generating the avatar (or animation) are described below in FIG. 11.

[0193] FIG. 11 illustrates examples of a method for generating animation for an avatar including a mouth shape according to various embodiments.

[0194] FIG. 11 illustrates examples (1100, 1150, 1155) of a method for generating an avatar performed in operation (640) of FIG. 6b.

[0195] Referring to FIG. 11, the processor (120) may generate the avatar including the mouth shape generated based on information for generating the mouth shape. For example, the processor (120) may generate a visual object representing the mouth shape based on the information for generating the mouth shape. The processor (120) may generate the avatar synthesized with the visual object. For example, the processor (120) may generate an animation for the avatar. For example, the animation may display visual information including the virtual environment and the avatar for a time corresponding to a plurality of frames. For example, the plurality of frames may be referenced as playback frames set for the animation. The animation may include the avatar having the mouth shape of each of the plurality of frames. The plurality of frames may include key frames of a specified period. For example, the processor (120) may generate an avatar having the mouth shape for each of the plurality of frames, or generate an avatar having the mouth shape for each of the key frames.

[0196] FIG. 11 illustrates an example (1100) of a method for generating an avatar having the mouth shape for each of the plurality of frames, and examples (1150, 1155) of a method for generating an avatar having the mouth shape for each of the key frames. The plurality of frames of FIG. 11 are illustrated as including 10 frames, but embodiments of the present disclosure are not limited thereto. For example, the plurality of frames may include 9 or fewer or 11 or more frames.

[0197] Referring to example (1100), the processor (120) may generate an avatar for each of the plurality of frames. For example, the processor (120) may identify the plurality of frames constituting a specified time length for the voice information (or input signal). For example, the processor (120) may identify a visual object representing a mouth shape for each of the plurality of frames, and generate the avatar by synthesizing the identified visual object.

[0198] Alternatively, referring to example (1150), the processor (120) may generate an avatar for each of the key frames, which are some frames among the plurality of frames. For example, the processor (120) may identify key frames (1161, 1164, 1167) of a designated period among the plurality of frames for the voice information (or input signal). The designated period may be three frames. For example, the processor (120) may identify a visual object representing a mouth shape for each of the key frames (1161, 1164, 1167) and generate the avatar synthesized with the identified visual object.

[0199] Referring to example (1155), the processor (120) may generate the avatar for other frames (1162, 1165, 1168) based on the avatar generated for key frames (1161, 1164, 1167). For example, the processor (120) may identify a visual object representing a mouth shape in frames (1162) for changing from the avatar of key frame (1161) to the avatar of key frame (1164), and generate the avatar synthesized with the identified visual object. In addition, for example, the processor (120) may identify a visual object representing a mouth shape in frames (1165) for changing from the avatar of key frame (1164) to the avatar of key frame (1167), and generate the avatar synthesized with the identified visual object. Additionally, for example, the processor (120) may identify a visual object representing a mouth shape of frames (1168) to be changed from the avatar of the key frame (1167) to the avatar of the last frame (1169), and generate the avatar synthesized with the identified visual object. The avatar of the last frame (1169) may be an avatar synthesized with a visual object representing a mouth shape in a closed state. For example, the last frame (1169) may be a time interval including a time point (timing) at which the voice information ends. For example, the processor (120) may use Bezier curves or interpolation to generate avatars of other frames (1162, 1165, 1168) between the key frames based on the key frames (1161, 1164, 1167). The above Bezier curve and the above interpolation are merely examples of a method for estimating the remaining frames based on some known frames among a plurality of frames, and the embodiments of the present disclosure are not limited thereto.

[0200] FIGS. 12A through 12C illustrate examples of how to play an animation for an avatar including a mouth shape according to various embodiments.

[0201] Figures 12a to 12c illustrate examples (1200, 1210, 1220, 1240) of a method for displaying an avatar (or an animation including an avatar) performed in operation (645) of Figure 6b. In order for a user to recognize that the avatar within a virtual environment is reflected in real time, the avatar's mouth needs to open quickly when the user speaks, and the avatar's mouth needs to close quickly when the user ends the speech. In this case, reducing the time for processing (e.g., voice enhancement, feature value identification, mouth shape identification) of the voice information spoken by the user may be limited. An electronic device and method according to an embodiment of the present disclosure can reduce the delay time based on a method for playing an avatar (or an animation including the avatar) including a generated mouth shape. The playing may include continuously displaying the avatar over time. For example, the processor (120) may play the animation for the avatar.

[0202] Referring to FIG. 12a, an example (1200) of playing an avatar (or animation) having a different time length than the recorded voice information and an example (1210) of changing the playback speed are illustrated.

[0203] Referring to example (1200), the processor (120) can obtain voice information #A (1201), voice information #B (1202), voice information #C (1203), and voice information #D (1204). For example, the processor (120) can record voice information #A (1201), voice information #B (1202), voice information #C (1203), and voice information #D (1204) over time. It is assumed that the time length of each of voice information #A (1201), voice information #B (1202), voice information #C (1203), and voice information #D (1204) is 10 ms. For example, the processor (1200) can start processing (1205) for voice information #A (1201) at the time (timing) of obtaining voice information #B (1202). The processor (120) may start processing (1206) for voice information #B (1202) at the time (timing) at which voice information #C (1203) is acquired. For example, the time required for processing (1205) for voice information #A (1201) and processing (1206) for voice information #B (1202) is assumed to be 8 ms. For example, the processor (120) may start playback (1207) from the time (timing) at which processing (1205) for the processed voice information #A (1201) ends. For example, playback (1207) for voice information #A (1201) may be extended beyond the time length (10 ms) of voice information #A (1201). For example, playback (1207) may be performed for 12 ms, which is extended by the time length (1208). For example, the time length (1208) may be 2 ms. The increase in the time length of playback (1207) may occur as the playback speed (or rendering speed) slows down as the processor (120) processes other operations.In this case, as the time length of playback (1207) increases, even though the processing (1206) for voice information #B (1202) is completed, voice information #B (1202) may not be played. Accordingly, the delay time for voice information #B (1202) may become longer by the time length (1208).

[0204] Referring to example (1210), the processor (120) can obtain voice information #A (1211), voice information #B (1212), voice information #C (1213), and voice information #D (1214). For example, the processor (120) can record voice information #A (1211), voice information #B (1212), voice information #C (1213), and voice information #D (1214) over time. It is assumed that the time length of each of voice information #A (1211), voice information #B (1212), voice information #C (1213), and voice information #D (1214) is 10 ms. For example, the processor (120) can start processing (1215) for voice information #A (1211) at the time (timing) of obtaining voice information #B (1212). The processor (120) can start processing (1216) for voice information #B (1212) at the time (timing) at which voice information #C (1213) is acquired. For example, the time required for processing (1215) for voice information #A (1211) and processing (1216) for voice information #B (1212) is assumed to be 8 ms. For example, the processor (120) can start reproduction (1217) from the time (timing) at which processing (1215) for the processed voice information #A (1211) ends. For example, reproduction (1217) for voice information #A (1211) can be extended beyond the time length (10 ms) of voice information #A (1211). For example, the time for reproduction (1217) can be performed for 12 ms, which is extended by the time length (1218). However, unlike example (1200), the processor (120) can adjust the time for playback (1217) to correspond to the time length (10 ms) of voice information #A (1211) by changing the playback speed for playback (1217).For example, the processor (120) can shorten the time for playback (1217) by relatively quickly changing the playback speed for playback (1217). Accordingly, the delay time for voice information #B (1212) may not be delayed by as much as the time length (1218). For example, the processor (120) can start playback (1219) immediately after the processing (1216) for voice information #B (1212) is finished.

[0205] Referring to the above, the processor (120) can process the acquired voice information and play (or display) an avatar (or animation) having a mouth shape corresponding to the voice information. At this time, the processor (120) can change the speed (or playback speed) at which the avatar is played. Accordingly, the delay time experienced by the user can be reduced.

[0206] Referring to FIG. 12b, an example (1220) is shown of playing an avatar (or animation) for the next voice information of the recorded voice information, while partially ignoring an avatar (or animation) having a different time length from the recorded voice information.

[0207] Referring to example (1220), the processor (120) can obtain voice information #A (1221), voice information #B (1222), voice information #C (1223), and voice information #D (1224). For example, the processor (120) can record voice information #A (1221), voice information #B (1222), voice information #C (1223), and voice information #D (1224) over time. It is assumed that the time length of each of voice information #A (1221), voice information #B (1222), voice information #C (1223), and voice information #D (1224) is 10 ms. For example, the processor (120) can start processing (1225) for voice information #A (1221) at the time (timing) of obtaining voice information #B (1222). The processor (120) may start processing (1226) for voice information #B (1222) at the time (timing) at which voice information #C (1223) is acquired. For example, the time required for processing (1225) for voice information #A (1221) and processing (1226) for voice information #B (1222) is assumed to be 8 ms. For example, the processor (120) may start playback (1227) from the time (timing) at which processing (1225) for the processed voice information #A (1221) ends. For example, playback (1227) for voice information #A (1221) may be extended beyond the time length (10 ms) of voice information #A (1221). For example, playback (1227) may be performed for 12 ms, which is extended by the time length (1228). For example, the time length (1228) may be 2 ms. The increase in the time length of playback (1227) may occur as the playback speed (or rendering speed) slows down as the processor (120) processes other operations.Unlike the example (1200) of FIG. 12A, the processor (120) may ignore the portion of the time length (1228) during the playback (1227) of the voice information #A (1221) extended by the time length (1228) and start the playback (1229) of the voice information #B (1222). For example, the playback (1229) of the voice information #B (1222) may start immediately after the processing (1226) of the voice information #B (1222) ends. For example, ignoring the portion of the time length (1228) during the playback (1227) may include stopping the playback (1227) at the point in time when the portion of the time length (1228) begins during the playback (1227). Accordingly, the delay time for voice information #B (1222) may not be delayed by the time length (1228).

[0208] As described above, the processor (120) may process the acquired voice information and play (or display) an avatar (or animation) having a mouth shape corresponding to the voice information. At this time, the processor (120) may stop playing the avatar having the mouth shape corresponding to the previous voice information and perform playing the avatar having the mouth shape corresponding to the next voice information. In other words, a portion of the list (content) regarding the playback of the avatar having the mouth shape corresponding to the previous voice information may be deleted. Accordingly, the delay time experienced by the user may be reduced.

[0209] Referring to FIG. 12c, an example (1240) of playing an avatar generated by processing recorded voice information through a serial processing method and an example (1260) of playing an avatar generated by processing it through a parallel processing method are illustrated.

[0210] Referring to example (1240), the processor (120) can obtain voice information #A (1241), voice information #B (1242), and voice information #C (1243). For example, the processor (120) can record voice information #A (1241), voice information #B (1242), and voice information #C (1243) over time. The time length of each of voice information #A (1241), voice information #B (1242), and voice information #C (1243) is assumed to be 10 ms. For example, the processor (120) can start processing (1245) for voice information #A (1241) at the time (timing) of obtaining voice information #B (1242). The processor (120) may start processing (1246) for voice information #B (1242) at the time (timing) at which voice information #C (1243) is acquired. For example, the time required for processing (1245) for voice information #A (1241) and processing (1246) for voice information #B (1242) is assumed to be 8 ms. For example, the processor (120) may start playing (1247) from the time (timing) at which processing (1245) for the processed voice information #A (1241) ends. In this case, the delay time for voice information #A (1241) experienced by the user may be 18 ms (10 ms + 8 ms).

[0211] Referring to example (1260), the processor (120) can obtain voice information #A (1261), voice information #B (1262), and voice information #C (1263). For example, the processor (120) can record voice information #A (1261), voice information #B (1262), and voice information #C (1263) over time. It is assumed that the time length of each of voice information #A (1261), voice information #B (1262), and voice information #C (1263) is 10 ms. For example, the first 5 ms of voice information #A (1261) can be referred to as the first part (1261-1) (#A1), and the last 5 ms can be referred to as the second part (1261-2) (#A2). The first 5 ms of the voice information #B (1262) can be referred to as the first part (1262-1) (#B1), and the latter 5 ms can be referred to as the second part (1262-2) (#B2). The first 5 ms of the voice information #C (1263) can be referred to as the first part (1263-1) (#C1), and the latter 5 ms can be referred to as the second part (1263-2) (#C2).

[0212] For example, the processor (120) may start processing (1265) the first part (1261-1) (#A1) and the second part (1261-2) (#A2) of the voice information #A (1261) at the time (timing) of acquiring the voice information #B (1262). The processor (120) may start processing (1267) the second part (1261-2) (#A2) of the voice information #A (1261) and the first part (1262-1) (#B1) of the voice information #B (1262) at the time (timing) of acquiring the second part (1262-2) (#B2) of the voice information #B (1262). The processor (120) may start processing (1266) the first part (1262-1) (#B1) and the second part (1262-2) (#B2) of the voice information #B (1262) at the time (timing) of acquiring the first part (1263-1) (#C1) of the voice information #C (1263). The processor (120) may start processing (1268) the second part (1262-2) (#B2) of the voice information #B (1262) and the first part (1263-1) (#C1) of the voice information #C (1263) at the time (timing) of acquiring the second part (1263-2) (#C2) of the voice information #C (1263). For example, it is assumed that the time length required for processing the first part and the second part of the voice information is 8 ms.

[0213] For example, the processor (120) may start playback (1271) of the second part (1261-2) (#A2) of the voice information #A (1261) from the time point (timing) at which processing (1265) ends for the first part (1261-1) (#A1) and the second part (1261-2) (#A2) of the processed voice information #A (1261). In other words, the processor (120) may skip playback of the first part (1261-1) (#A1) of the processed voice information #A (1261) and perform playback (1271) of the second part (1261-2) (#A2) of the voice information #A (1261). In this case, the delay time for the voice information #A (1261) experienced by the user may be 13 ms (5 ms + 8 ms). An avatar (or animation) having a mouth shape for a front part of a user's speech (e.g., a first part (1261-1)(#A1) of voice information #A(1261)) may have a lower need to be recognized compared to a rear part (e.g., a second part (1261-2)(#A2) of voice information #A(1261)). This may be because the rear part is scheduled to proceed continuously after the front part. Therefore, by omitting playback of the front part and playing only the rear part, the delay time experienced by the user can be reduced.

[0214] As described above, the processor (120) can process voice information in parallel with overlapping time. For example, the processor (120) can perform parallel processing on the voice information through a single processing circuit identified based on processing speed. For example, the processor (120) can perform the parallel processing by utilizing multiple threads of the single processing circuit. However, the embodiments of the present disclosure are not limited thereto. For example, the processor (120) can perform parallel processing on the voice information by utilizing one processing circuit identified based on processing speed together with another processing circuit. The other processing circuit may be of the same or different type as the single processing circuit. For example, if the other processing circuit is of the same type as the single processing circuit, the processing speed may correspond. For example, if the other processing circuit is different from the single processing circuit, the processing speed may correspond or may differ. Additionally, while FIG. 12c illustrates an example of two parallel processing operations, the present disclosure is not limited thereto. For example, the processor (120) may perform three or more parallel processing operations. As the number of parallel processing operations increases, delay time can be reduced.

[0215] For convenience of explanation, different voice information (e.g., voice information #A, voice information #B, voice information #C) are illustrated in FIGS. 12A to 12C , but the embodiments of the present disclosure are not limited thereto. The different voice information may be different input signals. In other words, the methods of FIGS. 12A to 12C can also be applied to multiple input signals within a single voice information.

[0216] FIG. 13 illustrates an example flowchart of a method for applying a specified mouth shape to an avatar including a closed mouth shape according to various embodiments.

[0217] At least some of the above method of FIG. 13 may be performed by the electronic device (101) of FIG. 5. For example, at least some of the above method may be controlled by the processor (120) of the electronic device (101). The above method of FIG. 13 may include exemplary operations for operations (620) to (625) of FIG. 6B.

[0218] Although not illustrated in FIG. 13, before performing operation (1310), the processor (120) may obtain voice information from an external source and identify whether the voice information includes a voice. For example, the processor (120) may display the avatar corresponding to the user of the electronic device (101) in response to the execution of a software application providing the virtual environment. For example, the processor (120) may obtain voice information from an external source within the above state. For example, the processor (120) may divide the obtained voice information into a plurality of input signals. For example, each of the plurality of input signals may have a specified time length. The processor (120) may sequentially process each of the plurality of input signals. For example, the processor (120) may perform processing in the order of a first input signal to a last input signal among the plurality of input signals. For example, the processor (120) can identify whether one of the plurality of input signals includes speech. In the above example, the speech information is described as including the plurality of input signals, but the embodiments of the present disclosure are not limited thereto. For example, if the time length of the speech information corresponds to the specified time length, the speech information may be composed of a single input signal.

[0219] In operation (1310), the processor (120) may identify whether the mouth of the avatar is closed. For example, in response to identifying that the input signal includes speech, the processor (120) may identify whether the mouth of the currently displayed avatar is closed. For example, the processor (120) may identify whether another input signal exists before processing the input signal. The processor (120) may identify whether an avatar is displayed for other speech information prior to the speech information including the input signal, or whether the input signal is the first input signal within the speech information. In operation (1310), if the mouth of the avatar is closed, the processor (120) may perform operation (1320). Alternatively, in operation (1320), if the mouth of the avatar is open, the processor (120) may perform operation (1340).

[0220] In operation (1320), the processor (120) may identify a designated mouth shape based on the volume of the voice. For example, the processor (120) may identify the volume of the input signal. For example, the processor (120) may identify a designated mouth shape based on the volume of the input signal. For example, information about the designated mouth shape may be stored in the memory (130). The information about the designated mouth shape may be mapped according to the volume of the input signal. For example, the designated mouth shape may include a mouth shape for pronouncing 'schwa'. When the volume of the input signal is a first value, a first mouth shape for pronouncing 'schwa' may be identified. When the volume of the input signal is a second value greater than the first value, a second mouth shape for pronouncing 'schwa' may be identified. The second mouth shape may have a mouth that is more open than the mouth of the first mouth shape. However, the embodiments of the present disclosure are not limited thereto. For example, the specified mouth shape may include a mouth shape for uttering a syllable other than 'schwa'.

[0221] In operation (1330), the processor (120) may display an avatar including the specified mouth shape. For example, the processor (120) may synthesize the specified mouth shape with the avatar with the closed mouth displayed through the display (510) of the electronic device (101). Accordingly, the processor (120) may generate an avatar including the specified mouth shape. For example, the processor (120) may generate an animation that sequentially includes the avatar with the closed mouth and the avatar including the specified mouth shape. For example, the processor (120) may display an avatar (or the animation) including the specified mouth shape through the display (510).

[0222] In operation (1340), the processor (120) may perform voice enhancement. For example, the processor (120) may perform the voice enhancement on the input signal. Specific details regarding operation (1340) are substantially the same as operation (625) of FIG. 6B and are therefore omitted.

[0223] As described above, the electronic device and method according to the embodiment of the present disclosure can display an avatar (or animation) including a designated mouth shape based on the size of the voice from the time (timing) when the user speaks until the time (timing) when the processing of the spoken voice is completed and the avatar is played back. Accordingly, the electronic device and method according to the embodiment of the present disclosure can provide a sense of consistency with the mouth opening action by providing an avatar including an open mouth that does not match the voice spoken by the user during a short period of time (e.g., 1 second or less) from the time of the speech until the avatar is played back. By using the electronic device and method according to the embodiment of the present disclosure, the user can experience a short delay time.

[0224] FIG. 14 illustrates an example flowchart of a method for identifying an avatar's mouth shape based on the performance of multiple processing circuits according to various embodiments.

[0225] At least some of the above methods of FIG. 14 may be performed by the electronic device (101) of FIG. 5. For example, at least some of the above methods may be controlled by the processor (120) of the electronic device (101).

[0226] Referring to FIG. 14, in operation (1410), the processor (120) may identify a first processing speed of each of a plurality of processing circuits for feature value identification. For example, the processor (120) may identify the first processing speed of each of the plurality of processing circuits for processing voice data for feature value identification of the voice data. For example, the plurality of processing circuits may include a central processing unit (CPU), a graphic processing unit (GPU), or a neural processing unit (NPU). For example, the processor (120) may include the CPU. The voice data may be referred to as voice information.

[0227] In operation (1420), the processor (120) may identify a second processing speed of each of the plurality of processing circuits for lip shape identification. For example, the processor (120) may identify the second processing speed of each of the plurality of processing circuits for processing the lip shape of the lip shape of the lip data. The lip shape identification may be performed in conjunction with the feature value identification. For example, the lip shape identification may be performed based on the feature values ​​identified through the feature value identification.

[0228] Although an example of identifying the first processing speed and the second processing speed is described in FIG. 14, this is merely an example for convenience of explanation, and the embodiments of the present disclosure are not limited thereto. For example, the processor (120) may also identify the third processing speed of each of the plurality of processing circuits for voice enhancement.

[0229] For example, the first processing speed can be identified by performing the feature value identification based on reference data in each of the plurality of processing circuits. For example, the second processing speed can be identified by performing the lip shape identification based on reference data in each of the plurality of processing circuits. For example, the third processing speed can be identified by performing the voice portion enhancement based on reference data in each of the plurality of processing circuits. The reference data may represent dummy data for identifying the performance of each of the plurality of processing circuits. For example, the first processing speed, the second processing speed, and the third processing speed can each be defined as a ratio of the processing time to the time length of input data (e.g., the length of the reference data). For example, the ratio can be referred to as a real time ratio (RT). For example, the first processing speed may include the processing speed of a CPU that performs the feature value identification using an artificial model, the processing speed of an NPU that performs the feature value identification using an artificial model, the processing speed of a GPU that performs the feature value identification using an artificial model, or the processing speed of a CPU that performs the feature value identification using an MFCC (mel frequency cepstral coefficient) algorithm.

[0230] In operation (1430), the processor (120) may obtain voice information from an external source while displaying an avatar. For example, the avatar may represent a virtual object corresponding to a user within a virtual environment. For example, the virtual environment may be provided by a software application. For example, the processor (120) may display the virtual environment and the avatar within the virtual environment in response to executing the software application. For example, the processor (120) may obtain voice information from an external source of the electronic device (101) while displaying the avatar. The voice information may be referred to as voice data. For example, the voice information may include voice, noise, or background sound.

[0231] For example, the voice information may be transmitted from an external electronic device (580) through a server or system for providing the virtual environment. For example, the voice information may be acquired through a microphone of the electronic device (101) as the user of the electronic device (101) speaks. For example, the voice information may include a text input for the electronic device (101) or the external electronic device (580). For example, the text input may include a machine-synthesized voice, such as text to speech (TTS). For example, the voice information may be composed of an entire utterance, a sentence, a word, or a specified length of the user of the electronic device (101) or another user of the external electronic device (580). For example, the specified length may be defined as a specified size (e.g., n bytes) or a specified length of time. For example, the voice information may be composed of a plurality of input signals. Each of the plurality of input signals may be composed of the specified length.

[0232] In operation (1440), the processor (120) may obtain a plurality of feature values ​​of the speech information using the first processing circuit. For example, the processor (120) may identify the first processing circuit among the plurality of processing circuits based on the first processing speed. For example, the first processing circuit may include a circuit having the fastest first processing speed among the plurality of processing circuits for the feature value identification. For example, the processor (120) may perform the feature value identification on the speech information based on the first processing circuit.

[0233] For example, the processor (120) can extract feature values ​​from the voice information, which is an analog signal. For example, the processor (120) can obtain the feature values ​​based on the MFCC (mel frequency cepstral coefficient) algorithm. For example, the processor (120) can obtain a spectrum by applying a fast-Fourier transform (FFT) to the voice information for each frame. For example, the processor (120) can obtain the spectrum for the frequency domain by applying the FFT to the voice information. The processor (120) can obtain a mel spectrum by applying a mel filter bank to the spectrum. For example, the processor (120) can obtain the mel spectrum based on a mel scale that represents the relationship between the frequency domain and a low frequency band recognized by an actual human. The processor (120) can obtain MFCCs by applying cepstral analysis to the mel spectrum. The MFCCs can be referred to as the feature values. For example, the processor (120) can obtain the feature values, which are a portion of the total feature values, which are peaks obtained based on the cepstral analysis. The peaks can be referred to as formants. For example, the feature values ​​can be 40. However, the embodiments of the present disclosure are not limited thereto. For example, the number of feature values ​​can be less than 40 or greater than 40.

[0234] For example, the processor (120) can learn an artificial intelligence model based on the acquired feature values. In other words, the processor (120) can learn the artificial intelligence model by inputting the feature values. Accordingly, the processor (120) can obtain refined feature values. In the above-described example, the processor (120) obtains the feature values ​​based on the MFCC algorithm, and exemplifies a method of using the feature values ​​without additional processing or refining the feature values ​​using the artificial intelligence model, but the embodiments of the present disclosure are not limited thereto.

[0235] For example, the processor (120) may obtain the feature values ​​without the MFCC algorithm based on the speech information by using the artificial intelligence model. For example, if a processing circuit (e.g., NPU, GPU) having a relatively fast processing speed is available among the plurality of processing circuits of the processor (120), the processor (120) may obtain the feature values ​​by using the artificial intelligence model. Alternatively, if a processing circuit (e.g., CPU) having a relatively slow processing speed is available among the plurality of processing circuits of the processor (120), the processor (120) may obtain the feature values ​​by using the MFCC algorithm.

[0236] As described above, the processor (120) may identify the plurality of feature values ​​based on the feature value identification performed using the MFCC algorithm and / or the artificial intelligence model based on the first processing circuit. For example, the first processing circuit may include a CPU capable of utilizing the MFCC algorithm or the artificial intelligence model. For example, the first processing circuit may include a GPU capable of utilizing the artificial intelligence model. For example, the first processing circuit may include an NPU capable of utilizing the artificial intelligence model.

[0237] Although not illustrated in FIG. 14, before operation (1440) is performed, the processor (120) may perform voice enhancement. For example, the voice enhancement may include removing noise from the voice information, enhancing the voice portion of the voice information, and normalizing the volume of the voice portion.

[0238] In operation (1450), the processor (120) may obtain information for generating a mouth shape using a second processing circuit. For example, the processor (120) may identify the second processing circuit among the plurality of processing circuits based on the second processing speed. For example, the second processing circuit may include a circuit among the plurality of processing circuits having the fastest second processing speed for the mouth shape identification. For example, the processor (120) may perform the mouth shape identification on the voice information based on the second processing circuit.

[0239] For example, the processor (120) may obtain information for generating the mouth shape based on the plurality of feature values ​​using the second processing circuit. The mouth shape may include visual information indicating that the voice of the voice information is being uttered.

[0240] For example, the information for generating the mouth shape may include at least one of a visual phoneme (viseme), a face landmark, a blend weight, or a face mesh for the speech information. For example, the visual phoneme may represent a mouth shape symbol of an avatar indicating that the speech of the speech information is being uttered. For example, the face landmark may represent coordinates for the face of the avatar indicating that the speech of the speech information is being uttered. For example, the face landmark may include three-dimensional coordinates or two-dimensional coordinates. The blend weight may represent an emotion parameter for changing the facial expression of the avatar. For example, the blend weight may be acquired based on a retargeting model. For example, the blend weight may be acquired from the face landmark or the speech information. For example, the face mesh may represent a mesh formed by points of the facial feature points.

[0241] For example, the processor (120) may obtain information for generating the mouth shape using an artificial intelligence model. For example, the processor (120) may utilize the artificial intelligence model based on the second processing circuit. For example, the second processing circuit may be one of a CPU, a GPU, and an NPU.

[0242] In operation (1460), the processor (120) may display an avatar including the generated mouth shape. For example, the processor (120) may generate an avatar including the generated mouth shape based on information for generating the mouth shape. For example, the processor (120) may generate an animation including an avatar having the mouth shape. For example, the animation may display visual information including the virtual environment and the avatar for a time corresponding to a plurality of frames. For example, the plurality of frames may be referenced as playback frames set for the animation. The animation may include an avatar having the mouth shape of each of the plurality of frames. The plurality of frames may include key frames of a specified period. For example, the processor (120) may generate an avatar having the mouth shape for each of the plurality of frames, or generate an avatar having the mouth shape for each of the key frames.

[0243] For example, the processor (120) may play the avatar or the animation through the display (510). At this time, the processor (120) may change the playback speed, delete some of the playlist (content), or use a parallel processing method to minimize the delay time felt by the user.

[0244] Although not illustrated in FIG. 14, the processor (120) may identify whether the mouth of the currently displayed avatar is closed before displaying the avatar including the mouth shape generated based on the voice information. For example, the currently displayed avatar may be displayed as the processor (120) executes the software application providing the virtual environment. The time before displaying the avatar may include the time before the processor (120) acquires the voice information and then performs processing on the acquired voice information. For example, if the mouth is closed, the processor (120) may display the avatar having a designated mouth shape based on the volume of the voice information through the display (510). In other words, if the currently displayed avatar does not open its mouth and voice information spoken by the user is acquired, the avatar having a designated mouth shape based on the volume of the voice information may be displayed in order to reduce a delay that the user may experience.

[0245] Referring to FIGS. 1 to 14, an electronic device and method according to an embodiment of the present disclosure can provide a video call service using an avatar having a mouth shape generated based on voice information in an environment where a video call using a camera is not possible. In addition, the electronic device and method according to an embodiment of the present disclosure can be applied not only to user equipment such as a smartphone, but also to wearable devices such as an HMD (e.g., wearable device (101-1)). When using the wearable device, there may be limitations in directly acquiring information about the user's face. Accordingly, the electronic device and method according to an embodiment of the present disclosure can provide a virtual environment service using an avatar having a mouth shape generated based on voice information. The electronic device and method according to an embodiment of the present disclosure can identify an optimal processing algorithm and a processing circuit for processing the processing algorithm for body gesture or emotion estimation as well as a mouth shape from voice information. Accordingly, the electronic device and method according to the embodiment of the present disclosure can provide a real-time service utilizing the acquired voice information.

[0246] An electronic device and method optimized for each electronic device are provided for generating an avatar based on real-time voice information according to an embodiment of the present disclosure. The electronic device and method according to an embodiment of the present disclosure can quickly and flexibly reduce lip sync even in an internal environment (or on-device environment) of the electronic device. In other words, the electronic device and method according to an embodiment of the present disclosure can generate an avatar (or an avatar's mouth shape, or an animation including an avatar having a mouth shape) with higher accuracy and speed by monitoring and efficiently utilizing resources within the electronic device. Accordingly, the electronic device and method according to an embodiment of the present disclosure can provide a more immersive user experience to the user. Furthermore, the electronic device and method according to an embodiment of the present disclosure can secure real-time performance even in a multi-tasking environment through optimized operations for generating an avatar having a mouth shape based on voice during the runtime of the electronic device. In addition, the electronic device and method according to the embodiment of the present disclosure can reduce the overall resource usage by utilizing only the resources of the electronic device itself (on-device) and not using the resources of the server providing the virtual environment and additional resources (e.g., data) for connection with the server.

[0247] As described above, the electronic device (101) may include a display (510). The electronic device (101) may include at least one processor (120). The at least one processor (120) may be configured to identify a first processing speed of each of a plurality of processing circuits for processing voice data with respect to identifying a feature value of the voice data. The at least one processor (120) may be configured to identify a second processing speed of each of the plurality of processing circuits with respect to identifying a mouth shape of the voice data, which is linked to identifying the feature value. The at least one processor (120) may be configured to acquire voice information from an external source of the electronic device (101) while displaying an avatar. The at least one processor (120) may be configured to obtain a plurality of feature values ​​of the voice information using a first processing circuit identified based on the first processing speed among the plurality of processing circuits. The at least one processor (120) may be configured to obtain information for generating a mouth shape based on the plurality of feature values ​​using a second processing circuit identified based on the second processing speed among the plurality of processing circuits. The at least one processor (120) may be configured to display the avatar including the mouth shape generated based on the information through the display (510).

[0248] According to one embodiment, the plurality of processing circuits may include a central processing unit (CPU), a graphic processing unit (GPU), or a neural processing unit (NPU). The at least one processor (120) may include the CPU.

[0249] According to one embodiment, the at least one processor (120) may be configured to obtain information about the plurality of processing circuits. The information about the plurality of processing circuits may include at least one of information indicating whether the NPU or the GPU is included in the electronic device (101), or information indicating the manufacturer of the CPU.

[0250] According to one embodiment, the at least one processor (120) may be configured to acquire the information based on the framework of the artificial intelligence model during the runtime of the artificial intelligence model.

[0251] According to one embodiment, the at least one processor (120) may be configured to identify that the plurality of processing circuits include the NPU or the GPU based on information indicating whether the NPU or the GPU is included in the electronic device (101). The first processing speed may include a processing speed for the feature value identification performed by the artificial intelligence model in the NPU, a processing speed for the feature value identification performed by the artificial intelligence model in the GPU, a processing speed for the feature value identification performed by the artificial intelligence model in the CPU, or a processing speed for the feature value identification performed using MFCC (mel frequency cepstral coefficient) in the CPU.

[0252] According to one embodiment, the at least one processor (120) may be configured to identify the first processing circuit based on the first processing speed in response to identifying that the plurality of processing circuits include the NPU or the GPU. The plurality of feature values ​​may be obtained based on the artificial intelligence model or the MFCC.

[0253] According to one embodiment, the at least one processor (120) may be configured to identify the first processing circuit, which is the CPU, in response to identifying that the plurality of processing circuits do not include the NPU and the GPU. The plurality of feature values ​​may be obtained based on the MFCC.

[0254] According to one embodiment, the at least one processor (120) may be configured to identify the first processing speed of each of the plurality of processing circuits by performing the feature value identification based on reference data. The at least one processor (120) may be configured to identify the second processing speed of each of the plurality of processing circuits by performing the mouth shape identification based on the reference data.

[0255] According to one embodiment, the at least one processor (120) may be configured to generate a plurality of input signals from the acquired voice information. Each of the plurality of input signals may be formed with a specified time length. The specified time length may be identified based on a delay time between the timing at which the voice information is acquired and the timing at which the avatar is displayed.

[0256] According to one embodiment, the at least one processor (120) may be configured to identify, during the designated length of time corresponding to the first input signal among the plurality of input signals, whether the first input signal includes speech. The at least one processor (120) may be configured, in response to the first input signal including the speech, to obtain the plurality of feature values ​​for the first input signal. The at least one processor (120) may be configured, in response to identifying that the first input signal does not include speech, to identify whether the plurality of input signals include a second input signal subsequent to the first input signal.

[0257] In one embodiment, the at least one processor (120) may be configured, in response to identifying that the first input signal includes the speech, to identify whether the mouth of the avatar within the state is closed. The at least one processor (120) may be configured, in response to identifying that the mouth is closed, to display, via the display (510), within the state, the avatar including a designated mouth shape based on a volume of the speech of the first input signal.

[0258] According to one embodiment, the at least one processor (120) may be configured to, in response to identifying that the first input signal is a final input signal, display the avatar including a mouth shape for the first input signal, and then display the avatar including a mouth shape indicating a closed mouth.

[0259] In one embodiment, the at least one processor (120) may be configured to, in response to identifying that the plurality of input signals include the second input signal, acquire a processing speed of at least one processing circuit used to acquire the mouth shape for the first input signal. The at least one processor (120) may be configured to identify the first processing speed and the second processing speed for the second input signal based on the processing speed of the at least one processing circuit.

[0260] According to one embodiment, the at least one processor (120) may be configured to identify a first input signal, a second input signal subsequent to the first input signal, and a third input signal subsequent to the second input signal among the plurality of input signals. The at least one processor (120) may be configured to perform the mouth shape identification for the first portion of the first input signal and the second portion of the first input signal from a timing at which the third portion of the second input signal begins to be acquired. The at least one processor (120) may be configured to perform the mouth shape identification for the second portion of the first input signal and the third portion of the second input signal from a timing at which the fourth portion of the second input signal begins to be acquired. The at least one processor (120) may be configured to, in response to completion of the mouth shape identification for the first portion and the second portion, display the avatar including the mouth shape for the second portion through the display (510). The at least one processor (120) may be configured to, in response to completion of the mouth shape identification for the second portion and the third portion, display, through the display (510), the avatar including the mouth shape for the third portion, which is continuous with the avatar including the mouth shape for the second portion. The second portion may be a time subsequent to the first portion within the designated time interval of the first input signal. The fourth portion may be a time subsequent to the third portion within the designated time interval of the second input signal.

[0261] According to one embodiment, the at least one processor (120) may be configured to identify a third processing speed of each of the plurality of processing circuits for voice enhancement of the voice data. The at least one processor (120) may be configured to perform noise removal of the voice information. The at least one processor (120) may be configured to perform enhancement of a voice portion of the voice information on which the noise removal has been performed using a third processing circuit identified based on the third processing speed among the plurality of processing circuits. The at least one processor (120) may be configured to adjust a volume of the voice information including the enhanced voice portion. The plurality of feature values ​​may be acquired for the voice information having the adjusted volume.

[0262] According to one embodiment, the at least one processor (120) may be configured to identify a mapping value for a visual phoneme (viseme) identified based on the plurality of feature values. The at least one processor (120) may be configured to identify information for generating the mouth shape based on a weight value identified based on the mapping value. The information for generating the mouth shape identified based on the weight value may include a face mesh.

[0263] According to one embodiment, the at least one processor (120) may be configured to identify a face landmark identified based on the plurality of feature values. The at least one processor (120) may be configured to identify information for generating the mouth shape based on the face landmark. The face landmark may include three-dimensional coordinate information or two-dimensional coordinate information. The information for generating the mouth shape identified based on the face landmark may include a face mesh.

[0264] According to one embodiment, the at least one processor (120) may be configured to identify information for generating the mouth shape based on a weight value identified based on the plurality of feature values. The information for generating the mouth shape identified based on the weight value may include a face mesh.

[0265] According to one embodiment, the at least one processor (120) may be configured to identify frames for playing an animation including the avatar. The at least one processor (120) may be configured to display the animation through the display (510). The mouth shape of the avatar may be acquired for each of the frames, or may be acquired for frames corresponding to a specified period among the frames.

[0266] As described above, the method performed by the electronic device (101) may include an operation of identifying a first processing speed of each of a plurality of processing circuits for processing the voice data with respect to identifying a feature value of the voice data. The method may include an operation of identifying a second processing speed of each of the plurality of processing circuits with respect to identifying a mouth shape of the voice data, which is linked to identifying the feature value. The method may include an operation of acquiring voice information from an external source of the electronic device (101) while displaying an avatar. The method may include an operation of acquiring a plurality of feature values ​​of the voice information using a first processing circuit identified based on the first processing speed among the plurality of processing circuits. The method may include an operation of acquiring information for generating a mouth shape based on the plurality of feature values ​​using a second processing circuit identified based on the second processing speed among the plurality of processing circuits. The method may include an operation of displaying the avatar including the mouth shape generated based on the information through the display (510).

[0267] The non-transitory computer-readable storage medium as described above may store one or more programs including instructions that, when executed by at least one processor (120) of an electronic device (101) including a display (510), cause the electronic device (101) to identify a first processing speed of each of a plurality of processing circuits for processing voice data with respect to feature value identification of the voice data. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the at least one processor (120), cause the electronic device (101) to identify a second processing speed of each of the plurality of processing circuits with respect to mouth shape identification of the voice data, which is linked to feature value identification. A non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the at least one processor (120), cause the electronic device (101) to acquire voice information from an external source while displaying an avatar. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the at least one processor (120), cause the electronic device to acquire a plurality of feature values ​​of the voice information using a first processing circuit identified based on the first processing speed among the plurality of processing circuits. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the at least one processor (120), cause the electronic device to acquire information for generating a mouth shape based on the plurality of feature values ​​using a second processing circuit identified based on the second processing speed among the plurality of processing circuits.A non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the at least one processor (120), cause the avatar, including the mouth shape generated based on the information, to be displayed through the display (510).

[0268] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0269] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0270] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0271] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0272] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0273] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101), display (510); A memory (130) storing instructions and including one or more storage media; and At least one processor (120) comprising processing circuitry, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: For identifying feature values ​​of voice data, a first processing speed of each of a plurality of processing circuits for processing the voice data is identified; Identifying the second processing speed of each of the plurality of processing circuits for the mouth shape identification of the voice data, which is linked to the identification of the above feature value; In a state where an avatar is displayed, voice information is acquired from outside the electronic device (101); Using a first processing circuit identified based on the first processing speed among the plurality of processing circuits, a plurality of feature values ​​of the voice information are obtained; Using a second processing circuit identified based on the second processing speed among the plurality of processing circuits, information for generating a mouth shape based on the plurality of feature values ​​is obtained; and Causing the avatar, including the mouth shape generated based on the above information, to be displayed through the display (510). Electronic devices (101).

2. In claim 1, The above multiple processing circuits include a central processing unit (CPU), a graphic processing unit (GPU), or a neural processing unit (NPU). The at least one processor (120) comprises the CPU, Electronic devices (101).

3. In claim 2, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: To cause information about the above plurality of processing circuits to be obtained, The information about the plurality of processing circuits includes at least one of information indicating whether the NPU or the GPU is included in the electronic device (101), or information indicating the manufacturer of the CPU. Electronic devices (101).

4. In claim 3, The above instructions, when individually or collectively executed by at least one processor (120), cause the electronic device (101) to: During the runtime of the artificial intelligence model, causing the information to be acquired based on the framework of the artificial intelligence model. Electronic devices (101).

5. In claim 2, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: Causing the plurality of processing circuits to identify that the NPU or the GPU is included based on information indicating whether the NPU or the GPU is included in the electronic device (101); The first processing speed includes a processing speed for the feature value identification performed by the artificial intelligence model in the NPU, a processing speed for the feature value identification performed by the artificial intelligence model in the GPU, a processing speed for the feature value identification performed by the artificial intelligence model in the CPU, or a processing speed for the feature value identification performed using MFCC (mel frequency cepstral coefficient) in the CPU. Electronic devices (101).

6. In claim 5, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: In response to identifying that said plurality of processing circuits include said NPU or said GPU, causing said first processing circuit to be identified based on said first processing speed. The above multiple feature values ​​are obtained based on the artificial intelligence model or the MFCC. Electronic devices (101).

7. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: By performing the feature value identification based on the reference data, the first processing speed of each of the plurality of processing circuits is identified, and By performing the lip shape identification based on the above reference data, causing each of the plurality of processing circuits to identify the second processing speed, Electronic devices (101).

8. In claim 1, The above instructions, when individually or collectively executed by at least one processor (120), cause the electronic device (101) to: To cause multiple input signals to be generated from the acquired voice information, Each of the above multiple input signals is formed with a specified time length, and The above specified time length is identified based on the delay time between the timing at which the voice information is acquired and the timing at which the avatar is displayed. Electronic devices (101).

9. In claim 8, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: During the specified time length corresponding to a first input signal among the plurality of input signals, identifying whether the first input signal includes speech; In response to the first input signal including the voice, obtaining the plurality of feature values ​​for the first input signal, and In response to identifying that the first input signal does not include the speech, causing the plurality of input signals to identify whether the second input signal follows the first input signal. Electronic devices (101).

10. In claim 9, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: In response to identifying that said first input signal comprises said speech, identifying whether said avatar's mouth within said state is closed, and In response to identifying that the mouth is closed, causing the avatar to be displayed, through the display (510), within the state, having a designated mouth shape based on the volume of the voice of the first input signal. Electronic devices (101).

11. In claim 9, The above instructions, when individually or collectively executed by at least one processor (120), cause the electronic device (101) to: In response to identifying that the first input signal is the last input signal, causing the avatar to be displayed including a mouth shape for the first input signal, and then causing the avatar to be displayed including a mouth shape indicating a closed mouth. Electronic devices (101).

12. In claim 11, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: In response to identifying that the plurality of input signals include the second input signal, obtaining a processing speed of at least one processing circuit used to obtain the mouth shape for the first input signal, and causing, based on the processing speed of said at least one processing circuit, to identify said first processing speed and said second processing speed for said second input signal; Electronic devices (101).

13. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: For voice enhancement of the above voice data, identify the third processing speed of each of the plurality of processing circuits, Perform noise removal of the above voice information, Using a third processing circuit identified based on the third processing speed among the plurality of processing circuits, reinforcement of the voice portion of the voice information on which the noise removal has been performed is performed, and To cause the volume of the voice information including the enhanced voice portion to be adjusted, The above multiple feature values ​​are obtained for the voice information having the adjusted size. Electronic device (101).

14. In a method performed by an electronic device (101), The above method: An operation for identifying a first processing speed of each of a plurality of processing circuits for processing the voice data, with respect to identifying a feature value of the voice data; An operation of identifying a second processing speed of each of the plurality of processing circuits, with respect to the mouth shape identification of the voice data, which is linked to the identification of the feature value; An operation of acquiring voice information from outside the electronic device (101) while displaying an avatar; An operation of obtaining a plurality of feature values ​​of the voice information by using a first processing circuit identified based on the first processing speed among the plurality of processing circuits; An operation of obtaining information for generating a mouth shape based on the plurality of feature values ​​by using a second processing circuit identified based on the second processing speed among the plurality of processing circuits; and Including an action of displaying the avatar including the mouth shape generated based on the above information through the display (510) of the electronic device (101). method.

15. A non-transitory computer-readable storage medium, when executed individually or collectively by at least one processor (120) of an electronic device (101) including a display (510): For identifying feature values ​​of voice data, a first processing speed of each of a plurality of processing circuits for processing the voice data is identified; Identifying the second processing speed of each of the plurality of processing circuits for the mouth shape identification of the voice data, which is linked to the identification of the above feature value; In a state where an avatar is displayed, voice information is acquired from outside the electronic device (101); Using a first processing circuit identified based on the first processing speed among the plurality of processing circuits, a plurality of feature values ​​of the voice information are obtained; Using a second processing circuit identified based on the second processing speed among the plurality of processing circuits, information for generating a mouth shape based on the plurality of feature values ​​is obtained; and storing one or more programs including instructions causing the avatar, including the mouth shape generated based on the above information, to be displayed through the display (510); A non-transitory computer-readable storage medium.

Citation Information

Patent Citations

  • Facial Animation for Social Virtual Reality (VR)

    JP2021505943A

  • Manufacturing method of pretreated carbon nanotube, carbon nanotube manufactured therefrom

    KR1020220110136A

  • English grammar correction system using robot

    KR1020240030841A

  • Synchronization animation output system of virtual characters and text

    KR102116309B1

  • KR20220040378A