Method and system for generating avatar for communication and interaction with virtual world and computer readable medium thereof

Abstract

This invention provides a method and system for generating virtual avatars that connect to and interact with the metaverse, used for voice calls between a user in the metaverse and a counterparty outside the metaverse. The method and system determine the identity of the counterparty based on their communication information, and then determine the counterparty's virtual avatar within the metaverse. Furthermore, the method and system can cause the virtual avatar to perform corresponding facial expressions and actions based on the counterparty's voice data. Moreover, the method and system can cause the virtual avatar to perform corresponding actions or change its state based on the interaction between the user and the virtual avatar, and based on scenes or activities in the virtual space of the metaverse. This invention also provides a computer-readable medium for performing the method of this invention.

Description

Technical Field

[0001] This invention relates to virtual avatars in the metaverse, and more particularly to a method, system, and computer-readable medium for generating virtual avatars that connect to metaverse interactions. Prior Technology

[0002] As the name suggests, the metaverse is a virtual world where people log in as virtual avatars to engage in various activities. Users in the metaverse sometimes receive or need to actively contact people in the real world; therefore, it is necessary to create virtual avatars for the other party in the real world and conduct voice calls.

[0003] However, existing technologies only focus on binding digital information with virtual avatar data. Furthermore, communication within the metaverse and the exchange of text messages or image data are only possible when both parties are logged into the metaverse. Existing technologies do not reveal how to proactively contact or receive communication intentions from real-world individuals within the metaverse based on the user's needs, nor do they reveal how to allow users to interact and communicate with those individuals through virtual avatars. Summary of the Invention

[0004] To address the aforementioned problems, the present invention provides a method for generating a virtual avatar that connects to a metaverse for interaction, comprising: having a user's electronic device in the metaverse conduct a voice call with a caller outside the metaverse; determining the caller's virtual avatar in the metaverse based on the caller's communication information; having the electronic device display the virtual space of the metaverse and the virtual avatar; and having the virtual avatar perform corresponding actions or change its state based on the interaction between the user and the virtual avatar, and based on the scene or activity in the virtual space.

[0005] The present invention also provides a system for generating virtual avatars to connect with the metaverse for interaction, including a voice transceiver module, a user data module, a virtual avatar module, an facial expression and action module, and an avatar interaction module. The voice transceiver module is used for voice communication between a user in the metaverse and a caller outside the metaverse; the user data module is used to determine the identity of the caller based on the caller's communication information; the virtual avatar module is used to determine the caller's virtual avatar in the metaverse based on the identity, and to display the virtual avatar on the user's electronic device; the facial expression and action module is used to make the virtual avatar perform corresponding facial expressions and actions based on the caller's voice data during the voice call; the avatar interaction module is used to make the virtual avatar perform corresponding movement actions, response actions, behavioral actions, or change states based on the interaction between the user and the virtual avatar, and based on the scene or activity in the virtual space of the metaverse.

[0006] The above-mentioned method and system further include: searching for the user information of the other party in the call in the database of the metaverse based on the communication information, in order to determine the identity of the other party in the call, and then reading the corresponding model as the virtual avatar.

[0007] The above-described method and system further include: analyzing the volume data in the voice data; converting the voice data into text data to extract keywords from the text data; and causing the virtual avatar to perform corresponding facial expressions based on the volume data and the keywords.

[0008] In the above method and system, the interaction refers to the user's movement in the virtual space, and the method further includes: causing the virtual avatar to follow the user's movement.

[0009] The above-described method and system further include: calculating the user's position in the virtual space and the perspective that changes when the user moves; and updating the position and facing direction of the virtual avatar according to the position and the perspective, so that the virtual avatar makes a movement action to follow the user's movement and faces the user.

[0010] In the above method and system, the interaction refers to the user giving an instruction to the virtual avatar, and the method further includes: causing the virtual avatar to perform a response action to the instruction to the user.

[0011] The above method and system further include: when the virtual avatar needs to perform multiple actions or change to multiple states, adjusting the animation rendering order of the virtual avatar according to the weight value of the multiple actions or multiple states.

[0012] The above method and system further include: allowing the one with a higher weight value in the complex action or the complex state to override the one with a lower weight value.

[0013] The present invention further discloses a computer-readable medium, which is used in a computing device or computer, and stores instructions to execute the aforementioned method for generating virtual avatars and connecting to the metaverse for interaction.

[0014] This invention provides a method, system, and computer-readable medium for generating virtual avatars that connect to the metaverse for interaction. When a user logs into the metaverse, the intent of a conversation from the real world can be converted into a virtual avatar and displayed in the virtual space. Furthermore, the user can converse with the other party through the virtual avatar. The virtual avatar will follow the user, respond to the user's actions, or interact with the current virtual scene, thereby achieving the purpose of connecting the metaverse and the real world through voice communication.

[0015] Furthermore, the methods, systems, and computer-readable media described above in this invention enable metaverse users to actively or passively connect with the real world. In addition, metaverse users do not need to switch applications and interrupt their immersive metaverse experience, and real-world personnel can establish communication connections with users in the metaverse without logging into the metaverse, allowing them to mutually understand the situation in both the virtual and real worlds. Simple Explanation of the Diagram

[0016] Figures 1 to 3 are flowcharts of the method for generating virtual avatars and connecting metaverse interactions according to the present invention.

[0017] Figure 4 is a block diagram of the system for generating virtual avatars and connecting metaverse interactions according to the present invention. Implementation

[0018] Figure 1 is a flowchart of the method for generating virtual avatars to connect with the metaverse interaction according to the present invention. In one embodiment, the method is executed collaboratively by the electronic device of the metaverse user and the metaverse server.

[0019] In step S100, the Metaverse application is run on the user's electronic device. In one embodiment, the electronic device may be a smartphone or a virtual reality (VR) headset, such as VR glasses or a VR helmet, and the Metaverse application may be downloaded and installed from the Internet or built into the electronic device.

[0020] In step S200, the metaverse application displays the user interface and three-dimensional space on the display of the electronic device to present an immersive virtual space image of the scene where the metaverse user is located.

[0021] In step S300, the Metaverse application receives the call intention of the other party or sends a call intention to the other party, thereby starting a voice call.

[0022] In one embodiment, the call method in step S300 can be achieved through a device with mobile or landline phone functionality, using communication networks, services, or technologies such as 3G / 4G / 5G cellular networks, Public Switched Telephone Network (PSTN), or Voice over LTE (VoLTE). The user's Metaverse application responds to the other party's voice call, or the user instructs the Metaverse application to make a voice call to the other party.

[0023] To conduct the aforementioned voice calls, the electronic device running the Metaverse application must possess a Subscriber Identity Module (SIM) and comply with the Global System for Mobile Communications (GSM) standard. Furthermore, if the VR headset running the Metaverse application does not have a SIM, it can establish a session with another electronic device that does have a SIM via short-range communication protocols such as Bluetooth to transmit call intent and subsequent voice call data. Moreover, the Metaverse application requires authorization from the Metaverse user to monitor and process all call intents sent and received by the aforementioned electronic device.

[0024] In step S400, based on the communication information of the other party (such as the other party's phone number), the user information such as the other party's name, identity, and virtual avatar model is searched in the metaverse database to read the corresponding model as the other party's virtual avatar in the metaverse.

[0025] If the other party in the call is an official user of the Metaverse, such as a representative of a formal entity like a company, government agency, academic institution, or non-profit organization, then the official avatar registered by that entity in the Metaverse can be used as the virtual avatar of the other party in the call.

[0026] If the other party in the call is a general user of the metaverse, that is, a non-official user, then the avatar freely edited by the other party can be used as the virtual avatar of the other party in the call.

[0027] If the user information of the other party cannot be found in the Metaverse database, it means that the other party is not a user of the Metaverse. In this case, a default stranger avatar can be used as the virtual avatar of the other party.

[0028] In step S500, the user's electronic device displays a call interface and renders a virtual avatar of the other party, displaying the virtual avatar in an image within the virtual space. In one embodiment, the virtual avatar appears at an appropriate location within the user's field of vision and integrates into the immersive virtual scene.

[0029] In step S600, the metaverse user conducts a voice call with the other party, and at the same time, the virtual avatar of the other party makes corresponding facial expressions and actions according to the content of the voice call.

[0030] In step S700, the virtual avatar of the other party in the call follows the virtual avatar of the user in the virtual space of the metaverse, and the virtual avatar of the other party in the call performs corresponding actions in accordance with the behavior of the user's virtual avatar, the scene in which the user is located in the virtual space, and / or the activities held in the virtual space.

[0031] Figure 2 is a detailed flowchart of step S600 in Figure 1, as explained below.

[0032] In step S601, the Metaverse application running on the Metaverse user's electronic device conducts a voice call with the other party.

[0033] In step S602, the voice data from the other party in the voice call is received and converted into text data.

[0034] In step S603, a text message interface is generated in the virtual space image displayed by the electronic device, and the aforementioned text data is displayed in the message interface.

[0035] In step S604, the volume values ​​in the voice data are analyzed, and keywords contained in the text data are extracted, such as keywords expressing emotions like "haha," "so funny," and "infuriating." In one embodiment, the aforementioned keywords can be pre-set in the metaverse application and then compared, or dynamically determined and inferred through artificial intelligence and other technologies to know the emotions and actions contained or implied in the text data.

[0036] In step S605, the virtual avatar of the other party in the call makes corresponding facial expressions based on the volume value and keywords. In one embodiment, the facial expressions may include movements of at least one of the eyes, mouth, face, and limbs to convey the emotions expressed by the volume value and keywords.

[0037] In step S606, the Metaverse application displays a virtual space image on the display of the electronic device, wherein the virtual space image includes the aforementioned text data and the facial expressions and actions of the virtual avatar of the other party in the call.

[0038] Figure 3 is a detailed flowchart of step S700 in Figure 1.

[0039] In this embodiment, step S700 specifically involves changing the appearance, actions, or state of the virtual avatar of the other party in the call, or restricting its actions or state, such as sitting in a specific position, standing continuously, moving, or performing specific actions. For example, when a user is watching a movie, the virtual avatar of the other party sits next to the user. Or, for example, when walking on a street in a virtual world, the virtual avatar walks alongside the user. This can also be implemented in various types of scenes in the virtual world, such as performance halls, formal venues, streets, open tourist attractions, sports fields, or in various specific events, such as concerts, festivals, and competitions. The purpose is to facilitate communication between the two parties in the virtual world without interfering with their current activities; therefore, it is necessary to integrate the virtual avatar into the virtual scene. Details of step S700 are explained below.

[0040] In step S701, the Metaverse application displays images of the virtual space and virtual avatars on the display of the electronic device.

[0041] In step S702, the position of the metaverse user in the virtual space and the perspective changes when moving are calculated.

[0042] In step S703, based on the aforementioned position and perspective, the position and facing direction of the virtual avatar of the other party in the call are updated so that the virtual avatar of the other party in the call can make a movement action to follow the metaverse user and face the user.

[0043] In step S704, the metaverse user commands their virtual avatar to actively give instructions to the virtual avatar of the other party in the call.

[0044] In step S705, the virtual avatar of the other party in the call performs a response action to the command action in response to the metaverse user. For example, if the command action of the metaverse user is to wave, the response action of the other party in the call can be the same wave.

[0045] In step S706, the scene or activity in the virtual space controls the behavior of the virtual avatar of the other party in the call.

[0046] In step S707, the virtual avatar of the other party in the call is controlled by the above-mentioned scenario or activity to perform corresponding actions and / or change its status.

[0047] For example, if the scene is a park, the virtual avatar of the caller and the virtual avatar of the metaverse user can stand together, and the virtual avatar of the caller and the virtual avatar of the metaverse user can face each other.

[0048] For example, if the scenario is a swimming pool, the virtual avatar of the other party in the call and the virtual avatar of the metaverse user can be made to float in the water wearing swimming rings, and the virtual avatar of the other party in the call and the virtual avatar of the metaverse user can be made to face each other.

[0049] For example, if the event is a concert, the virtual avatar of the person making the call can sit next to the virtual avatar of the user in the metaverse, and both the virtual avatar of the person making the call and the virtual avatar of the user in the metaverse can face the stage.

[0050] The virtual avatar of the other party in the call may receive more than one of the aforementioned movement actions, response actions, behavioral actions, and states. Therefore, in step S708, the aforementioned actions and states received by the virtual avatar of the other party in the call are sorted and merged.

[0051] In step S709, conflicting actions and states are processed, and the animation rendering order of the virtual avatar of the other party in the call is adjusted according to the weight of the aforementioned actions and states, or actions or states with higher weights are made to override those with lower weights, so as to improve its performance. For example, the weight of actions and states caused by scenes and activities in the virtual space can be set to be higher than the weight of actions and states caused by the metaverse user.

[0052] In step S710, the Metaverse application displays the adjusted virtual avatar animation on the display of the electronic device to present the interaction between the virtual avatar of the other party in the call and the Metaverse user and the virtual space.

[0053] Figure 4 is a block diagram of the system for generating virtual avatars and connecting metaverse interactions according to the present invention.

[0054] In one embodiment, the system includes a voice transceiver module M100, a voice analysis module M200, an facial expression and motion module M300, and a privacy dialogue module M400, all connected in a serial communication manner. The system also includes a user data module M500, a virtual avatar module M600, an avatar interaction module M700, and a privacy interaction module M800, all connected in a serial communication manner, wherein the virtual avatar module M600 is also communicatively connected to the facial expression and motion module M300.

[0055] In addition, the system includes a follow module M710 and a response module M720, wherein the avatar interaction module M700 communicates with the follow module M710 and the response module M720.

[0056] In Figure 4, each module can be software, hardware, or firmware. If it is hardware, it can be a processing unit, processor, computer, or server with data processing and computing capabilities. If it is software or firmware, it can include instructions executable by the processing unit, processor, computer, or server, and can be installed on the same hardware device or distributed across multiple different hardware devices. The aforementioned hardware devices can include the electronic devices of the Metaverse user and the Metaverse server.

[0057] The following describes each module in Figure 4. The technical details of each module in Figure 4 can be interchanged and / or combined with the technical details of each step in Figures 1 to 3.

[0058] The voice transceiver module M100 is installed on the electronic device of the user of Metaverse to control the microphone of the electronic device to receive voice messages and control the audio equipment of the electronic device to play the voice messages, so as to conduct the aforementioned voice calls.

[0059] The M200 voice analysis module converts the voice data of the other party in a call into text data. This voice data may come from a natural person's speech or from a preset answering voice message. The converted text data is then displayed as a message on the screen of the electronic device.

[0060] In addition, the M200 voice analysis module dynamically analyzes the volume values ​​in the voice data and extracts keywords from the text data.

[0061] The M300 facial expression and motion module drives the facial expressions and motions of the virtual avatar of the other party in the call based on the volume value and the keyword, and applies the facial expressions and motions to the animation of the virtual avatar so that the virtual avatar makes corresponding facial expressions and body movements.

[0062] The M400 privacy conversation module protects and isolates text data from the other party in the virtual space of Metaverse, making the content of the text data visible only to the Metaverse user who is making the voice call.

[0063] The user data module M500 is used to search for the user data of the other party in the metaverse database based on the other party's communication information in order to determine the identity of the other party.

[0064] The M600 virtual avatar module is used to transmit the virtual avatar of the other party in a call to the Metaverse application, so that the Metaverse application can display the virtual avatar as the medium for voice calls.

[0065] Virtual avatars fall into at least three categories: First, official avatars applied for by formal organizations, which are marked with special tags to enhance credibility and recognizability. Second, freely editable avatars, which are external avatars created by users after registering within the Metaverse application. Third, preset stranger avatars, used by non-Metaverse users to indicate that the call was initiated by a stranger.

[0066] The M700 Avatar Interaction Module handles the interactive behavior of the virtual avatar of the other party in the call. It mainly obtains the user's movement and perspective rotation data in the metaverse and receives and processes events of virtual space activities (such as making the virtual avatar sit down or stand up).

[0067] In this embodiment, S700 is a step within the process of changing the state of the virtual avatar of the other party in the call, and M700 is the module corresponding to the function. Specifically, it changes the appearance of the avatar or restricts its posture, such as sitting in a specific position, standing continuously, moving, or performing actions. For example, in watching a movie, the virtual avatar sits next to the user; or when walking on a street in the virtual world, the avatar walks alongside the user. It can also be implemented in various types of scenes in the virtual world, such as performance halls, formal venues, streets, open tourist attractions, sports fields; or various events held specifically, such as concerts, festivals, competitions, etc. The purpose is to facilitate communication between the two parties in the virtual world without interfering with their current activities, and therefore it must be integrated into the virtual scene.

[0068] The M710 Follow Module allows the virtual avatar of the other party in a call to follow the Metaverse user. In detail, it combines the Metaverse user's movement and perspective rotation data with the scene conditions in the virtual space of both parties and applies them to the virtual avatar of the other party in the call, so that the virtual avatar can move around and continuously face the Metaverse user in order to maintain the conversation between the two parties.

[0069] The M720 response module enables the virtual avatar of the other party in a call to respond based on the actions of the metaverse user (such as waving).

[0070] In addition, activity control or event triggering in the virtual space can also change or override the existing state of the virtual avatar (e.g., from walking to sitting).

[0071] The M800 privacy interaction module is used to protect the privacy of the virtual avatar interaction between the Metaverse user and the other party in the call, so that the immersive interactive experience of the virtual avatars of both parties in the virtual space is only visible to the Metaverse user in the current call.

[0072] The technical details of each step in the method flow shown in Figures 1 to 3 can be incorporated into the relevant modules of the system shown in Figure 4, and vice versa.

[0073] Furthermore, the present invention discloses a computer-readable medium applied in a computing device or computer having a processor (e.g., CPU, GPU, etc.) and / or memory, storing instructions, and capable of being executed by the computing device or computer through the processor and / or memory to perform the aforementioned methods and steps or processes when executing the computer-readable medium. In one embodiment, the computer-readable medium is a non-transitory computer-readable storage medium.

[0074] In summary, the method, system, and computer-readable medium for generating virtual avatars that connect to the metaverse for interaction, as described in this invention, have at least the following advantages and effects compared to existing conventional technologies.

[0075] Firstly, this invention allows users in the metaverse to handle calls from or to the real world at any time and to talk to virtual avatars of people in the real world without switching applications, thus maintaining an immersive experience in the metaverse.

[0076] Secondly, this invention allows people in the real world to communicate and converse with users in the metaverse without logging into the metaverse, simply by using existing voice communication technology. This eliminates the need to switch applications or log into the metaverse, thus improving convenience.

[0077] Thirdly, this invention can convert the intent of a real-world call to be transmitted to the metaverse into a virtual avatar and text message, which can then be integrated into the virtual scene of the metaverse user without affecting the activities of the metaverse.

[0078] Fourth, this invention allows people from the real world to appear in the metaverse through officially certified avatars, customizable avatars, or preset stranger avatars, and to follow the metaverse user as they move within the metaverse to continue communicating, thereby enhancing the connection and social experience between the metaverse and the real world.

[0079] Fifth, the present invention can make the virtual avatar of a person from the real world change its facial expressions according to the content of its voice call, and respond to the virtual scene activities by making corresponding actions. It can also convert the voice content into text display, so as to make the virtual avatar more vivid and improve the interactivity of the call and the degree of integration into the virtual scene.

[0080] The above embodiments are illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify the above embodiments without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be as set forth in the following patent claims.

[0081] S100~S710: Steps M100: Voice transceiver module M200: Voice Analysis Module M300: Expression and Animation Module M400: Privacy Dialogue Module M500: User Data Module M600: Virtual Avatar Module M700: Transform into an Interactive Module M710: Follower Module M720: Response Module M800: Privacy Interaction Module

Claims

1. A method for generating virtual avatars that connect to metaverse interactions, comprising: The electronic devices of users of the metaverse display the virtual space of the metaverse; After the electronic device displays the virtual space, it initiates a voice call between the electronic device and the other party outside the metaverse; based on the communication information of the other party, it determines the virtual avatar of the other party in the metaverse; it causes the electronic device to display the virtual avatar in the virtual space; it analyzes the volume value in the voice data of the other party during the voice call; it converts the voice data into text data; it extracts keywords expressing emotions from the text data; based on the volume value and the keywords, it causes the virtual avatar to make corresponding facial expressions and actions; and based on the interaction between the user and the virtual avatar, and based on the scene or activity in the virtual space, it causes the virtual avatar to make corresponding actions or change its state.

2. The method for generating virtual avatar connections to metaverse interactions as described in claim 1 further includes: Based on the communication information, the database of the other party in the call is searched for user information, and then the corresponding model is read to serve as the virtual avatar. And to protect and isolate the text data of the other party in the call within the virtual space of the metaverse, so that the content of the text data is only visible to the user.

3. The method for generating virtual avatar connections to metaverse interactions as described in claim 1, wherein, The interaction refers to the user's movement in the virtual space, and the method of generating virtual avatars to connect with the metaverse interaction further includes: causing the virtual avatar to follow the user's movement.

4. The method for generating virtual avatar connections and metaverse interactions as described in claim 3 further includes: Calculate the user's position in the virtual space and the perspective changes as the user moves; And based on the location and the viewpoint, update the position and facing direction of the virtual avatar so that the virtual avatar can make a movement action to follow the user and face the user.

5. The method for generating virtual avatar connections to metaverse interactions as described in claim 1, wherein, The interaction refers to the user giving a command to the virtual avatar, and the method of generating the virtual avatar connection to the metaverse interaction further includes: causing the virtual avatar to perform a response action to the command to the user.

6. The method for generating virtual avatar connections to metaverse interactions as described in claim 1 further includes: When the virtual avatar needs to perform multiple actions or change to multiple states, the animation rendering order of the virtual avatar is adjusted according to the weight value of the multiple actions or states.

7. A computer-readable medium, applied in a computing device or computer, storing instructions for performing a method of generating virtual avatar-linked metaverse interactions as described in any one of claims 1 to 6.

8. A system for generating virtual avatars that connect to metaverse interactions, comprising: A voice transceiver module, installed on the user's electronic device in the metaverse, is used to conduct voice calls between the user and a third party outside the metaverse after the electronic device displays the virtual space of the metaverse; a user data module is used to determine the identity of the third party based on their communication information; a virtual avatar module is used to determine the virtual avatar of the third party in the metaverse based on their identity, and to display the virtual avatar in the virtual space on the electronic device; a voice analysis module is used to analyze the volume values ​​in the voice data of the third party during the voice call, and to convert the voice data into text data to extract keywords expressing emotions from the text data; The facial expression and action module is used to make the virtual avatar perform corresponding facial expressions and actions based on the volume value and the keyword; and the avatar interaction module is used to make the virtual avatar perform corresponding movement actions, response actions, behavior actions or change states based on the interaction between the user and the virtual avatar, and based on the scene or activity in the virtual space of the metaverse.

Images