Virtual communication terminal, virtual communication management server, virtual communication management system, virtual communication program and virtual communication method

The system addresses the lack of relationship consideration in virtual communication by using AI to analyze user emotions and interactions, enhancing avatar-based communication by linking and updating information for deeper user engagement.

JP2024161857APending Publication Date: 2024-11-20株式会社KOIKOI
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023076965
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-08
Publication Date
2024-11-20

AI Technical Summary

Technical Problem

Existing communication systems fail to consider the relationship between virtual characters (avatars) in virtual communication, and they do not effectively link or update information based on the interaction between users, leading to superficial and unengaging interactions.

Method used

A system that includes a virtual communication terminal, server, and method for exchanging, linking, and updating virtual communication target information based on predetermined conditions, using AI to analyze user emotions and interactions to enhance the relationship between avatars.

Benefits of technology

Enhances user engagement by linking and updating avatar information based on conversation content, allowing users to better understand each other's emotions and preferences, facilitating deeper and more meaningful interactions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024161857000001_ABST
    Figure 2024161857000001_ABST
Patent Text Reader

Abstract

To analyze video images and voice of users and chain a communication analysis result with a partner to an avatar, when carrying out avatar communication between two or more users.SOLUTION: A communication system is composed of an avatar B displayed on a portable information terminal used by a user A, an avatar A displayed on a portable information terminal used by a user B, and a user management server for managing whether or not which user carries out communication utilizing the avatar. In the system, the users A and B carry out inter-avatar communication in synchronization with motions of the avatars based on the voice and the video image of the partner, while confirming screens where the avatars are displayed by the mutual portable information terminals. The user management server receives feature points of bodies from the portable information terminals or an analysis server, estimates emotions of the users A and B from the intonations of the voices, and updates the avatar A and / or B when the emotions satisfy a predetermined condition.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a virtual communication terminal, a virtual communication management server, a virtual communication management system, a virtual communication program, and a virtual communication method that reflect user-related information input from a user's terminal or the like in a communication service. [Background technology]

[0002] The technology described in Patent Document 1 describes that in an application that displays a virtual character (such as an avatar), the amount of change in each of a plurality of specific parts related to an actor is acquired, and when a second score based on the sum of each of the first scores based on the amount of change exceeds a threshold, the character is made to express an impressive facial expression as the emotion expressed by the actor. With this system, it is possible to make the virtual character express the facial expression that the actor wants to express by a simple method.

[0003] In addition, the invention described in Patent Document 2 discloses a system that automatically determines gender and other information from a face photo captured as a profile image through an image sensor operated on a user terminal using machine learning data, and displays an image received according to the output gender on a monitoring system. This system can minimize or eliminate distrust that users of anonymous video chat services may have regarding the gender of the other party.

[0004] Furthermore, among the services described in Non-Patent Document 1, a service known as "LINE (registered trademark) Avatar" uses technology to control the facial expressions of a virtual character displayed in an application based on the facial expressions of the performer. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2023-15074 A [Patent Document 2] Patent Publication No. 2021-72119 [Non-patent literature]

[0006] [Non-Patent Document 1] “How to create and play with LINE avatars”, June 15, 2022, LINE Corporation, [Retrieved February 1, 2023], Internet URL (https: / / guide.line.me / ja / account-and-settings / account-and-profile / avatar.html) Summary of the Invention [Problem to be solved by the invention]

[0007] In the above-mentioned conventional technology, it is possible to detect changes in extracted feature parts using data such as face captured through an image sensor operated on the user's terminal, and reflect the changes in the character image (avatar, etc.) used by the user and display them on the monitoring system, but the relationship between the avatars is not taken into consideration at all, and the speaker and avatar are merely paired to reflect the facial expression of the user, and the avatar returns to its normal state when communication ends. Also, in Patent Document 2, the received image is input into an artificial neural network model, and the gender associated with the face included in the received image is verified, and a user acting under a false gender can be detected, but this is not reflected in the character image (avatar, etc.) used by the user.

[0008] Therefore, in order to create a tomorrow we can't wait for, the present invention aims to provide a virtual communication terminal, a virtual communication management server, a virtual communication management system, a virtual communication program, and a virtual communication method that can link, i.e., CHAIN, information about a speaker to a virtual communication target of a listener, taking into account the relationship between virtual characters (avatars, etc.). [Means for solving the problem]

[0009] The virtual communication terminal, virtual communication system, virtual communication method and virtual communication program of the present invention according to claims 1, 7, 8 and 10 are characterized in that they have an exchange means for exchanging virtual communication target information based on predetermined conditions for the virtual communication, a linking means for linking the virtual communication target information of the sender and the virtual communication target information of the receiver based on the content of the virtual communication, and a means for updating the virtual communication target information before the exchange based on one or more linked pieces of virtual communication target information.

[0010] In addition, the connecting means of the virtual communication terminal and virtual communication method according to claims 2 and 11 of the present invention is characterized by a means for exchanging virtual communication target information with one or more receivers corresponding to a selection of pre-stored virtual communication target information, and an updating means for updating the virtual communication content according to the analysis results.

[0011] In addition, the linking means of the virtual communication terminal and virtual communication method according to claims 3 and 12 of the present invention is characterized in that it further comprises an analysis means for estimating the speaker's emotions from changes in the features of the speaker's movements or voice in response to the listener's movements or voice, and links the analysis results of the communication content with each of the information.

[0012] In addition, the updating means of the virtual communication terminal and virtual communication method according to claims 4 and 13 of the present invention is characterized in that it updates the virtual communication target information stored by at least two or more speakers and listeners to absorb the difference between the virtual communication target information stored by at least two or more speakers and listeners.

[0013] In addition, the updating means of the virtual communication terminal and the virtual communication method according to claims 5 and 14 of the present invention is characterized in that the feature points of the objects included in the virtual communication target information are updated so as to be common.

[0014] In addition, the virtual communication management server of claim 6 of the present invention is characterized in having a means for pre-storing user information and permission conditions related to virtual communication as virtual communication target information, a communication means for encouraging the sender and / or receiver's terminal to start virtual communication in accordance with the communication conditions included in the virtual communication request from the sender and the receiver's communication permission conditions, and a data processing unit for continuing the virtual communication by linking the listener's virtual communication content to the speaker's virtual communication target information and the speaker's virtual communication content to the listener's virtual communication target information.

[0015] The virtual communication method according to claim 9 of the present invention is characterized in that the predetermined condition is that the users have mutually obtained authentication regarding single certificates. Effect of the Invention

[0016] According to the invention described in claims 1, 7, 8 and 10, by exchanging virtual communication target information based on specified conditions, linking the content of the virtual communication of the sender with the virtual communication target information of the receiver, and updating the virtual communication target information before the exchange based on one or more linked virtual communication target information, the content of the virtual communication between the virtual communicators can be mutually linked to the virtual communication targets, and the virtual communication target information can be changed, i.e., updated, in accordance with the content of the mutual conversation, making it possible to easily confirm the relationship between users.

[0017] Furthermore, according to the inventions described in claims 2 and 11, virtual communication target information is exchanged with one or more receivers as virtual communication targets by selecting pre-stored virtual communication target information, so that receivers can be efficiently specified, particularly when starting virtual communication involving three or more parties.

[0018] According to the inventions described in claims 3 and 12, the speaker's emotions are estimated from changes in the speaker's movements or voice features in response to the listener's movements or voice, and the analysis results of the communication content are linked to each virtual communication target information, so that virtual communication can be carried out to develop the conversation further. Furthermore, since the communication content with the other party is linked to the other party's virtual communication target along with the reactions of the other party, including joy, anger, sadness, and happiness, it is possible to check the conversation including the other party's emotions even after the virtual communication ends.

[0019] In addition, according to the invention described in claims 4 and 13, the virtual communication terminal information stored by at least two or more senders and receivers is updated to absorb the differences, so that the changes in the virtual communication targets resulting from emotions such as liking or disliking the other person become similar, making it possible to more concretely grasp the communication relationship between users.

[0020] Furthermore, according to the invention described in claims 5 and 14, the features of the objects contained in the virtual communication target information are updated so that they have common features. This makes it possible to maintain the overall features of each piece of virtual communication target information while updating only the parts that need to be updated efficiently and with low communication traffic, thereby reducing the load caused by the update process.

[0021] Furthermore, according to the invention described in claim 6 of the present invention, by storing user information and permission conditions regarding virtual communication in advance, and initiating virtual communication at the sender's and / or receiver's terminals according to the communication conditions included in the virtual communication request from the sender and the receiver's communication permission conditions, virtual communication can be carried out with peace of mind, even with parties with different conditions, while checking the other party's virtual communication target information each time to see whether or not to carry out virtual communication.

[0022] According to the invention described in claim 9, by objectively proving that virtual communication users have mutually obtained certification of their single status, users who are seeking serious encounters through various matching services can feel safe in matching for marriage and other purposes. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a network configuration diagram of an avatar-to-avatar communication system according to one embodiment of the present disclosure. [Figure 2a] FIG. 2 is a hardware configuration diagram of a general mobile information terminal 200 that does not perform video analysis. [Figure 2b] FIG. 2 is a hardware configuration diagram of a mobile information terminal 210 of the present invention that performs video analysis. [Diagram 3] FIG. 3 is a diagram showing the hardware configuration of avatar distribution server 300. [Figure 4] FIG. 2 is a hardware configuration diagram of a user management server 400. [Diagram 5] FIG. 5 is a hardware configuration diagram of an authentication server 500. [Figure 6a] FIG. 6 is a diagram showing the hardware configuration of an analysis server 600 that analyzes video and audio. [Figure 6b] FIG. 6 is a diagram showing the hardware configuration of an analysis server 610 that performs analysis of only audio. [Figure 7a] 11 is a flow diagram of an information server device according to an embodiment of the present disclosure analyzing and processing communication between avatars. [Figure 7b] FIG. 11 is a flow diagram for analyzing communication between avatars on a mobile communication terminal according to another aspect of the present disclosure.

[0024] [Figure 8] 2A is a diagram showing a user management table in a user information storage unit 402, and FIG. 2B is a diagram showing an avatar storage table in an avatar information storage unit 401 of the present disclosure. [Figure 9] Specific examples of the present disclosure include (a) a table diagram for storing the comprehensive consent conditions set by a user of an avatar ID in the comprehensive consent conditions memory unit 502 of the authentication server 500, (b) a voice / avatar storage table diagram used in the analysis data memory unit 602 and / or the voice / avatar storage unit 202, (c) an emotion analysis data storage table diagram used in the analysis data memory unit 602 and / or the data memory unit 202, and (d) a conversation analysis data storage table diagram used in the analysis data memory unit 602 and / or the data memory unit 202. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] The essential parts of the present invention will be specifically described with reference to the attached drawings, taking the first embodiment as an example. In the following description, where possible, the same parts will be given the same reference numerals, and duplicated descriptions will be omitted. Note that the virtual communication target is not necessarily limited to an avatar, but may include objects that can be expressed in a virtual space, real estate in a virtual space such as a conference room or a building, and movable property such as a car, a bicycle, or an airplane. In the embodiment, the description will be unified with an avatar, but it goes without saying that the present invention is not limited to this.

[0026] In addition, the virtual communication target information may include profile information of the user who uses the virtual communication target, permission conditions for virtual communication, and information linked as virtual communication content with other virtual communication target information, and in the embodiment, is described as information that is tabulated by avatar ID and user ID and can be linked, but it goes without saying that this is not limited to this.

[0027] Additionally, the term CHAIN ​​is not necessarily limited to a chain, link, connection, or construction of a connection between communicating avatars, but is defined as an expression that includes any update, reflection, or deformation of an avatar that facilitates communication between users who use the avatar. EXAMPLES

[0028] Fig. 1 is a network configuration diagram of an avatar communication system 100 according to one embodiment of the present disclosure. As shown in Fig. 1, this avatar management system includes a mobile information terminal 200 that communicates with an information distribution server 101 via a communication network 102, an avatar distribution server 300 that distributes an application for avatar communication between user terminals 200a and 200b, an authentication server 500 that certifies predetermined conditions designated by users, a user management server 400 that manages attributes of avatar users, and an analysis server 500 that performs AI analysis of avatar communication.

[0029] The avatar distribution server 300, user management server 400, analysis server, and authentication server 500 (if the authentication server is an optional service and more added-value usage permission is processed through API integration with an external server, it is not included in the information distribution server, but may be constructed as a whole by the distributor) that make up the information distribution server 101 will be described as an example of constructing a server in an on-premise environment as the most basic form, but this is not limited to this.

[0030] It goes without saying that it is also fine to theoretically separate them within a virtual environment using DevOPS, and use a construction method that takes advantage of the advantages of a public cloud with the same or greater potential as an on-premise environment.The explanation focuses on constructing a system environment for each server role, taking into consideration the difference in operational load between processes that require real-time performance and processes that irregularly distribute applications, and it goes without saying that it can be constructed in any way that balances with operations.

[0031] Also, although a smartphone is exemplified as the mobile information terminal 200, a tablet terminal, a notebook computer, a FAT terminal such as a desktop computer, or a virtual terminal configured on a server may be used as the mobile information terminal. An avatar used by user B is displayed on the screen of mobile information terminal a, and an avatar used by user A is displayed on the screen of mobile information terminal 200b, and the other side of these avatars is displayed on the screens of mobile information terminals 200a and 200b through an application distributed from avatar distribution server 300.

[0032] The mobile information terminal a captures an image of user A with the in-camera, and the mobile information terminal 200b captures an image of user B with the in-camera, and they are mobile information terminals that are also capable of general video call communication. Note that, although the mobile information terminal 200 is shown as being equipped with an imaging means such as a camera, it is not necessarily limited to being a portable terminal, and each may function as a separate terminal at home or in a public place, such as a combination of a camera or security camera installed at the entrance of a home with a display panel or digital signage terminal at the entrance.

[0033] In the present invention, as a predetermined condition for communication between avatars described later, a preselected avatar is displayed to the other party without displaying the user's image to the other party, enabling anonymous video calls. Note that, although one embodiment will be described with reference to the individual drawings, it is not necessary to construct each server separately. For example, it goes without saying that authentication server 101 may be constructed integrally with the authentication and authorization process in user management server 400, and avatar distribution server 300 may also be constructed integrally with user management server 400 and analysis server 500. Hereinafter, it will be specifically described what configurations avatar distribution server 300, user management server 400, authentication server 500, and analysis server 600 have and how they communicate with and process mobile information terminal 200.

[0034] Next, FIG. 2a is a configuration diagram of a mobile information terminal 200 that does not perform video analysis and that smoothly performs communication between avatars according to one embodiment of the present disclosure. The screen display unit 201 has a function of displaying data from the data processing unit 305, such as data of a site for downloading an application described later, and an avatar installed after downloading. The data storage unit 202 stores downloaded applications and avatars, and stores information on the avatars that the user uses in advance. The communication unit 203 downloads applications from the avatar distribution server 300, and communicates with the outside, such as communicating with the user management server 400 and the authentication server 500 when using communication between avatars.

[0035] The input data synchronization unit 204 has a function of applying a time stamp to a series of input data in order to synchronize the input data from the audio input unit 207 and the data input from the imaging unit 209 with the video analysis processing data described later by the analysis server 600. Note that the same time stamp is used at least between the applications of the information communication terminal 200 that is the other party of the communication (because when avatar communication is performed without performing any analysis on the server side, communication can be performed with the lowest delay and synchronization with the server's time stamp is not required). However, when the analysis of the video or audio data is performed on the server side and avatar movement information and audio data described later are synchronized, it goes without saying that it is better to use a time common to various servers and increase the synchronization accuracy as much as possible in a manner close to real communication.

[0036] The data processing unit 205 plays the role of a control tower within the information communication terminal, and has a function of processing data communication between modules, which are the components from the screen display unit 201 to the imaging unit 209, according to the purpose of each piece of data. The key input unit 206 has a function of passing key information entered by the user for selection items displayed on the screen, etc., to the data processing unit 205.

[0037] The audio input unit 207 has a microphone function consisting of a sound collection module to input the user's voice, and the audio input unit 208 has a function of outputting the voice of the user who is the other party in the call and the voice corresponding to a predetermined operation of the application. The imaging unit 209 has a function of imaging the facial expression and hand and body movements of the user captured by the camera function consisting of a CCD module, and transferring the data to the data processing unit 205.

[0038] Next, Fig. 2b is a configuration diagram of a mobile information terminal 200 that performs video analysis to facilitate communication between avatars in another embodiment of the present disclosure. Compared to Fig. 2a, an audio / video analysis unit 219 that analyzes the user's voice and / or video captured by the audio input unit 207 and / or the imaging unit 209 is added as a new component of the terminal. In the mobile information terminal 210 of Fig. 2b, the audio / video analysis unit 219 has a function of detecting the user's emotions by analyzing the content of the conversation spoken by the user, the strength and tone of the voice, and / or analyzing the difference between image frames included in the video of the user's facial expression, hand and body movements, and detecting the feature points of the face and hands as a motion sensor.

[0039] In addition to faithfully reflecting the movements of the face and hands on the avatar, the avatar has a function of detecting changes in the user's emotions, etc., using AI analysis technology based on changes in facial feature points. This function is not included in the mobile information terminal 200 in FIG. 2a, but in the case of FIG. 2a, a video analysis unit is provided in the analysis server 600, and similarly, the movements of the user's hands and face are motion captured, and changes in emotions, etc. are detected on the analysis server 600 side. Therefore, the cloud AI can infer emotions, etc., and learn the depth of the communication relationship between users by repeating the analysis in proportion to the conversation. The CHAIN ​​structure table that accumulates and accumulates the analysis results will be explained in detail later from FIG. 8 onwards.

[0040] Furthermore, the effect of analyzing all audio and video on the mobile information terminal 200 side as an edge AI and reflecting avatar movement information and the user's emotions in the avatar is, needless to say, that the amount of communication between networks can be reduced compared to video calls. However, by exchanging communication between users based on data such as anonymized identifiers on the edge side, even the distribution provider, who cannot violate the secrecy of communications, can accumulate and store the analysis results and provide services to users without conducting conversation analysis on the communication path.

[0041] This point is also the case with the optional service for the work management system of Patent No. 6964160, which was obtained by the lead inventor. Rather than collecting log data within the communication path, providing added value to the usage spread by utilizing information collected from the user's terminal with the individual's consent has been taboo and avoided in the communications industry until now, but we thought that conversations could be analyzed based on one's own will and the other party's consent, and used for learning, becoming a familiar presence that can help build better human relationships.

[0042] Therefore, the lead inventor, while taking into consideration the secrecy of communications, which is an absolute constitution for telecommunications carriers, can construct a system that, by providing next-generation communication between avatars, which can learn from conversations with specific people by simply memorizing the content of one's conversation and the emotions the other person felt, or vice versa, the one's own emotions received from the content of the other person's conversation, and reflecting these in the avatar, makes it possible to envision a world in which people all over the world can share and empathize with the content of conversations and the emotions felt by others, overcoming language barriers such as Japanese, English, or French.

[0043] Needless to say, language issues can be resolved by using a translation API or other high-precision interpretation / translation technology to convert speech into the other person's language. In this case, it goes without saying that language processing can be performed in parallel when analyzing emotions from speech, and the converted speech and emotions can be superimposed on the output in the other person's language.

[0044] 3 is a configuration diagram of a distribution server 300 that distributes applications for smooth communication between avatars in one embodiment of the present disclosure. A communication unit 301 has a function of communicating with the mobile information terminal 200, the user management server 400, and the analysis server 500. An application storage unit 302 stores application files for performing communication between avatars according to the present application, which correspond to the OS type, model, and language used of the mobile information terminal 200.

[0045] The application update management unit 303 has a function of processing, by the data processing unit 305 via the communication unit 301, application distribution requests and file synchronization periodically or when adding functions or releasing new services to each distribution server (not shown) which differs depending on the language used and the model of the Android (registered trademark) OS, iOS, Windows (registered trademark) OS, etc. Note that the avatar distribution server 300 itself may be capable of directly receiving downloads and updates from users, and may download and update version upgrades and correction patches for each application as well as requests to download the application, as required when using the application.

[0046] The avatar storage unit 304 stores parts that constitute an avatar used by the user in the application, and stores data so that the avatar creation unit 306 can select shape parts and colors of the face (including hair, outline, eyes, eye corners, nose, cheeks, and mouth parts) and body (including body parts such as hands, elbows, shoulders, and chest, as well as parts such as clothes, pants, hats, hair accessories, and pin badges) stored based on the user's operation to create the user's own avatar. Note that some avatars created by the provider as completed versions may also be stored in advance so that the user can immediately use the avatar without customizing it. In addition, the avatar may be a 3D type that can reflect motion-captured movements on the avatar during communication, or a 2D still image.

[0047] Data processing unit 305 issues commands to each component of the avatar distribution server to process and store various types of data, and also performs central processing of communication between avatars by sending and receiving data to and from mobile information terminal 200, user management server 400, authentication server 500, and analysis server 600. Specific data flows will be described in detail in Figures 7a and 7b, including the data flows.

[0048] The avatar update processing unit 307 has a function of analyzing the emotions and actions caused by the communication from the video and audio between users who communicate between avatars, and updating the avatars used by each of them to reflect the individual emotions and actions. In addition, the avatar update processing unit 307 comprehensively analyzes the conversation or meeting from start to finish, and determines whether to update the avatars between the users based on the relationship between the users. When the avatar update processing unit determines that the communication between the users is good, the avatar synthesis unit 308 performs synthesis processing, for example, by synthesizing the characteristic parts of the parts of the avatars used by each user with each other, and the more good the conversation continues, the more similar the shapes of the parts become to make the two parts look similar. Alternatively, the same update processing may be performed so that the corners of the eyes of both avatars are smiling.

[0049] The essential part of the present invention is that, which was not disclosed in the prior art, the results of communication between the two parties are analyzed, and the images and videos of the avatars used by each party are updated relatively or synchronously based on the analysis results.Specifically, as will be explained later with reference to the analysis server, for example, if a man and woman who are looking for a partner of the opposite sex display their avatars anonymously and talk to each other, and it is analyzed that there are many smiles and laughter detected, and it is then analyzed that each party feels that they would like to talk again, it is desirable that the avatar name and avatar of the other party displayed in the call history and contact list in the avatar-to-avatar communication app reflect these analysis results.

[0050] With this configuration, the avatars of the users are chained, and after the conversation ends, the user can easily check the quality of the conversation from the updated avatars stored in the memory. Even if the conversation does not end well, the analysis results from past conversations can be displayed, such as what remarks or actions of the user caused the other person to feel negative emotions, making it possible for the user to reflect on their own conversation before having a future conversation.

[0051] Previously, this was limited to communication between avatars, but also to phone book apps provided by SNS communication app providers and mobile phone companies, which stored contact information (including phone numbers, names, birthdates, and other personal information) and displayed the avatar and name selected by the user each time a conversation took place.

[0052] From the world of ad-hoc communication, which has been unchanging for many years due to the high hurdle of confidentiality of communication, conversations, like human relationships, can improve or deteriorate depending on the results of previous conversations, and by defining what is good or bad for the user as a predetermined condition, conversations can be evaluated, emotions, and other results can be accumulated and reflected in the avatar. As a result, it is possible to develop into a world of cumulative communication in which the avatar's growth can be improved by continuing communication, providing a very innovative and visible communication between avatars, which is more intimate and allows users to learn compared to simple one-sided communication such as email or SNS.

[0053] The billing management unit 309 has a function that allows paid functions to be used by purchasing or subscribing to paid content or services within the application, such as when the user wants to make the avatar reflect the user's own movements, perform synthesis processing, or save the avatar of the other user in the mobile information terminal. In order to consider which services and contents should be charged and how they should be differentiated from services and contents that can be used for free, it is necessary to consider not only the needs of the expected users and the costs required for building a system, but also the presence or absence of charges for general avatar content, and to consider various forms of provision, such as using existing mechanisms such as in-app billing or building one's own system, that can differentiate the content by adding value to the company's services, and a business model that further colors the present invention.

[0054] 4 is a configuration diagram of a user management server 400 that manages users of avatar-to-avatar communication in one embodiment of the present disclosure. The avatar information storage unit 401 stores information that identifies the type of avatar and its combination by an identification code, with each of the avatars and avatar parts used by the user in the application treated as objects. Avatars created by a user who has downloaded the avatar-to-avatar communication application from the avatar distribution server 300 by combining avatar parts in the initial settings are all reproduced and stored by a combination of the object identification codes of each part.

[0055] A user uses at least one avatar when using this service, but may use different avatars for work and private use, or may create separate avatars to belong to community objects related to the user's hobbies and interests. In the user management server 400, the user information storage unit 402 has a user ID for uniquely identifying a user, and a table that is linked to the user ID and can be identified by at least one avatar. The table configuration will be described in detail with reference to FIG. 8.

[0056] The authentication processing unit 403 has a function of authenticating whether or not the user using the mobile information terminal 200 matches the user when the avatar was initially set, for example, by comparing feature points of the face image at the time of initial setting with feature points of the face image to be authenticated received when a communication request is received from the mobile information terminal 200 to a specific avatar, and authenticating whether or not the user is the real user. Also, when a communication request is made from a user to a specific avatar, the authentication processing unit 403 has a function of making an authentication request to the user information storage unit 402 or the authentication server 500 to determine whether or not the user is capable of communication, and giving authorization.

[0057] The determination of whether a user is available for communication is based on information such as the user's attributes, purpose of communication, hobbies and interests, whether communication is public, and the scope of publicity, which are included in the user table configuration described below, that is registered in advance by the user or the organizer of a meeting or event as avatar profile information. This information has a function to determine whether the avatar user who has made a communication request meets the conditions. The processing flow will be described in detail in Figures 7a and 7b.

[0058] The data synchronization unit 404 has a function of converting the user's movements from the video captured by the imaging unit 209 into avatar movement information as motion in the analysis server 600 described later, and synchronizing it with the avatar displayed on the mobile information terminal together with the user's voice and facial expression. Specifically, the data processing unit 205 of the mobile information terminal monitors the time when the user's voice is input, and attaches a timestamp to the user's voice data input by the voice input unit 207, and also attaches the same timestamp as the time when the user started speaking to video data relating to the time from the voice start time to the voice end from the imaging data continuously input from the imaging unit 209.

[0059] The data processing unit 405 plays the role of a control tower in the user management server, and has a function of processing data communication between modules, which are the components from the avatar information storage unit 401 to the communication unit 409, according to the purpose of each data. For example, when communication is performed in a motion capture mode that does not perform emotion analysis, the voice data and the movement information for the avatar are transmitted to each mobile information terminal 200, and the input data synchronization unit 204 of the mobile information terminal controls the data processing unit 205 to reflect the synchronization result of the input data synchronization unit 204 on the avatar based on the movement information data from the analysis data synthesis unit described later, and to display the avatar on the screen display unit 201 as if the person is having a conversation with body language.

[0060] Although an embodiment has been described in which the analysis process is performed on the analysis server side, it is also possible to provide a similar analysis function to an application installed on the mobile information terminal 200, and to transmit information on the movement of the avatar after it has been returned to the other user via the communication network 101 or various servers, without having to continue communicating large volumes of video data. Needless to say, this would enable communication between avatars that reflects the motions of each user with less data communication volume and shorter delays.

[0061] In addition, although the details will be described later in the analysis server function, analysis of video data is not necessarily required, and especially when it is desired to shorten the time between synchronous conversations, users may select the data to be analyzed for avatar-to-avatar communication in advance, and a conversation may be conducted only by voice in general avatar-to-avatar communication (the avatars send and receive voice while displaying still or moving images). In addition, the analysis target may be only the motion capture from the video data reflected on the avatar, or the motion capture and analysis of facial expressions, emotions, etc. from the video data may be turned on and reflected on the avatar. Furthermore, emotions from the voice data may be analyzed from the volume of laughter and the tone of the laugh, and reflected on the avatar together with the analysis results of emotions from the video data.

[0062] Needless to say, the user can register the analysis target data, analysis setting information, and what kind of communication is allowed and selected for which avatar type in advance on the usage management server via the application setting screen. For example, in matchmaking in the metaverse space, since conversation that emphasizes the person's humanity and personality rather than appearance is emphasized, a deeper analysis can be performed from video and audio data, and profile information entered by the user can also be utilized and analyzed to perform compatibility diagnosis from the development of conversation between avatars. Since various applications are possible for this AI analysis, the core embodiment is described as one aspect in this application.

[0063] In real conversations, only people with insight can intuitionally tell the difference between a nice smile and a suspicious smile that is trying to hide something, and the difference between a smiling face and a smirking face based on subtle differences in expression and smile, not in the shape of the face. People are divided into those who can sense this without any logic and those who have difficulty in sensing it, depending on the amount of experience, outlook on life, how they have lived up until now, etc. In particular, for people who are not confident in their communication skills, those who feel complex, and those who are hard of hearing and ask to repeat themselves many times, this invention can create a world of communication where such people can confidently express their humanity and honest feelings.

[0064] Therefore, because communication is via avatars, some users, especially those who place importance on the appearance of the other person, may find it to be an unsatisfactory means of communication. However, human relationships do not last on appearances or temporary cover-ups, and just like inventions, you can quickly find out what a person's true nature lies within them through an avatar, and it goes without saying that even if the appearance of a person does not meet your ideal, you will be able to have a long-lasting relationship with someone who is compatible with you.

[0065] For example, if you are transferred to a company and can no longer see someone who has changed company or work environment, or if you have a good conversation chemistry with people who are a generation older than you and you don't feel the generation gap, you can store memorable conversations and fun conversations you've had as communication results with your avatar. If the present invention provides a mechanism that allows you to check that history later even after time has passed, it can be a trigger for wanting to talk or meet again, or to wait for a drink and talk. It goes without saying that the desire to increase such communication led to the creation of this invention.

[0066] Everyone has a fun and reassuring way of talking to people who understand them and get along well with them, and there are also people who feel reassured by the facial expressions of people when they are talking. Therefore, it goes without saying that we would like to continue to provide better communication that values ​​humanity. We have filed this application as an invention for that purpose.

[0067] Returning to the detailed explanation of the configuration of the user management server, the analysis data synthesis unit 406 counts the emotion analysis data from the audio analysis unit 608 or 618 and / or the emotion analysis data from the video analysis unit 609 in the analysis server 600 of users A and / or B for each user and emotion type, and compiles the communication between avatars from the start to the end to determine whether the communication was good, which remarks caused the most laughter, etc., and then, after aggregating the AI ​​analysis results, for example, if the user has good conversational performance, the facial expressions of both avatars are updated so that they are smiling.

[0068] Similarly, the voice analysis unit 407 counts the user's emotion analysis data based on the voice data from the data received from the analysis server 600 via the communication unit 409, for each user and emotion type, and compiles the data from the start to the end of the communication between the avatars to determine whether the communication was good, which remarks caused the most laughter, etc., and then compiles the AI ​​analysis results. For example, if the user has a good conversation record, the avatar's facial expression is updated so that both the avatars are smiling.

[0069] It determines whether the movement information data matches a facial expression or emotion that satisfies specified conditions, and if the specified conditions are met, it has the function of synchronizing the analysis result data of facial expression or emotion obtained by AI analysis of video or audio with the avatar used by the user based on the timestamp information added by the data synchronization unit 404 when a subtle emotional expression that satisfies specified conditions is detected.

[0070] Specifically, it also has a function of transmitting facial expression and / or emotion analysis data from the video analysis unit 408 as CHAIN ​​information for the avatar of the communication partner stored in the data storage unit 202 of the mobile information terminal 200a and / or 200b so that the data can be synthesized with avatar B and / or A stored in the data storage unit 202 of the mobile information terminal 200a and / or 200b of the user A and / or B.

[0071] In this way, not only are the avatars reflected as motion capture each time a conversation takes place, but, for example, the emotions between users throughout the entire conversation could be analyzed by AI, and the avatars could grow to resemble more realistic portraits of the users with each conversation. Or, even if users A and B initially use avatars that are completely different, the differences between the avatars could be eliminated with each conversation, and each part of the avatar could be updated to make them more similar to each other, so that the users' avatars would start to resemble each other, like a couple. There are an infinite number of ways in which the results of a conversation can be accumulated as CHAIN ​​information in subsequent apps.

[0072] Therefore, by avoiding the negative effects of slander and libel that can occur through one-sided text-only communication methods such as modern SNS and chat, and by enjoying conversations to have good conversations even when you don't know how the other person will feel, and by visualizing the growth of each avatar and developing conversations that take the other person into consideration, it goes without saying that the personal relationships between users will be reflected in their avatars, and both parties will be able to work together to build good relationships.

[0073] 5 is a configuration diagram of an authentication server 500 that authenticates users of inter-avatar communication in one embodiment of the present disclosure. The authentication server 500 stores in a comprehensive consent condition storage unit 502 comprehensive consent conditions related to inter-avatar communication that are selected and determined by users at the time of initial registration of avatars, such as conversation permission information that user A desires from user B when user A starts inter-avatar communication with user B, for example, whether to permit analysis of facial expressions and emotions in video and / or audio, whether to permit motion capture from video data to be reflected in the avatar, and whether to permit CHAINing of the analysis results to the avatar of user B stored in the app at the end of the conversation.

[0074] The authentication judgment unit 503 issues commands to each component of the authentication server to process and store various data, and also performs central processing to authenticate users of avatar-to-avatar communication by transmitting and receiving data between the portable information terminal 200 and the user management server 400. Specifically, when user A transmits a communication request for avatar-to-avatar communication to user B, the authentication judgment unit 502 first checks whether the user satisfies the requirements of the user management server by referring to a table in the comprehensive consent condition storage unit 502 to see whether user B has permitted the communication method desired by user A.

[0075] For example, suppose that user A allows user B to reflect motions from user B's video and audio on his / her avatar, and further requests avatar-to-avatar communication in which facial expressions and emotions are analyzed and reflected on the avatar. In response to this request, the setting information for avatar-to-avatar communication set by user B is referenced from a table, and if all information matches, it is determined that comprehensive consent has been given, and avatar-to-avatar communication between users A and B begins.

[0076] On the other hand, if at least one of the setting information of the inter-avatar communication set by user B does not match the request (for example, the function of analyzing emotions from video is set to OFF), it is determined that comprehensive consent has not been given, and the consent request unit 504 transmits data to user B of the mobile information terminal 200b via the communication unit 501 that there is an inter-avatar communication request with the emotion analysis function ON. Note that the consent request is not limited to this example, and it goes without saying that when the content of the inter-avatar communication between the parties recorded in the past is shared by an avatar in the CHAIN ​​range other than the conversation parties, a method of obtaining consent from each of the conversation parties each time may be adopted, so that the content of the conversation may be shared with more sufficient consideration for the secrecy of communication. Note that information that is not the direct content of the conversation, such as the type of content of the conversation, the type of emotion such as fun, or the number of combos in the conversation, may be anonymized so that it can be treated as information that does not fall under the secrecy of communication, and then shared within the CHAIN ​​range.

[0077] User B of mobile information terminal 200b, who receives the consent request each time, turns on the function for analyzing emotions from video on the screen and selects on the screen to permit avatar-to-avatar communication with user A. The result is then sent to the consent confirmation unit 505 of authentication server 500 each time, and the authentication judgment unit 503 permits avatar-to-avatar communication from user A, thereby starting avatar-to-avatar communication between users A and B.

[0078] In addition, at least one of the profile information of the user may be set as a permission condition for communication between avatars, and at least one of the conditions such as age, sex, occupation, purpose of conversation, hobby, presence or absence of a certificate of single status, etc. may be set instead of or in combination with the conditions for communication between avatars. Note that a certificate of single status is a certificate issued at a city, town, village, or ward office when a marriage registration has not been submitted, or when a marriage registration has not been submitted again after a divorce registration has been submitted, and it goes without saying that the documents submitted by the user to the city office may be digitized and sent to a server as evidence to register condition data proving that the user is single.

[0079] 6a and 6b are configuration diagrams of an analysis server 600 that analyzes audio and / or video to facilitate communication between avatars in one embodiment of the present disclosure. In FIG. 6a, video is analyzed on the video analysis unit 609 side of the analysis server 600, whereas in FIG. 6b, the audio / video analysis unit 219 of the mobile information terminal 210 analyzes the content of the conversation spoken by the user, the strength and tone of the voice to detect the user's emotions and / or detects the user's movements and emotions by analyzing the video through motion capture, facial expressions and emotions, and reflects the analysis results in the avatar movement reflection unit 604 of the analysis server 600. However, most of the components are common, and will be described with the same numbers.

[0080] The voice / avatar storage unit 601 has a function of storing the avatar selected by the user and voice data generated in inter-avatar communication, which is received from the mobile information terminal 200 via the communication unit 603. The analysis data storage unit 602 has a function of storing the analysis results of the user's facial expressions and emotions based on the user's voice and / or video data analyzed by the voice analysis unit 608 and / or video analysis unit 609.

[0081] The communication unit 603 has the function of communicating with the outside, such as receiving the user's voice data and video data, or movement information data obtained by motion capturing the video data, analyzing the data in the analysis server 600 and transmitting the results to the avatar distribution server 300 to reflect the avatar's movements during communication, and collating the analysis results of the emotion data and movement data for each user generated during the conversation when communication between avatars is terminated, and transmitting the analysis and compilation results to the user management server 400.

[0082] The avatar movement conversion unit 604 has a function of converting the video data analyzed by the video analysis unit 609 into avatar movement information when the motion capture is processed on the analysis server 600 side. The data processing unit 605 issues commands to each component of the analysis server to process and store various data, and also transmits and receives data to and from the mobile information terminal 200, avatar distribution server 300, and user management server 400, and performs central processing of communication between avatars.

[0083] When starting avatar communication, the avatar information input unit 606 receives the latest avatar information from each of the mobile information terminals 200a and 200b, as well as movement information received from the avatar movement conversion unit 604 via the user management server 400, superimposes this information on the avatar information, stores it in the voice / avatar storage unit 601, and transmits the avatar information with the movement reflected to the mobile information terminals 200b and 200a, respectively.

[0084] The audio analysis unit 608 has a function of analyzing the audio data received from the mobile information terminals 200a and 200b in the order of time stamps, and the video analysis unit 609 has a function of analyzing the video data received from the mobile information terminals 200a and 200b in the order of time stamps. In this embodiment, each analysis is performed by a different component, but this is not limited to this, and it goes without saying that audio and video may be analyzed together, and service collaboration with an external server may be performed by API.

[0085] For example, it goes without saying that the "AI suite (registered trademark)" API provided by NTT Resonant Inc. can be used to infer the target time period you want to analyze from the timestamp and input signal, such as time periods when there are a lot of actions in the audio and video that can be assumed to express emotions, or parts where the audio is louder than the overall conversation, or parts where the pitch is higher than in normal conversation, and then highly accurate analysis can be performed focusing on those parts using accurate API integration, etc.

[0086] In short, the essential part of the present invention is that, although various AIs are emerging for the technology of analyzing voice and video, the facial expressions and emotions that are transferred to the voice and video that each person utters or expresses in a conversation are not only captured as points and displayed on the edge of an avatar, but are also captured as lines, and the pause (the time it takes to analyze and reflect on the avatar) is used to analyze emotions that arise in a conversation between users. For example, the emotions felt by user B when seeing and hearing the content of user A's conversation and video can be associated as the embodiment of voice, facial expressions, body language, etc., and the "emotions of the other person toward one's remarks, etc." can be recorded and stored, and then CHAINed to avatar information.

[0087] At first glance, this pause may make conversation difficult, but it can actually be used to help you get to know and feel the other person better. For example, when the other person is talking, you may forget your own thoughts, and shy people may listen to the other person's conversation for too long and miss out on expressing their own opinions and feelings. Also, impatient people may start talking over the other person's conversation, or talk unilaterally about their own needs without listening to the other person's story to the end.

[0088] Even in these types of unbalanced conversations, the avatars will only start to move and speak once the analyzed data has been synchronized, and the other avatar will only start to speak once one side has finished speaking. This will eliminate one-sided online voice overlaps and inability to hear voices due to the network, and will create a back-and-forth style of communication, allowing for equal conversations. It goes without saying that the avatars can also play a role in mediating even for those who are not good at communicating.

[0089] Next, Figures 7a and 7b are process flow diagrams from the start to the end of avatar-to-avatar communication in one embodiment of the present disclosure. Before user A in Figure 1 performs avatar-to-avatar communication with user B, user A and user B each request and download the application of the present invention from avatar distribution server 300 at their own timing (S701, S702). At this time, the application is stored in data storage unit 202 together with the avatar (including parts) of the user who set the avatar earlier and the avatar on the server side that is the virtual communication target such as a conference room or event venue. After that, a screen prompting installation is displayed, and the user installs the application by following the instructions on the screen.

[0090] Once the installation is complete, users A and B register their profile information, such as their name, age, marital status, occupation, hobbies, and avatar name, and set the avatars they will use for avatar-to-avatar communication (S703, S704). The profile and avatar information of the user for which this initial setting has been completed is also stored in user information storage unit 402 of user management server 400, enabling avatar-to-avatar communication with existing avatars.

[0091] As one aspect of avatar setting, communication with an avatar distribution server is performed, and based on the user's operation, shape parts and colors of the face (including hair, outline, eyes, eye corners, nose, cheeks, and mouth parts) and body (including body parts such as hands, elbows, shoulders, and chest, as well as clothes, pants, hat, hair ornaments, pin badges, etc.) are selected from objects stored in avatar storage unit 304 as elements that make up the avatar, and an avatar of the user's own is created via avatar creation unit 306, and the selection results are stored in data storage unit 202.

[0092] Although an example in which the avatar is created on the avatar distribution server side has been described, the present invention is not limited to this. Needless to say, an installed application may have a function similar to avatar creation unit 306, and the user's avatar may be created within the application. Also, several completed avatars created by the provider may be stored in advance, allowing the user to use the avatar immediately without customizing it. Needless to say, even after a completed avatar has been set, each part can be reselected and customized.

[0093] When storing this information, it is possible to store and transmit / receive it as an image of the entire avatar when processing the memory capacity of the mobile information terminal or linking avatar information with various servers, but it goes without saying that it is also possible to store information specifying the type of avatar and its combination by an identification code, with each avatar part as an object. By reproducing and storing all avatars, including those prepared in advance by the provider, as a combination of the object identification codes of each part, it is possible to reduce the amount of data to be processed in communication between avatars and to minimize delays in time other than analysis processing.

[0094] Now, after user A completes the initial settings, user B has not completed the initial settings, so there may be cases where the avatar of user B does not exist in the avatars in the mobile information terminal 200a. Even in such a case, it is possible to identify user B by accessing the user information storage unit 402 of the user management server 400 and entering a user ID that allows user B to be searched for by hashtag (@xxxxxx) or account name, although this is a known method. In addition, the personal authentication setting may be set in advance using a personal authentication function (for example, fingerprint authentication, iris authentication, face authentication, password authentication, multi-factor authentication, etc.) that is already provided in the mobile information terminal 200. In this case, it is possible to prevent the mobile information terminal 200 from being used by others without the need to authenticate with an authentication server.

[0095] In addition, when setting profile information, for example, an identification card (with date of birth) of the person proving that the person is an adult aged 18 or over may be used for eKWC authentication, or an external authentication server that has electronic authentication of a certificate of single status for a government agency may be linked. If electronic linking is not possible, the user may be required to submit PDF data or image data of various certificates, and before the data is registered in the user information storage unit, the server administrator may manage the data as secure personal information data separate from the user information storage unit, and information such as age and whether or not the user is single may be registered.

[0096] By adding an identity authentication item in this way and linking with an external authentication server, matchmakers, particularly those looking for sincere single encounters, can set the conditions for allowing avatar-to-avatar communication in advance, such as the other person being an adult and / or single, in their profile information, creating an app environment in which avatar-to-avatar communication is only possible with users who meet those conditions, preventing married people from pretending to be single and communicating with their avatars.

[0097] Although a user is required to set at least one avatar during initial setup, this is not limited to this. For example, a user can use different avatars for the different communities he or she participates in, such as for work (main job), work (part-time job), friends after entering the workforce, friends from college, friends from high school, family, relatives, private use, clubs, hobby clubs, etc., and it goes without saying that a user can use different avatars for each community.

[0098] On the other hand, avatar-to-avatar communication, in which the video or audio of the avatar-to-avatar communication is analyzed and reflected in one's own avatar and stored and updated in the other party's mobile information terminal 200, can be stored in association with the other party's conversation and avatar video information that are the basis for reflecting the emotion, so care must be taken to avoid any privacy issues between users. Therefore, for example, an avatar registered for marriage purposes should be set up to emphasize the confidentiality of the individual and proof of being single, and it is also desirable to give permission to the other party if it is desired to reflect the heightened emotions caused by conversations on the avatars of both parties and store the compatibility, etc., in association with the avatar along with the conversation and emotional movements.

[0099] When user A designates user B's account to perform avatar-to-avatar communication, the mobile information terminal 200a first performs the above-mentioned personal authentication process, and then, after authenticating that the avatar communication request is from user A, transmits user A's avatar information to user B (S706). At this time, the avatar image itself may be transmitted, but it goes without saying that the amount of data can be reduced by transmitting the avatar identifier (if the same avatar does not exist when stored on the server side, it is stored together with a unique ID).

[0100] The user information storage unit 402 of the user management server 400 is queried as to whether or not user B meets the avatar usage conditions desired by user A and is therefore capable of communication (S707). If all of the permission setting information requested by user A matches, then avatar information for user B is exchanged (S709), and the avatar-to-avatar communication is successful. User A receives avatar B used by user B on the mobile information terminal 200b, displays it on the screen display unit 201 without storing it in the data storage unit 202 (streaming it as video at all times), and avatar-to-avatar communication begins in which video and audio are simultaneously sent and received between users A and B (S710, S711).

[0101] On the other hand, if at least one of the permission setting information requested by user A does not match, the mobile information terminal 200b displays to user B via the screen display unit 201 the usage conditions that did not match in the avatar communication with user A, and displays a message such as "Do you want to permit the usage conditions from user A?" to leave it to user B to decide whether to permit the communication (S708). User B selects "yes" if he permits the conditions requested by user A and starts the avatar communication, or "no" if he does not permit it, according to the displayed contents on the screen, and if it is "no", a message that the communication has failed is displayed on the mobile information terminal 200a of user A (S708). If it is "yes", the same processing as in S709 to S711 is performed, and the avatar communication is started. Note that when the message that the communication has failed is displayed, user A may recognize that the avatar BBB has different usage conditions, and may accept the different conditions and make a request again. In that case, in S708, it is determined again whether the usage conditions of avatar BBB match according to the flow of the dotted line, and the same processing as described above is performed.

[0102] The data processing units 205, 305, and 405 perform exactly the same processing in Fig. 2a and Fig. 2b, and Fig. 7a and Fig. 7b, but the targets of data transmission and reception after the start of communication between avatars are different. In Fig. 2a, the mobile information terminal 200a does not analyze the video, capture the motion, or detect the facial expression or emotion, but performs all processing in the audio analysis unit 608 and video analysis unit 609 in Fig. 6a, which is a so-called cloud AI communication, so in Fig. 7a, both the audio and the video are analyzed by the cloud AI. On the other hand, in Fig. 2b, the edge AI of the mobile information terminal 200b analyzes the video, captures the motion, and detects the facial expression and emotion, and the results are reflected in the avatar in the avatar movement conversion unit 614 and avatar information input unit 616 in Fig. 6b, which is a so-called edge AI analysis.

[0103] In the data analysis process by the cloud AI after S712 in FIG. 7a, the edge side of the mobile information terminals 200a and 200b only collects video and audio data, and the data sent from the mobile information terminals 200a and 200b to the analysis server 600 side is learned and inferred by the AI ​​installed on the cloud. On the other hand, in the data analysis process by the edge AI after S712 in FIG. 7b, the edge side of the mobile information terminals 200a and 200b collects video and audio data, motion captures user movement information, infers facial expressions and emotions, and performs learning processing on the inference data sent to the analysis server 600 side on the cloud. Therefore, since the processing flows are different, they will be described in detail step by step.

[0104] In Fig. 7a, after the start of communication between avatars, the voice of each user is input and collected from the voice input unit 207 of the portable information terminals 200a and 200b, and the video is input and collected from the imaging unit 209, and the voice and video data are transmitted to the analysis server 600 (S712, S714), and basically the voice of user A who started the conversation can be transmitted to the other party via communication between avatars without delay (S713, S715). Specifically, from the start of communication between avatars until the end is selected, the communication channel is connected as one session, and data communication is possible in real time.

[0105] Here, since the analysis process on the information distribution server side intervenes depending on the conditions of use for each user who performs communication between avatars, whether or not the user's own movements are reflected in the avatars A and B, and whether or not the user's own facial expressions and emotions are reflected in the avatars A and B, are determined by inquiring of the user information storage unit 402 of the user management server 400 as conditions of use (S717, S722). When the movements of user A are to be reflected, the movement information data obtained by converting the video analysis results on the analysis server 600 side into the movement conversion unit 604 for avatar A is transmitted to the mobile information terminal 200b (S718), and the movement information of the avatar is reflected as difference / additional information in the avatar A stored in the mobile information terminal 200b (S719), and is displayed on the screen display unit 201 of the mobile information terminal 200b as if the avatar were moving in real time as body language together with the voice of user A to user B.

[0106] Similarly, when reflecting the movements of user B, the movement information data converted from the video analysis results on the analysis server 600 side by the movement conversion unit 604 to avatar B is transmitted to the mobile information terminal 200a (S720), and the avatar movement information is reflected as difference / additional information in avatar B already stored in the mobile information terminal 200a (S721). To user A, the avatar is displayed on the screen display unit 201 of the mobile information terminal 200a as if it were moving in real time as body language together with the voice of user B.

[0107] Next, when the facial expressions and emotions of user A are to be reflected, the emotion inference results based on the audio and video analysis on the analysis server 600 side are transmitted to the mobile information terminal 200b to be reflected in the facial expressions and movements of avatar A. The analysis data is then linked to the conversation and avatar movement information of user B to be analyzed, and stored in the analysis data storage unit 602 of the mobile information terminal 200b and the analysis server 600 (S723, S725).

[0108] Similarly, when reflecting the facial expressions and emotions of user B, the emotion inference results based on the audio and video analysis on the analysis server 600 side are transmitted to the mobile information terminal 200a so as to be reflected in the facial expressions and movements of avatar B, and the analysis data is linked to the conversation and avatar movement information of user A to be analyzed, and stored in the analysis data storage unit 602 of the mobile information terminal 200a and the analysis server 600 (S723, S724).

[0109] On the other hand, if the facial expressions and emotions of user A and / or user B are not reflected in the avatars, then, based on the conditions of use, either the avatar A and / or avatar B displayed on the mobile information terminal 200b and / or mobile information terminal 200a faithfully reflects the user's movements and displays an avatar image, or a still image of avatar A is displayed on the screen and audio is played in a case where no movements are reflected, resulting in inter-avatar communication. Then, audio and video data are transmitted from the mobile information terminals 200a and 200b to the analysis server 600 (S712, S714), and the processes from S713 to S727 are repeated until the end of the inter-avatar communication is detected by either or both of the mobile information terminals 200a and 200b (S726, S727).

[0110] It goes without saying that the decision as to whether or not to reflect the actions and emotions in S717 and S722 may be made by a common reflection process during the avatar-to-avatar communication session after the terms of use have been confirmed in the initial analysis process, or the setting as to whether or not to reflect the actions and emotions may be received from the key input unit 206 of the mobile information terminal 200a and / or 200b as appropriate thereafter, and a decision may be made each time.

[0111] In this way, the converted movement information data and analysis data are reflected as difference / additional information in each avatar (S719, S721, S724, S725), and compared to communication between avatars in which voice is superimposed on motion capture video that does not reflect emotions, it is possible to naturally analyze what users A and B felt through their conversation with each other, and to display an avatar that will improve the content of communication between users (not only simply changing the movements and expressions of an avatar that resembles a user, but also superimposing an object other than the avatar to update the avatar video to one that is more emotional, and including additional processing using any avatar that has been pre-registered as a process for adding emotion), and this is displayed on the screen display unit 201 of the portable information terminal.

[0112] In particular, in communications between users, highlight conversations and the reflection of the other party's emotions, actions, and other movement information on the avatar are stored in the user management server like an answering machine system, and the content of the conversation that one person made and the emotions the other party felt as a result of the conversation are linked and saved via the avatar, so it goes without saying that it is easy to analyze emotions within the entire conversation after the communication between avatars is terminated. Furthermore, since the analysis results can be fed back to show whether emotions were expressed between the two parties, it is possible to introduce a mechanism that treats overdetection by the analysis server as false positives, or allows manual emotion registration for conversations where detection was missed.

[0113] Specifically, emotion estimation is performed on the analysis server 600 side when cloud AI is adopted, and on the mobile information terminal 200 side when edge AI is adopted, but not all results from the analysis server are necessarily correct. Rather, when starting to use, it is not unusual for there to be missed detections or false positives, and if feedback can be obtained from the other side, especially regarding emotions, learning will deepen, and by reflecting and saving the feedback on the other avatar (S728, S729, S730), that is, the more avatar-to-avatar communication is performed between users, the more learning data will be obtained and the higher quality emotion estimation will be possible.

[0114] This allows you to look back on past avatar-to-avatar communications that are fond memories through the avatar of the present invention, and to make an effort to capture the heart of the person you like in the future by using the avatar of the present invention to CHAIN ​​the conversation of the other person and your own avatar. If this invention had been made sooner, it would have been possible to use generative AI to imagine new conversations as digital heritage, and leave it to future generations as a digital heritage that will not deteriorate in the future, rather than a fragile memory environment of human memory that simply fades as memories, and contribute to strengthening the bonds of family and kinship in Japanese society, which have become weak, such as letting descendants know what kind of ancestors they were.

[0115] Therefore, this invention is the essential part that the lead inventor decided to evolve the technology in this digital and AI era, which has been limited to recording one-sided recorded messages and recording conversations to improve service quality for many years, and to provide better communication, and to change you and the world through the realization of this invention. Each user can link data related to their own words and actions, along with the results of an analysis of their emotions at that time according to their settings, to the avatar of the person they exchanged with and record it on their mobile information terminal and / or user management server. If the CHAIN ​​range is "only the other person," the person is the direct party to the conversation, and the recording and recording are made with the consent of both parties, so both parties can check it at any time later when they need it. If the CHAIN ​​scope is not "only the other party", for example, out of consideration for privacy, the contents of the conversation, the emotions at the time, the number of consecutive conversations (for example, the topic of sushi continued for several combos with both parties feeling happy), etc., are "matters belonging to the parties of the conversation" and "contents for which the consent of the parties can be assumed", so only the gist of them may be made available for viewing and sharing in the CHAIN ​​scope comprehensively, or, if a third party viewed from the perspective of the parties of the conversation in the CHAIN ​​scope requests to "listen to the recording", a confirmation request may be made to the speaker's avatar each time, as long as it does not go against the will of the parties' consent. Needless to say, audio and video data may be viewed and shared without the consent of the parties as a "legitimate business act" by institutions with the authority to issue orders to produce recordings in some cases, such as the police or courts. Furthermore, obtaining the aforementioned audio and video data from the user management server may be legally determined to be obtaining it from a network that mediates communications between the parties. In such a case, if the same audio and video data can be obtained from a mobile information terminal, based on the doctrine of substantial identity regarding the secrecy of communications, consent may be requested each time from the mobile information terminal of the party in question, and the data may be received from the mobile information terminal of the party in question that stores the data.

[0116] Data utilization and data analysis are fundamental innovations that should be popularized to accelerate new communication in the future, freeing us from the constraints of enjoying and learning from conversations we have not noticed before (for example, things that only remain in memory or voicemails), and providing a better environment for enjoying revolutionary conversations in the new AI era. They are the basis for expanding the fields of application. We will later describe in the notes to the user table how this invention can be utilized in various business scenarios for each combination of users.

[0117] Now, regarding FIG. 7b, which employs edge AI, the differences from FIG. 7a, in which user C starts avatar-to-avatar communication with user D, will be mainly described. Up to S711, it is the same as FIG. 7a, and at the beginning of avatar-to-avatar communication, only audio can be sent and received simultaneously. The video data is analyzed by the data processing unit 205 of the mobile information terminals 200c and 200d, whereas the processing performed by the avatar motion conversion unit 604 in the analysis server 600 in FIG. 7a is performed by the data processing unit 205 of the mobile information terminals 200c and 200d. Therefore, the analysis processing of audio and video S716, S717, S722, S723, and S728 is not performed on the analysis server side, but is performed by the mobile information terminal 200 on the edge side. Therefore, the processing by the mobile information terminal 200c is indicated by a dash in the sequence number, and the processing by the mobile information terminal 200d is indicated by a double dash, but it can be considered that the same processing is basically performed.

[0118] The data linked to the stored analysis target voice and / or video data and the analysis result data is aggregated and analyzed, for example, to determine the extent to which the fluctuation and frequency of each emotion type (joy, anger, sadness, and happiness) of user C's emotions during avatar-to-avatar communication and the type of topic discussed with user D (for example, favorite foods such as sushi and grilled meat) were detected in the entire conversation between user C and user D. Specifically, for the emotion type for each topic that meets a predetermined condition, the topic type of user C himself, the avatar identifier that identifies the avatar, the emotion type, the number of times that emotional fluctuations occurred, and the amount of emotional fluctuation are analyzed. If it is determined that the number of times that both parties felt joy or the amount of emotional change in the same topic of sushi are high, the information is accumulated and stored in the user information storage unit 402 as a common topic, hobby, and thought of both parties, separate from their own hobbies and thoughts.

[0119] Then, the conversation or meeting is analyzed comprehensively from start to finish, and a decision is made as to whether or not to update the avatars between the users based on their relationship (S731). If it is determined from the conversation between users C and D that their relationship is good, then, for example, avatar synthesis unit 308 of avatar distribution server 300 performs synthesis processing to synthesize the characteristic parts of the avatars used by each user with the respective parts, and the more favorable the conversation, the more similar the shapes of the parts of both users become to make them look similar.

[0120] Alternatively, the same update process may be performed so that the corners of the eyes of both avatars are changed to smiling faces. In this way, the analysis results are reflected in the avatar communication history between avatars and the avatar list, which are equivalent to the phone books of the mobile information terminals 200c and 200d, and the avatar information of each other is stored, making it possible to visually confirm the accumulation of conversations through changes in the avatars.

[0121] Note that avatar-to-avatar communication is a service that allows communication after anonymizing personal information, and the condition for use is age authentication processing using public authentication information, such as a certificate of single status or My Number card, and the system is designed so that users cannot be identified from the information CHAINed to the avatar identifier, except in cases where the user is seriously looking for a marriage partner, etc. However, for users who authenticate using a certificate of single status, this information is pseudonymized in the sense that the personal information can be identified by matching the avatar identifier with the information in the user information storage unit 402 stored in the user management server 400, but it goes without saying that this is not limited to this.

[0122] 8(a) and 8(b) are specific examples of table diagrams for managing users of avatar-to-avatar communication in the user information storage unit 402 of the user management server 400 and for storing avatars performing avatar-to-avatar communication in the avatar information storage unit 401 of the user management server 400, which are used in common in one mode in which the analysis process is performed on the analysis server 600 side and another mode in which the analysis process is performed on the mobile information terminal 200 side. For example, R001 stores profile information entered by item from the key input unit 206 of the mobile information terminal 200 when the user D sets the avatar. In this input, the user can enter as he or she wishes to register. However, for example, when a user obtains a single certificate from a certification server and wants to have a conversation with a user who wants to have avatar-to-avatar communication only with single people, the real name is entered and a check is performed in cooperation with a certificate server issued by an administrative agency, or the server administrator performs identity verification and / or age authentication of the certificate, so the real name is entered as the name and age.

[0123] The avatar name may be fictitious, and the real name may be displayed or transmitted only to those authorized by the user during authentication or to whom disclosure is permitted. Needless to say, it is also possible to set an item such as name not to be disclosed to other users, as in the case of user F. Even in this case, if data has already been entered, it goes without saying that authentication processing is possible based on the conditions of use, such as whether the user has been authenticated, whether the user is 18 years of age or older, whether the user is single, etc. In the communication between avatars of the present invention, an anonymous distribution means is basically provided based on a combination of the avatar name, avatar, and profile information (some of which can be masked as non-disclosure), and the user's name is not made public, and when various certifications are made, identity authentication, age authentication, and single status verification can be performed by internal processing.

[0124] As shown in record R001 in Fig. 8(a), user registration has been completed with an avatar name of Ichiro and an avatar ID of AAA. This avatar ID is stored in avatar storage unit 304 of distribution server 300, where applications are downloaded and avatar settings are performed, as shown in the avatar storage table in Fig. 8(b). In the case of avatar AAA, each of the face and body objects constituting the avatar is selected individually to form and register a single avatar, and an action file that reflects predetermined standard movements on the avatar is stored together with the part identifiers of each object.

[0125] It goes without saying that this avatar may correspond one-to-one with a user ID, or may correspond many-to-one in the user management table. It also goes without saying that the same user may create different avatars depending on the purpose of use (for example, for work, for personal use, for a girlfriend / boyfriend, for a community, etc.) and link the avatar ID and user ID in a many-to-many manner. Also, if no object identifier is stored in an item in the avatar storage table, "-" is registered.

[0126] This is because, when an object created in advance with an avatar is selected at the time of initial registration, the combination of the avatar ID (e.g., CCC or FFF), avatar type, and action file is stored in the table. If the user subsequently updates the avatar, as described below, the object to be updated is given an identifier different from the predetermined object identifier, and if the record to be updated exists in a known table, the avatar ID is changed to that avatar ID, and if it does not exist in the known table, it is added to the avatar storage table as a new record.

[0127] 9(a) to 9(d) are specific examples of (a) a table diagram for storing the general consent conditions set by the user of the avatar ID in the general consent conditions storage unit 502 of the authentication server 500, (b) a voice / avatar storage table diagram for use in the analysis data storage unit 602 and / or the voice / avatar storage unit 202, (c) an emotion analysis data storage table diagram for use in the analysis data storage unit 602 and / or the data storage unit 202, and (d) a conversation analysis data storage table diagram for use in the analysis data storage unit 602 and / or the data storage unit 202, which are used in an embodiment in which the analysis process of the present disclosure is performed on the analysis server 600 side and / or the mobile information terminal 200 side. Based on the explanation of the tables so far, avatar data, etc. are generated at sequence number S705 for storing profile information A and B in response to transmission at sequence numbers S703 and S704 of avatar setting (profile registration) in FIG. 7a, and will grow through future communication between avatars. Thereafter, user A makes a communication request to user B, and sequence number S707, which determines whether communication is possible, determines whether the various conditions for user B's avatar ID: BBB stored in the comprehensive consent conditions table of Figure 9a match the usage conditions desired by user A.

[0128] In this example, the usage conditions of avatar AAA, who made the communication request, are: Motion: Yes, Audio analysis: Yes, Video analysis: Yes, CHAIN: Yes, CHAIN ​​range: Partner only, Avatar reflection: Yes, and the items that differ from the usage conditions of avatar BBB are: Motion: No, CHAIN ​​range: Family. The differences in CHAIN ​​range will be explained later, but if a user with a narrow CHAIN ​​range makes an avatar-to-avatar communication request to a user with a wide CHAIN ​​range, it can be determined that the conditions are met. Avatar AAA only CHAINs with the partner, which means that the connection and promise are made as a secret between the two of them, and even if avatar BBB is set up to CHAIN ​​with family members, the video data communicated between user A and avatar BBB will not be shared with avatar BBB's family.

[0129] Since the judgment result of sequence number S707 differs between "yes" and "no" for motion reflection, a message is displayed on the screen display unit 201 of the mobile information terminal 200b asking user B whether it is OK to reflect motion in order to permit communication, i.e., avatar-to-avatar communication with user A. User B is then asked to operate the key input unit 206 to press either the yes or no button; if the answer is no, communication fails and the communication request is terminated.

[0130] If yes, avatar information is exchanged at sequence number S709. It should be noted that, not only when avatar-to-avatar communication is performed for the first time after profile registration, but also when a user has already had a conversation with another user, the avatar may have been updated through avatar-to-avatar communication with another user, so it is desirable to exchange avatar information with each other each time avatar-to-avatar communication is performed. It should be noted that even in this case, it is possible to identify and use an avatar from the avatar information stored in the information management server 102 and the mobile information terminal 200 simply by communicating via the avatar ID.

[0131] Now, the CHAIN ​​range is a setting that enables avatar-to-avatar communication between users who are active with multiple avatars, and even if the video data is from anonymized avatars, unless the user has set the CHAIN ​​range to be the same or wider than their own out of consideration for privacy, the video data will not be shared over a wide range. Therefore, it goes without saying that it is possible to share video data with a specific comprehensive consent range or avatar ID to ensure the confidentiality of communications.

[0132] In the present disclosure, data managed by another system such as the virtual communication terminal 200 may be linked to the data (for example, operation logs of smartphone apps provided by other companies, and log information such as call history and location information managed by the server of another mobile phone company) and managed as virtual communication target information. In reality, it is difficult to collect logs due to communication secrecy and laws imposed on telecommunications carriers. This can also be applied to the virtual communication management server of the present invention, and normally, disclosing the contents of conversations between users to a third party, even if it is a family member, without the consent of the person in question is an inviolable area because it is something that telecommunications carriers must thoroughly manage confidentially.

[0133] Therefore, in this disclosure, it is assumed that a shrink-wrap contract is made for a virtual communication program, which assumes that the user agrees to the acquisition of virtual communication target information when installing the application (which is designed to enable unique identification and centralized management of user IDs and terminal IDs).

[0134] As a review, the definition of CHAIN, explained in item 0008, is not necessarily limited to the connection between communicating avatars being linked, linked, connected, or constructed, but is an expression that includes any update, reflection, or deformation of an avatar that facilitates communication between users who use avatars. Specifically, in the case of communication between avatars AAA and BBB, the voice data and analysis data of the call contents are analyzed, and the motions of each user are converted into avatar movements, and emotions are also reflected in each avatar, and the synchronization and connection data is stored as video data based on the time flag.

[0135] This video data is assigned an identifier, a video data ID, for each conversation and user that the user conveys as voice and / or body language, and the transmission and recording time of the audio and video, the avatar ID of the sender, the audio data, movement data, emotion ID, conversation ID, and CHAIN ​​destination avatar ID, etc. are stored as record E001 in the audio and avatar storage table in Figure 9b.

[0136] Now, the emotion ID is determined by analyzing emotions such as joy, anger, sadness, and neutrality from the human voice contained in the video and conversation using a voice AI function (which may be realized through external API integration) provided in the audio analysis unit 608 and video analysis unit 609 of the analysis server 600 and the audio / video analysis unit 219 of the mobile information terminal 200, and is stored in the emotion analysis data storage table of Figure 9c as an emotion ID for each conversation of one user.

[0137] There are various theories for the classification of emotions in this emotion analysis data storage table. For example, if we take the Japanese voice emotion analysis AI "Empath (registered trademark)" as an example, it can estimate emotions such as joy / gratitude, favor / interest, gratitude / trust, surprise, sadness, fear, anger, disgust, and neutral / peace based on an emotion estimation algorithm, and estimates and stores each level as a degree. Even if the same congratulations are conveyed, whether it is a private or public scene such as work, and whether the detected emotions are multiple emotions such as "joy", "favor", and "surprise" that are estimated at the same time, the sincerity and strength of User A's feelings are expressed and converted into Avatar AAA.

[0138] This makes it easy for people who are not good at sensing the emotions of others to understand because the emotions are recorded as specific video ID data and the avatars can visually confirm the changes and connections in real time. In addition, the classification of each emotion can be a more detailed breakdown of the major categories, or if it is possible to estimate which conversation content caused the other person to feel that emotion, the content can be stored.

[0139] Here, examples of words that are used when speech is converted into text, which are important elements for detecting and estimating each emotion, are given below (please note that as we are not experts in this field, some of the lists or classifications may not be appropriate): (1) joy includes ecstatic, happy, delighted, feeling good, overjoyed, delighted, thankful, fulfilled, gratitude, happiness, thanks to you, etc.; (2) favorability and interest includes fun, delightful, interesting, hilarious, laughable, adorable, like, enthusiastic, absorbed, engrossed, interested, curiosity, fascinated, etc.; (3) gratitude and trust includes interest, moved, awe, admiration, admiration, moved, impressed, dependable, trust, entrusted, believe, etc.; and (4) surprise includes energy, encouragement, excitement, achievement, motivation, lively, thrilled, etc.

[0140] (5) Sadness includes isolated, lonely, alienated, gloomy, disheartened, miserable, depressed, shocked, gloomy, heavy-hearted, overwhelmed, pain, hurt, sorrowful, pathetic, etc.; (6) Fear includes scared, frightened, anxious, worried, impatient, panic, perplexed, troubled, upset, confusion, etc.; (7) Anger includes humiliation, indignation, irritation, irritation, rage, feeling uneasy, etc.; (8) Disgust includes resentment, hatred, hostility, resentment, guilt, remorse, cold, unpleasant, bad mood, contempt, distrust, etc.; (9) Normality and peace include kind, warm, secure, relaxed, relieved, easy-going, calm, ecstatic, composed, etc.

[0141] Next, the conversation ID is stored in the conversation analysis data storage table in Figure 9d as the conversation analysis results when analyzing the voices between users, and is linked to the video ID. Conversation ID: 400001 is what avatar AAA said to avatar BBB, and in addition to the conversation type and content type, the analysis results of "when, who, to whom, what, why, and how" are stored as items after conversation analysis of the elements of the conversation content.

[0142] To explain CHAIN ​​in more detail based on conversation IDs 400001 and 400002, in the case of users A and B, the voice and body movement information of "Congratulations on entering XXXX University" generated by user A through avatar AAA is reflected in the avatar along with various analysis results, causing a change in the avatar. The first CHAIN ​​is to link each video data with the avatar ID of avatar BBB, which is the listener of the inter-avatar communication.

[0143] Based on this avatar ID, video ID, and other identifiers, conversations can be played back based on previously recorded video data IDs at any time during avatar-to-avatar communication between users, and even when no avatar-to-avatar communication is taking place, each user can check the video data recorded in past avatar-to-avatar communication on their mobile information terminal 200.

[0144] This CHAIN ​​defines the range of access rights that allows a user to refer to the chain of video data IDs between multiple avatar IDs included in the CHAIN ​​range preset by the user. Even if the CHAIN ​​range is set to "other party only," the spiral connection of multiple video IDs generated from multiple conversations is also considered to be CHAIN, and the conversation analysis results for the entire conversation between users of avatar-to-avatar communication who are continuing a conversation that continuously brings out the other party's feelings of joy, such as an EXCELLENT combo in the Taiko no Tatsujin (registered trademark) app game, will provide CHAIN ​​data that proves that they have a good relationship.

[0145] Of course, when a similar CHAIN ​​is performed between a family of four, statistically there are 4P2+4P3+4P4=4×3+4+1=17 combinations of avatar IDs and video IDs between mutual users, which are aggregated into 4C2=4×3 / 2!=6 combinations of avatar connections, and the avatar ID to the CHAIN ​​destination avatar ID may be in a many-to-one relationship of 1 or more. Needless to say, references to such video information can be shared between family members. In addition to family members, it may be between colleagues or friends in the same company, or between communities of one's hobbies, or it may be openly CHAINed without limiting the scope of the CHAIN. Needless to say, the conversations between these users are stored in the conversation analysis data storage table of FIG. 9d, and may be used for calling from individual video IDs, conversation types, content types, and other statistical data analysis processing.

[0146] Now, after the emotion ID of avatar AAA, the movement data converted into avatar movement through motion capture based on the audio data and video data to be analyzed, the recording time and conversation 400001 are stored and linked, user B who has viewed the video similarly stores conversation 400002 through avatar BBB, and the "video data reflecting user B's facial expressions and emotions in response to user A's conversation" is linked to the CHAIN ​​destination avatar AAA.

[0147] In this way, user A can connect to avatar BBB, which has anonymized the content of user B's conversation as well as facial expressions and emotions, in order to analyze future conversations and improve the content of future conversations with other users, and above all, user A can replay and use the data as a digital legacy, such as memorable conversations with his or her late father, as a treasure to encourage him or her.

[0148] In addition, the user's voice may be obtained as a sample when the app is used in advance, such as differences in pitch from normal intonation, and if the volume or intonation is different from normal, the emotion level may be increased to estimate the emotion. In the consent for use when using the app, a guide voice may be registered in advance containing comprehensive agreements regarding the confidentiality of communication between users, and important sentences may be narrowed down to the points that allow intonation analysis, and the user may repeat each sentence. In this way, audio evidence of agreement to the terms of use is also recorded and can be used as a reference point for subsequent communication between avatars, making it possible to more accurately analyze emotions and conversations according to the user's speaking style and tone of voice. EXAMPLES

[0149] Up to this point, the configuration of the present invention has been illustrated hierarchically, and the embodiments have been explained in detail, broadly divided into edge AI and server AI. In the postscript, the effect of the CHAIN ​​communication between avatars of the present invention will be explained in more detail, and it will be explained that other modified examples can also be applied.

[0150] In order to create a tomorrow we can't wait for, the present invention provides a virtual communication terminal, a virtual communication management server, a virtual communication method, and a virtual communication program that link the conversation content between the listener's avatar and the speaker's conversation content according to the communication content of the virtual communication and a character (avatar, etc.) selected by the user as the virtual communication target information, allowing users to confirm each other's humanity through conversation with the avatar.

[0151] The virtual communication terminal, virtual communication system, virtual communication method and virtual communication program of the present invention according to claims 1, 7, 8 and 10 are characterized in that they have an exchange means for exchanging virtual communication target information based on predetermined conditions for the virtual communication, a linking means for linking the virtual communication target information of the sender and the virtual communication target information of the receiver based on the content of the virtual communication, and a means for updating the virtual communication target information before the exchange based on one or more linked pieces of virtual communication target information.

[0152] With this configuration, the contents of virtual communication between virtual communicators are mutually linked to the virtual communication target, and the virtual communication target information can be changed, i.e., updated, according to the contents of the conversation. For example, if you have a good chemistry with someone who often makes you laugh and entertains you, you can link the contents of the conversation with an image of the other person's avatar laughing out loud, or display it in a list equivalent to a communication history or phone book, so that you can visually recognize the person and want to talk to them again. You can also check later the contents of the conversations that made the other person laugh, the level of happiness you felt, and other things that made you happy, and you can easily check the relationship between users.

[0153] Therefore, this invention is not limited to simple virtual communication, i.e., communication that guarantees anonymity, but is an invention that helps people enjoy conversations with others by being purely their true selves in voice and actions, rather than being clad in a shell of superficial or social status that is not an essential part of humanity, such as the other person's appearance or profile information, so that they do not get nervous when talking to strangers and are unable to talk about their true selves.

[0154] For example, in the virtual communication CHAIN ​​of the workplace community, if a boss scolds a subordinate in virtual communication, the subordinate may become depressed. Until now, there are many cases in the world where people have no one to talk to at work, and they are unable to handle their work execution ability, including their own communication, and the expectations and amount of work assigned to them by their boss, leading to mental illness.

[0155] Even if you are careful, your colleagues often don't notice, and I remember the days when I would make things public hoping that someone would consult with me before it exceeded the threshold, but unless you see or hear about the communication between the boss and the subordinate directly, you won't be able to sense anything, and you won't even have an opportunity to help. Even in such daily individual human relationships, future DX-enabled One Team members will be able to check the gist of the conversation (type and gist of the conversation, type and level of emotion felt, etc.) with the person's comprehensive consent, and will be able to check past interactions, etc.

[0156] Even if a conversation is between superiors, other subordinates can also see it. This will make the conversation more open, avoid the pitfalls of individual relationships, and allow members to help each other and face any issues that may be triggering, strengthening the team's ability to form a unit. CHAIN ​​can be used to solve such issues that are difficult to create manuals or deal with centrally in labor management.

[0157] From this perspective, as an invention that digitalizes simple and orthodox conversations such as telephone, answering machine, and chat in the mobile communications field, where there is a sense of stagnation and no sense of progress at all, and incorporates various AI functions developed as a result of various cutting-edge research, in order to connect human relationships through user conversations, it goes without saying that it can be applied to various cloud services for ordinary mobile phone services, as well as to the VR matching services mentioned earlier as an application field.

[0158] Furthermore, in fields such as call center business for services provided by various companies, where clients primarily request conversations in the form of questions regarding specific products and services, it can also be applied to a system in which multiple call center personnel can use one shared avatar to centrally manage virtual communication responses to a certain user, and there is no doubt that it could become an ideal business field in which a shared avatar can be utilized by specializing the conversation analysis dictionary as a way to deal with complaints in call centers that focus on emotion analysis.

[0159] Next, the connecting means of the virtual communication terminal and virtual communication method according to claims 2 and 11 of the present invention is characterized by a means for exchanging virtual communication target information with one or more receivers corresponding to a selection of pre-stored virtual communication target information, and an updating means for updating the virtual communication content according to the analysis results.

[0160] With this configuration, for example, when a communication request is made from a virtual communication terminal to virtual communication target information simulating a community room for people with the same hobby, or, in the case of a workplace environment, a conference room set up in a virtual space, it becomes possible to automatically exchange avatars with users who are already participating in the community room (this depends on the virtual communication permission conditions for each user described above, but it is also possible to set the CHAIN ​​range to suit the needs of protecting privacy and seeking broader connections), making it possible to efficiently specify recipients, especially when starting virtual communication with three or more people.

[0161] Next, the connection means of the virtual communication terminal and virtual communication method according to claims 3 and 12 of the present invention is characterized in that it further comprises an analysis means for estimating the speaker's emotions from changes in the features of the speaker's movements or voice in response to the listener's movements or voice, and links the analysis results of the communication content with each of the information.

[0162] With this configuration, for example, instead of estimating emotions only from video and audio data including facial expressions at a certain moment, there is always a previous conversation that led to the speaker's actions and emotions, and the previous conversation is also linked to the avatar and conversation data, so the process and reasons that led to that emotion can be easily confirmed, and virtual communication can be carried out to further develop the conversation. Furthermore, since the content of communication with the other party is linked to the other party's virtual communication target along with the other party's reactions of joy, anger, sadness, and happiness, the conversation including the other party's emotions can be confirmed even after the virtual communication ends.

[0163] One possible application would be to accumulate this data and analyze it as statistical data. For example, one possible application would be to compile statistics on the current content of popular conversations, the percentage of the total number of virtual communication users, and the topics and content managed by conversation IDs contained in the virtual communication target information, to determine what topics are trending at what time and in what area, and to use the data from a macro perspective to provide a data matching service for companies that want to do targeted advertising.

[0164] In addition, it is also conceivable that users themselves may utilize data, that is, provide a service for viewing data for a fee. For example, before a salesperson who is planning to dine or meet with a venture company president, a celebrity, or someone with some influence in the industry, can provide information to get to know the other person in advance, such as the conversations that the person has had in the past, and the type of conversations, for example, talking about sushi with various people, within the range permitted for disclosure, after the discloser has appropriately set the CHAIN ​​range, in order to get the president interested in his proposal, and there are times when he needs to weave in some favorable conversation with the other person, and when he wants to know the key points of the conversation with the other person in advance, it is conceivable that the president himself can provide extremely useful reference information as statistical data (with the personal information of the person with whom the statistical target user has spoken being masked, of course).

[0165] Everyone wants to have a conversation with someone who understands them, who can draw out the conversation by hitting the sweet spot, who can talk to them in a way that suits their current feelings, and who is not just a simple sales pitch or a pushy sales pitch. By applying this invention, salespeople can hone their sales pitches and build relationships with people through conversations that are customized for their audience, which increases the likelihood of a successful business negotiation.

[0166] Next, the updating means of the virtual communication terminal and the virtual communication method according to claims 4 and 13 of the present invention is characterized in that it updates the virtual communication target information stored by at least two or more speakers and listeners so as to absorb the difference between the virtual communication target information stored by at least two or more speakers and listeners.

[0167] With this configuration, for example, changes in the virtual communication target resulting from feelings such as liking or dislike for the other person become similar, allowing the users to grasp the communication relationship between them more concretely.

[0168] Next, the updating means of the virtual communication terminal and the virtual communication method according to claims 5 and 14 of the present invention is characterized in that the feature points of the objects included in the virtual communication target information are updated so as to be common.

[0169] With this configuration, for example, the characteristic points of objects contained in virtual communication target information are updated to be common, so that while the overall characteristics of each piece of virtual communication target information are maintained, only the parts that need to be updated can be updated efficiently and with low communication traffic, thereby reducing the load caused by the update process.

[0170] As an example of the ultimate application, if edge AI could be made more powerful and virtual communication terminals could be made more efficient in processing, it would be possible to digitalize real-time conversations, which have traditionally been considered untouchable territory due to the need for calling, session establishment and maintenance (resource occupation), and secrecy of communications (anonymity). This could enable high-added-value communication using even fewer resources, such as no need for calls, virtual communications based only on parameters such as identifiers, no need to maintain sessions, blockchainization of communication packets, and the elimination of the need to manage the secrecy of communications through the use of blanket consent and one-time consent acquisition processes.

[0171] In addition, the virtual communication management server of claim 6 of the present invention is characterized in having a means for pre-storing user information and permission conditions related to virtual communication as virtual communication target information, a communication means for encouraging the sender and / or receiver's terminal to start virtual communication in accordance with the communication conditions included in the virtual communication request from the sender and the receiver's communication permission conditions, and a data processing unit for continuing the virtual communication by linking the listener's virtual communication content to the speaker's virtual communication target information and the speaker's virtual communication content to the listener's virtual communication target information.

[0172] With this configuration, for example, even with a party having different user information and permission conditions for virtual communication, virtual communication can be performed with peace of mind while checking the virtual communication target information of the other party each time to determine whether or not to perform virtual communication.

[0173] The virtual communication method according to claim 9 of the present invention is characterized in that the predetermined condition is that the users have mutually obtained authentication regarding single certificates.

[0174] With this configuration, for example, by objectively certifying that users of virtual communication have mutually obtained certification of their single status, users who are seeking serious encounters through various matching services can feel safe in matching for marriage hunting and the like. [Industrial Applicability]

[0175] In order to create the tomorrow we can't wait for, the present invention can be applied to a virtual communication terminal, a virtual communication management server, a virtual communication method, and a virtual communication program that can link the conversation content of the listener's avatar and the speaker's avatar according to the communication content of the virtual communication and the character (avatar, etc.) selected by the user as the virtual communication target information, and allow users to confirm each other's humanity through conversation with the avatar. As an ultimate application example, if the high functionality of edge AI and the processing efficiency of the virtual communication terminal are realized, it will be possible to apply it to real-time conversations that were previously thought to be inviolable areas of calling, session establishment and maintenance (resource occupation), and communication secrecy (concealment) as the idea of ​​communication. In addition to mobile information terminals, we will also aim to digitalize everyday conversations using smart rings, smart watches, smart glasses, etc. that do not have a screen display unit, and it may be possible to apply it to communication with even lower resources and higher added value, such as no need for calls, virtual communication only with parameters such as identifiers, no need to maintain sessions, blockchainization of communication packets, and elimination of the need for communication secrecy management by encouraging the acquisition of comprehensive consent and consent each time. [Explanation of symbols]

[0176] 101...information distribution server, 102...communication network / network unit, 200a, 200b, 200c, 200d...portable information terminal, 300...avatar distribution server, 400...user management server, 500...authentication server, 600...analysis server, 201...screen display unit, 202...data storage unit, 203, 301, 409, 501, 603...communication unit, 204...input data synchronization unit, 205, 305, 405, 605...data processing unit, 206...key input unit, 207...audio input unit, 208...audio output unit, 209...imaging unit, 219...audio / video analysis unit, 302...application storage unit, 303...application update management unit, 304...avatar storage unit, 30 6...avatar creation unit, 307...avatar update processing unit, 308...avatar synthesis unit, 309...billing management unit, 401...avatar information storage unit, 402...user information storage unit, 403...authentication processing unit, 404...data synchronization unit, 406...analysis data synthesis unit, 407...audio analysis result input unit, 408...video analysis result input unit, 502...comprehensive consent condition storage unit, 503...authentication judgment unit, 504...consent request unit, 505...consent confirmation unit, 601...audio / avatar storage unit, 602...analysis data storage unit, 604...movement conversion unit to avatar, 606...avatar information input unit, 607...audio data reflection unit, 608...audio analysis unit, 609...video analysis unit

Claims

1. A virtual communication terminal in which a sender communicates with one or more receivers via virtual communication target information, characterized in that the virtual communication terminal has an exchange means for exchanging the information based on predetermined conditions for the communication, a linking means for linking the information of the speaker and the information of the listener based on the content of the virtual communication, and a means for updating the information before exchange based on one or more linked pieces of information.

2. 2. The virtual communication terminal according to claim 1, characterized in that the exchange means exchanges the information with one or more recipients corresponding to a selection of the information stored in advance, and the update means updates the communication content according to an analysis result.

3. The virtual communication terminal according to claim 1 or 2, characterized in that the linking means further comprises an analysis means for estimating the emotions of a speaker from changes in the features of the speaker's movements or voice in response to the movements or voice of a listener, and links the analysis results of the communication content with each of the information.

4. 3. The virtual communication terminal according to claim 1, wherein said updating means updates said information so as to absorb differences between said information stored by at least two or more speakers and listeners.

5. 5. The virtual communication terminal according to claim 4, wherein said absorbing of said difference is performed by updating said information so that feature points of objects included in said information become common.

6. A virtual communication management server that manages virtual communication between a sender and a receiver via virtual communication target information, characterized in that it has a means for pre-storing user information and permission conditions related to the virtual communication as the information, a communication means for encouraging the sender and / or receiver's terminal to start virtual communication in accordance with the communication conditions included in the virtual communication request from the sender and the receiver's communication permission conditions, and a data processing unit that continues the virtual communication by linking the receiver's communication content to the sender's information and the sender's communication content to the receiver's information.

7. A virtual communication system comprising two or more virtual communication terminals and a management server that manages virtual communication between the terminals via virtual communication target information, wherein the virtual communication terminals have an exchange means for exchanging the information based on predetermined conditions for the virtual communication and a linking means for linking the information of the sender with the information of the recipient, the management server has a data processing unit that continues virtual communication while linking the communication content of the recipient to the information of the sender and the communication content of the sender to the information of the recipient, and the virtual communication terminals and / or the management server have a means for updating the information of both parties based on one or more linked pieces of information.

8. A virtual communication method between two or more pieces of virtual communication target information, characterized in that the pieces of information are exchanged with each other based on predetermined conditions for the communication, the pieces of information are linked to each other based on the content of the virtual communication, and the pieces of information before the exchange are updated based on one or more linked pieces of information.

9. 9. The virtual communication method according to claim 8, wherein the predetermined condition is that the users have mutually obtained authentication regarding single certificates.

10. A virtual communication program for communicating between two or more virtual communication terminals via virtual communication target information, characterized in that the program functions to cause the processors of the terminals to exchange the information based on predetermined conditions for the communication, link the information of the sender with the information of the receiver based on the content of the virtual communication, and update the information before the exchange based on one or more linked pieces of information.

11. The virtual communication program according to claim 10, characterized in that the processor is operated so that the exchange exchanges the information with one or more virtual communication targets in response to a selection of the information stored in advance, and the update updates the information in accordance with an analysis result of the communication content.

12. The virtual communication program according to claim 10 or 11, characterized in that the linking further comprises estimating the emotions of the sender from changes in characteristic points of the sender's movements or voice in response to the receiver's movements or voice, and linking the analysis results of the communication content with the information of the receiver.

13. 12. The virtual communication program according to claim 10, wherein the updating is performed so as to absorb differences between the information stored by at least two or more senders and receivers.

14. 14. The virtual communication program according to claim 13, wherein said absorbing of said difference includes updating said information so that feature points of objects included in said information are common.

Citation Information

Patent Citations

  • Information verification method and system

    JP2021072119A

  • Computer program, server device, terminal device, system and method

    JP2023015074A